Aspect-based Sentiment Classification via Reinforcement Learning
Authors: Lichen Wang
Bo Zong
Yunyu Liu
Can Qin
Wei Cheng
Wenchao Yu
Xuchao Zhang
Haifeng Chen
Yun Fu
Venue:  (2021)
Keywords: Sentiment classification
reinforcement learning

Abstract: Aspect-based sentiment classification aims to predict sentimental polarities of one or multiple aspects in texts. As texts always contain a large proportion of task-irrelevant words, accurate alignment between aspects and their sentimental descriptions is the most crucial and challenging step. State-of-the-art approaches are mainly based on word-level attention learned from recurrent neural network variants (e.g., LSTM) or graph neural networks. From another view, these methods essentially weight and aggregate all possible alignments. However, this mechanism heavily relies on large-scale supervision training: without enough labels, it could easily overfit with difficulty in generalization. To address this challenge, we propose SentRL, a reinforcement learning-based framework for aspect-based sentiment classification. In this framework, input texts are transformed into their dependency graphs. Then, an agent is deployed to walk on the graphs, explores paths from target aspect nodes to their potential sentimental regions, and differentiates the effectiveness of different paths. By limiting the agent's exploration budget, our method encourages the agent to skip task-irrelevant information and focus on the most effective paths for alignment purpose. Our method considerably reduces the impact of task-irrelevant words and improves generalization performance. Compared with competitive baseline methods, our approach achieves the highest performance on public benchmark datasets with up to $3.7\%$ improvement. 

Under review as a conference paper at ICLR 2021
ASPECT-BASED SENTIMENT CLASSIFICATION VIA REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Aspect-based sentiment classification aims to predict sentimental polarities of one or multiple aspects in texts. As texts always contain a large proportion of task- irrelevant words, accurate alignment between aspects and their sentimental de- scriptions is the most crucial and challenging step. State-of soate-of-the-ae thett approaches are mainly based on word-level attention learned from recurrent neural network vari- ants (c.g.. LSTM) or graph neural networks. From another view, these methods essentially weight and aggregate all possible alignments. However, this mecha- nism heavily relies on large-scale supervision training: without enough labels, it could easily overfit with difficulty in generalization. To address this challenge, we propose SentRL, a reinforcement learning- based framework for aspect-based sentiment classification. In this framework, input texts are transformed into their dependency graphs. Then, agent is deployed to walk on the graphs, explores paths from target aspect nodes to their potential sentimental regions, and differ- entiates the effectiveness of different paths. By limiting the agent's exploration budget, our method encourages the agent to skip task-irrelevant information and focus on the most effective paths for alignment purpose. Our method considerably reduces the impact of fusk-irrelevant words and improves generalization perfor- mance. Compared with competitive baseline methods, our approach achieves the highest performance on public benchmark datasets with up to 3.7% improvement.
INTRODUCTION
The goal of aspect-based (also known as aspect-level) sentiment classification is to predict the sen- timent polarities of individual aspects. As shown in Figure given sentence "I like this computer but do not like the screen", , the sentiment of the aspect "computer" is positive because of "like". Meanwhile, the sentiment of the aspect "screen" is negative for "do not like".
Aspect-based sentiment classification is challenging, where the core problem is to correctly align aspects with their sentiment descriptions. State-of-the-art methods rely supervision signals to automatically learn such alignment. By leveraging textual context and word-level attention learned
from deep models ( (Vo & Zhang 2015 Dong et al. 2014 Bahdanau al. 2014 Luong al. 2015
Xu et al. 2015 Wang et al.
2016 Tang et al. 2016b] Ma et al. 201 7a He et al.] [018]] Zhang
al.) 2018 2019 Gao al. 2019 Tang et al. 2020). existing methods have made great progress
on discovering aspect-specific sentimental statements.
Meanwhile, the existing methods could suffer serious overfitting problems, as natural language in- evitably includes large proportion of iereevvateevant texts, or noise from the perspective of machine learning. Ideally, with a sufficient amount of training labels, the existing methods could effectively contain the negative impact of such task-irrelevant information. In practice, because of the high variance language expression,
costly to collect large number of task-specific labels, and is difficult to guarantee the expected label sufficiency. With limited labels, the existing approaches could casily include task-irrelevant information into decision processes, overfit training data, and end up with inferior generalization performance unseen data.
To effectively reduce the impact of iiseeirreeevatt information, we propose SentRL, a reinforcement learning based framework for aspect-level sentiment classification. In approach, input texts firstly transformed into graph objects (e.g., dependency graphs (Covington) (00T)), where nodes words and edges indicate syntactic dépendennciss/rllttions between them. Next, we deploy

---

Under review as a conference paper at ICLR 2021
dobj
conj
dobj
nsubj
poss
prep
pobj
neg
poss
like
my computer
like
others
but
do
not
like
the
PRON
VERB
DET
NOUN
SCONJ
NOUN
CCONJ
AUX
PART
VERB
DET
NOUN
Figure 1: Dependency graph of a given sentence. Blue words are the aspects. Green and red are "positive" " and "negative" sentiment respectively. The dependency graph effectively reduces the distances between aspects and sentimental descriptions and avoids polysemy words (e.g., "like"). In our approach, an agent is deployed to walk from the aspect word the sentimental regions, which avoids fask-irrelevant information and achieves more effective and efficient performance.
policy-based agent to discover aspect-related sentiment descriptions in the graphs. This agent geared with a language understanding module so that it is able to update exploration states and make sentiment decisions for individual aspects. Unlike existing methods that aggregate potential sentiment information from all possible textual contexts words, most relevant exploration paths under a limited budget. This strategy not only requires the agent to focus the most effective paths but also encourages the agent skip tusk-irrelevan regions. Using standard back pack-propagation methods, the policy network and the language understanding module are jointly trained. From public benchmark datasets, observe our method could achieve up 3.6% improvement compared with competitive state-of-the-ar methods. The main contributions of our work are listed below.
agent strives to leverage the
novel reinforcement learning framework for aspect-based sentiment classification is proposed. It accurately pinpoints the most effective path between sentiment descriptions and the target pects, and effectively avoids the impact of the task-irrelevant regions.
policy network is developed to provide an agent with exploration guidance. This network iteratively provides suggestions next-hop selection. In partieular, the framework is permutation invariant and guarantees the consistency and reliability of the model.
A language understanding module is developed to help an agent "remember" its exploration his- tory and make the final sentiment prediction.
Extensive experiments on representative benchmark datasets are evaluated. The results demon- strate the effectiveness, efficiency, and robustness of our approach.
2 RELATED WORK
2. ASPECT-BASED SENTIMENT CLASSIFICATIONS
Aspect-basee sentiment classification is to identify sentiment polarities of one or more aspects in given texts (Thet al. 2010). Aspects could be either substantial objects (e.g., computer and car) or conceptional objects (e.g., service and atmosphere). There are usually three sentiment categories including positive, neutral, and negative, while more sophisticated categories could be explored. Conventional approaches (Kiritchenko et al. 2014) treat input texts as word sequences, and deploy separate feature extraction modules as well classification modules. Deep learning-basse
senti-
ment analysis methods (Tang et al.] (2016a) take contextual information regarding the word order into consideration by using LSTT (Hochreiter & Schmidhuber, 1997 Liu al. [2088) Attention- based approaches are proposed (Tang et al.
2016c] Bahdanau et al. 2014 Luong et al. 2015) Xu al. 2015; Wang et al.) 2016 Tang et al.
2016b Ma et al. 2017 7a Huang & Carley. 2019 Ma
al. 2017b Huang et al. 2018 Li et al.
2018) to improve the effectiveness of contextual Tea-
ture extraction. While such approaches utilize sequential models and attention mechanisms to learn features from word sequences, they could require large amount of training labels to be well gener- alized for natural language expressions with non- trivial variance (e.g., long sentences with majority of irrelevant contextual words).
2

---

Under review as a conference paper at ICLR 2021
2.2 SENTIMENT ANALYSIS ON GRAPHS
Syntactic dependency tree (Covington 2001 tactic dependencies between words in input texts. To this end, aspect-based sentiment classification problems can be cast into node classification problems on syntactic dependency graphs. Existing
widely adopted data structure that encodes syn-
methods develop variants of graph convolutional neural networks (Yao et al., 2019; Linmei et al.
2019 Zhang et al. 2019; Ghosal et al. 2019; Bai al. 2020 Wang et al. 2019) to extract contex- tual features that represent aspect nodes. However, such methods still suffer the following issues.
First, due the complexity of natural language, these models require large amount of training
data to be well generalized. Second, such methods (Kipf & Welling 2016 Yao et al. 2019 Linmei
al. 2019) usually perform under transductive settings with limited application scopes.
2.3 REINFORCEMENT LEARNING IN NLP
In the domain of NLP, reinforcement learning (RL) is usually explored in interactive tasks (Wang jet al., 2018), such as text-based games (Narasimhan et al. (015b). Recently, RL methods are also developed for complex tasks, such as relation extraction (Narasimhan et al. 2015a Qin et al., 2018). image captioning (Pasunuru Bansal 2017), popularity prediction (He al. [016 Zhou & Wang 2018). coreference resolution (Yin et al. 2018), and reasoning in question answering (Xiong al.
2017 Wu et al. 2018). Unlike previous works, we are the first to study RL methods that intelligently
collect information from syntactic dependency graphs for aspect-based sentiment analysis.
OUR APPROACH
In this work, we propose reinforcement learning-based framework, SentRL, for solving the chal- lenges. The details are introduced below:
Environment Feedback
Language Understanding Module
Sentiment
Wo
Wi
Wg
Sentiment Classifier
lile
but
don't
Feetume
Reward
Node
Candidate Walk
Path
prefer
laptop
sareen
A spect
3
Path State
Policy Network
Update
Dependency Graph Environment
Sg
S1
S2
Action Selection
Figure 2: Framework of our SentRL. method. A dependency graph is first obtained. The related sen- timent clues could be more close in dependency graph compared with the original sentence space. An agent starts from an aspect node and walk though the graph edges, The walk direction is de- termined by policy network which utilizes the current walk state and the direction candidates for making the decision. A semantic understanding module is used to update path states and sentiment classifier is designed to have the final sentiment prediction based on the walk state.
3.1 PRELIMINARY
Given the target texts [wf, wh, wg. w. where wf represents the i-th word in n is the total number of the words. The target aspect of e is denoted as a, = {wf+1r..., w where + indicates the start location of the aspect and m is the length of the aspect. The aspect is either a single-word format (e.g.. "computer", "service". and "screen") ") or multiple-won word format (e.g., "HDMI port" and "sport mode"). There could be one more aspects in c, and different aspects could have different sentimental categories. The goal of aspect-based sentiment classification is to predict sentiment polarity (i.e., "positive", "neutral". and "negative") for each given aspect.
3

---

Under review as a conference paper at ICLR 2021
3.2 DEPENDENCY GRAPH EXTRACTION
A dependency graph G. - (V(G.), B(G.)) is extracted from c. V(G.) indicates all vertices/nodes in Ge. and each vertice corresponds to word in E(G,) denotes the edges of G.- Each edge in Ge represents the syntactical relation categories between a pair of words. By deploying the dependency graph, we transfer the sequential text data into graph-structured format. In Ge, the long-distance pair of words in texts could be close in the graph. This provides extra syntactical knowledge and makes down-streat algorithms easier to pinpoint the sentimental words.
3.3 PATH SEARCHING VIA REINFORCEMENT LEARNING
Compared with ofate-of-the-art approaches utilizing graph learning algorithms (e.g., GCN, attention, and RNN), we proposed a reinforcement learning framework to explore the most effective aspect- sentiment path on the dependency graph. First, an agent starts walking from the target aspect node. Then, policy network selects the most effective walk based on the previous walk history and all feasible walk candidates. Next, semantic understanding module is deployed to aggregate the path state and sentiment classifier is used to obtain the final sentiment prediction. In summary, there are three components in our framework, 1) A Semantic Understanding Module which aggregates the comprehensive walk state, 2) A Policy Network which makes the walk decision, and 3) A Sentiment Classifier which obtains the final prediction of the sentiment categories. The details of each component are introduced in the following sections.
3.4 SEMANTIC UNDERSTANDING MODULE
The semantic understanding module aims provide comprehensive state of the walk for 1) al- lowing the policy network to make the effective walk action and 2) letting the sentiment classifier obtain a final prediction. There is requirement that the state updating mechanism should make the walking procedure as Markov Decision Process (MDP). The expression is shown below:
1 Ao...., S,, A,) = P(S (St, Ar),
(1)
where S,. A,, and R; are the state, action, and reward of the i-th move respectively. Eq. equation indicates that the path state of (L 1) move, 1,+1, should be only relevant to the current state S, and action A,, and irrelevant to earlier states. While, the information of the entire path crucial and necessary for sentimental analysis tasks. To this end, S, is required to preserve both the current and all previous walk information.
To achieve this goal, we deploy the general Long Short-Term Memory (LSTM) Hochreiter & Schmidhuber (1997) structure in our framework, as LSTM and its varieties have been well validated effective way capture both the feature and the sequential knowledge in given sample. In our model, the function shown below:
S. = L.TMMA,,
where S, the current state, which could also considered as the hidden state updated in each loop. A, is the node feature of the corresponding /-th action (i.e., word embedding) which is walked though by agent in the t-th move. Semantic understanding module keeps updating S, for each walk. We consider S, is effective enough to aggregate and preserve the entire path information.
3.5 POLICY NETWORK
The goal of the policy network is to guide the agent to find the most effective paths between aspect nodes and sentiment description in the obtained dependency graph. Specifically, designed to select the action based on the previous walk path and the next move candidates. Since the semantic understanding module effectively captures the knowledge of existing path, another crucial clue is the candidates nodes of the next walk.
However, there are two unique challenges for deploying conventional policy network in graph- structured data. First, the policy network should perceive all possible candidates for making the most effective action. While, general deep network structure requires consistent data format as in- put (e.g., images and videos with consistent resolution). In G.,, each node could have different
4

---

Under review as a conference paper at ICLR 2021
number of neighbor (1-hop) nodes associated with various connection categories. Thus, it is chal- lenging to define an action space which covers all the candidate paths in the same output. Second, the policy network should be permutation irrelevant the candidate input, which means the action of the agent would not be affected by the order of the candidates.
To this end, proposed structural input mechanism to address these challenges. When the agent has arrived at a node, obtains all the connected nodes (including the previously walked nodes and itself). We assume the set of the nodes are V = {vi, U2, ... Um where m is the number of all candidate nodes around i-th nodes. We concatenate current state St with each node 8;, i
{1, 2,...... ..., m) and obtain vector. We put the vector into the policy network to get a candidate score of the node The expression is formulated as shown below:
(3) where cat(-) indicates the concatenation operation. This strategy considers the probability of each candidates separately and it relaxes the input inconsistency challenge while still preserves the local structural information of the dependency graph. When the scores of all candidates are obtained, the agent would go to the node which has the highest score. The function is shown below:
8, pollccycct u:), i= (1,2,...,,,
a = arg max (L...mm. (8t).
(4)
In general sentiment analysis tasks, the aspect is usually single word while there few aspects containing multiple words (e.g., "HDMI port"). In this scenario, we merge all the edges of the corresponding nodes together and consider the aspect as a single node, consistent walk length is set to stop the walk process. In our experiments, we set walk length of which is effective enough for achieving high performance. If the distances are not equal 3, we observed that if the distances are shorter, the repetitive path exists around the final aspect (e.g.. walk through "good" to "good") Even if the consistent lenk-l not enough, the agent still tries to explore the most likely path for accurate prediction. More details are discussed in Appendix C
3.6 SENTIMENT CLASSIFIER
After the agent finishes path, a sentiment classifier is deployed to obtain the final sentiment pre- diction. After the last move, we consider S, is able provide the comprehensive information of the entire path. To this end, we directly forward S. into a sentiment classifier. The expression of the sentiment classifier is shown below:
p= SW 6p).
(5)
where 5-) is the non-linear activation. In our inplementation, deploy Softmax activation to predict one polarity from the candidate pool. Wp is the weights and by is the bias.
3.7 REWARD
The reward provides the information which directly guides the training procedure. Conventional RL frameworks have a clear reward design, which lets the policy network performs well in each stage. However, in our framework, the ultimate goal is high classification performance which relays on multiple modules (i.e., Semantic Understanding Module, Policy Network, and Sentiment Classifier). To this end, we directly consider the correct prediction as the reward for all the modules. We deploy Mean Squared Error (MSE) as the accuracy reward and the function is shown below:
Tace = - pll
where Face is the accuracy reward, is the ground truth label. Face would only be calculated once after the walk is finished. a straightforward yet effective reward. In the training procedure, all the three modules are simultaneously optimized to achieve the best performance. Other evaluation metrics (e.g., Cross Entropy) could also deployed. We empirically evaluate different metrics, and found most metrics are effective while MSE achieves the best performance.
3.8 IMPLEMENTATION & OPTIMIZATION
In our implementation, we use a 2-layer fully-connected (FC) neural network to parameterize the policy network that maps the state vector S, probability distribution over all possible actions.
5

---

Under review as a conference paper at ICLR 2021
ReLU activation is deployed in the first layer while activation in the second layer. However, to obtain a smooth score across different candidates, the sigmoid function is deployed across all actions. The goal of this step is to make the optimization procedure more smooth and robust.
In the optimization procedure. some strategies are used to improve its efficiency and effectiveness. For the policy network, simply choosing the candidate with the highest score is non-differentiable sampling strategy making it infeasible to calculate the weight gradients. Gumbel Softmax Jang et al. (0117) employs continuous distribution we can have a one-hot vector as the output of the agent while still preserving the parameter gradients. Meanwhile, Gumbel Softmax also fully explores the distribution of candidate scores and add more diversity to the training process. The core functions of Gumbel Softmax is shown as follows:
approximate a non-differentiable sample. In this way,
exp(( + g.)T)
23
"J=1 exppp 9,)/T)
where Ii are i.i.d samples drawn from Gumbel (0, 1) distribution, म; are the class probabilities. 2; are the generated samples, т is the temperature parameter. The smaller the т, the final result is more close to a one-hot vector. the training process our implementation, we directly apply Gumbel Softmax in PyTorch. The output would be one-hot vector, while the gradients are calculated from the Gumbel Softmax. For the testing process, since we do not need calculate the gradients and add diversity, simply use ar argmaz(-) function to get the one-hot vector as the output.
4 EXPERIMENTS
4.1 DATASETS
We consider five public benchmark datasets on aspect-based sentiment analysis Yooo & Gretzel 2008). Table [ illustrates their basic information, including label distribution and training-testing sample split settings. brief description of the datasets is as follows.
Table Statistics the evaluation datasets
Dataset Split
Lapl4
Twitter
Rest14
Rest15
Restl6
Train
Test
Train
Test
Train
Test
Train
Test
Train
Test
Positive
994
341
1561
173
2164
728
912
326
1240
469
Neutral
464
169
3127
346
637
196
36
34
69
30
Negative
870
128
1560
173
807
196
256
182
LAP14 (Pontiki al., 2014) includes review/descriptions of various laptops from users, where human annotators tagged both aspects and their polarities.
TWITTER (Dong et al. 2014) contains mentions in tweets on a diverse types of aspects, includ- ing celebrities (e.g., Bill Gates), products (e.g., xBox), and companies (e.g., Google), where their sentiment labels are manually annotated.
REST14 (Pontiki et al. larities annotated.
2014) focuses on restaurants' reviews, where aspect terms and their po-
2015 includes reviews for laptops, restaurants, and hotels, where anno- tators identify opinions expressed towards specific entities and their attributes.
RESTI5 (Pontiki et al.
REST16 (Pontiki et 2016) also includes reviews on restaurants, where each review may con-
tain multiple aspects.
We follow the commonly adopted experimental setting described in (Tang et al. 2016c). One sample will be removed it is associated with conflicting polarities or contains unclear aspects. For fair comparisons, identical word representations adopted in SentRL and baseline methods: A 300- dimensional pretrained GloVe model (Pennington et al.) 2014) is further fine-tuned by a bidirectional LSTM (Zhang al. 2019). The dimensionality of the hidden state in SentRL is set 300 in all the cases. ADAM optimizer (Kingma & Ba 2014) is used to train SentRL with a learning rate of 0.0005.

---

Under review as a conference paper at ICLR 2021
4.2 BASELINES
We consider the following state-of-the-art approaches as baselines in this study, While a more detailed description included Appendix A a brief overview is as follows.
SVM. Kiritchenko et al. (2014) deploys in-house sequence tagger to detect aspect terms with SVMs serving as the classifiers, where input texts are modeled naturally-ordered word quences.
LSTM Tang et al. (2016a) extends conventional LSTM (Hochreiter & Schmidhuber, 1997 to predict sentiment polarity, where input texts modeled naturally-ordered word sequences.
MemNet. Tang al. (2016c) uses external memories and multi-hop architecture learn the importance of context words for aspect-sentiment representations, where input texts are modeled
naturally-ordered word sequences.
AOA. Huang al. (0188) leverages the idea of citeetiion-vver-tteene for sentiment classifica- tion, where input texts are modeled as naturally-ordered word sequences.
IAN. Ma et al.
2017b) employs attention networks to learn contextual representations for aspects, are modeled as naturally-ordered word sequences.
where imput
TNet-LF. Li et al. (0018) develops target-specific transformation to obtain context information
of aspects, where input texts are modeled as aaturally-orderdd word sequences.
ASGCN, Zhang et al. 2019) considers the contextual information by utilizing dependency graphs, and GCN and an attention mechanism used learn sentimental representations. Note that input texts are modeled as graphs of words with syntactic structures.
We employ the same evaluation metrics discussed in (Zhang et al. 2019). In particular, evaluation results are obtained by averaging 3 runs with the random initialization, where accuracy (ACC) and macro-F1 (F1) are adopted the metrics. In our implementation. dependency graphs are obtained from spcCy while SentRL is compatible with various syntactic parsers.
Table 2: Sentiment classification performance over public benchmark datasets
Lap14
Rest
Rest15
Rest16
Twitter
Methods
ACC
F1
ACC
Fl
ACC
ACC
ACC
F1
SVM (Kiritchenko et al
2014
70.49
N/A
80.16
N/A
NIA
N/A
N/A
N/A
63.40
63.30
LSTM [Tang [ et al. 201ba
69.28
63.09
78.13
67.47
77.31
55.17
86.80
69.56
67.70
MemNet
Tang
70.64
65.17
79.61
69.64
77.31
85.44
65.99
71.48
69.90
AOA Huang al.
72.62
67.52
79.97
70.42
78.17
57.02
87.50
66.21
72.30
70.20
IAN Ma
72.05
67.38
79.26
70.09
78.54
52.65
84.74
55.21
72.50
70.81
TNet-LF [L
2018
74.61
70.14
80.42
71.03
78.47
59.47
89.07
70.43
72.98
71.43
ASCNN
Zhang et al.
2019
72.62
66.72
81.73
73.10
78.47
58.90
87.39
64.56
71.05
69.45
ASGCN- DI
Zhang el al.
2019
74.14
69.24
80.86
72.19
79.34
60.78
88.69
66.64
71.53
69.68
ASGCN-DE DG
Zhang et al.
2019
75.55
71.05
80.77
72.02
79.89
61.89
88,99
67.48
72.15
70.40
SentRI.
75.55
71.48
82.05
73.40
80.07
65.54
87.82
71.17
73.12
71.64
4.3 PERFORMANCE ANALYSIS
The overview of classification performance reported in Table We observe that SentRI. outper- forms all the baseline methods in 4 out of 5 cases in terms of accuracy (ACC). In terms of macro-F1, SentRL outperforms in the cases, with up to 3.7% improvement over the best baseline method.
In addition, to further confirm the effectiveness of SentRL, we visualize the learned sentimental representations and compare with the ones generated from the second best model, ASGCN (Zhang et al. 2019) by t-SNE method (Maaten & Hinton 2008). As shown in Figure red, green, and blue dots denote "negative", "neutral". and "positive" aspect-based sentiment, respectively. We observe that approach could better separate different sentiment. Especially for "neutral" and "negative" ones, our approach could better differentiate them.
https://spccyiio

---

Under review as a conference paper at ICLR 2021
30
20
10
0
-10
-30
-30
400
-20
-10
0
10
20
30
(b)
Figure 3: I-SNE visualization of the learned sentimental representations, where (a) is AS- GCN (Zhang et al. timental categories.
2019) method and (b) is our approach. Different color denotes different sen-
4.4
CASE STUDY
acomp
acomp
conj
xcomp aux
nsubi
advmmdd
Case
Food NOUN
is
a always
fresh
and
hot- ADJ
ready
to PART
eat VERB
AUX
ADV
ADJ
CCONJ
ADJ
nsubj
acamp
Duk
dct napdvmod advmod
Case
The
staff NOUN
should VERB
be
a
bit
more ADV
friendly
DET
AUX
DET
NOUN
ADI
Figure 4:
case study demonstrating traces from aspect words to their related sentimental words: Case is sample correctly predicted by SentRL, while case 2 is an incorrect prediction. Grey edges denote extracted syntactic dependencies, and ones denote traces generated by a learned agent. For example, in case 1, the agent trace is Food-Fioo-iis///ii
We present a case study and visualize traces from aspect words to their related sentimental words dis- covered by SentRL. One correct (case 1) and one incorrect prediction (case 2) are shown in Figure where words blue boxes are aspects, grey edges denote syntactic dependencies between words, and red edges traces left by learned agent. In case 1, the agent starts from the aspect word food and ends at the word fresh, collecting necessary information within three hops as expected. 2, notice the agent has hard time to make correct prediction. Ideally, starting from the aspect word staff. within three hops, a well-learned agent may collect words such as friendly and more, which may provide minimal evidence that lead the agent to correct prediction. However, in investigation, we find there are no similar statements (e.g., should be a bit more) that expresses such negative sentiment in the training data. To this end, without accessing commonsense knowledge, could still be difficult for SentRL to make the correct prediction in case 2.
case
CONCLUSION
In this paper, we propose SentRL, novel aspect-based sentiment classification approach via re- inforcement learning. By investigating input texts from their syntactic structures (e.g. dependency graphs), we are able to effectively reduce input variance introduced by diverse expressions natural languages. On top of the syntactic structures, an agent is deployed to discover the most effective paths that link aspects with their sentimental descriptions. By collecting evidences from the dis- covered paths, semantic understanding module in SentRL learns make the accurate sentimental classification. All the modules in SentRL are simultancously trained in an end-to-end fashion. Ex- tensive experiments and case studies demonstrate the effectiveness of method.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. PrXiv:/409.0477 2014.
Xuefeng Bai, Pengbo Liu, and Yue Zhang. Exploiting typed syntactic dependencies for targeted sentiment classification using graph attention neural network. arXiv preprint arXiv:2002.09885, 2020.
Michael A Covington. A fundamental algorithm for dependency parsing. In Proceedings of the Annual ACM southeast conference, pp. 95-102. Citeseer, 2001.
Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. Adaptive recursive neu- ral network for target-dependent twitter sentiment classification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 19-54, 2014.
Zhengjie Gao, Ao Feng, Xinyu Song, and Xi Wu. Target-dependent sentiment classification with bert. IEEE Access, 7:154290-154299, 2019.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp-
154-164, 2019.
He, Mari Ostendorf, Xiaodong He, Jianshu Chen, Jianfeng Gao, Lihong Li, and Li Deng, Deep reinforcement learning with combinatorial action space for predicting popular Reddit threads. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp-
1838-1848, 2016.
Ruidan He, Wee Sun Lee, Hwee Tou Ng. and Daniel Dahlmeier. Effective attention modeling for aspect-level sentiment classification. In Proceedings of the International Conference Compu- tational Linguistics, pp. 1121-1131, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Binxuan Huang and Kathleen Carley. Syntax- aspect level sentiment classification with graph attention networks. Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 5468-5476, 2019.
Binxuan Huang, Yanglan Ou, and Kathleen M Carley. Aspect level sentiment classification with attention-over-attetioon neural networks. In Proceedings of the International Conference on So- cial Computing, Cilauvioral-Cultrree Modeling and Prediction and Behavior Representation in Modeling and Simulation, pp. 197-206. Springer, 2018
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with SombleeSofftma. In Proceedings of the International Conference on Learning Representations. OpenReview. net, 2017.
Diederik Kingma and Jimmy Ba. ADAM: A method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.6600 1412.6980, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 16tp::0900000 2016.
Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif Mohammad. NRC-Canada-2011 De- tecting aspects and sentiment in customer reviews. In Proceedings of the International Workshop on Semantic Evaluation, pp. 437-442, 2014.
Xin Li, Lidong Bing, Wai Lam, and Bei Shi. Transformation networks for target-oriented sentiment classification. In Proceedings of the Association for Computational Linguistics, pp. 946-956, 2018.

---

Under review as a conference paper at ICLR 2021
Hu Linmei, Tianchi Yang, Chuan Shi, Houye Ji, and Xiaoli Li. Heterogeneous graph attention networks for semi-supervised short text classification. In Proceedings of the Conference on Em- pirical Methods in Natural Language Processing, pp. 4823 4832, 2019.
Bing Liu, Gokhan Tür, Dilek Hakkani- Tür,, Pararth Shah, and Larry Heck. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Pro- ceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2060-2069 2018.
Minh- Mhan-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. arXiv: 1500.04058. 2015.
Dehong Ma, Sujian Li. Xiaodong Zhang, and Houfeng Wang. Interactive attention networks for aspect-level sentiment classification. In Proceedings of the International Joint Conference Artificial Intelligence, pp. 4068 4074, 2017a.
Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. Interactive attention networks for aspect-level sentiment classification. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 4068 4074. AAAI Press, 2017b.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov): 2008
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. In Proceedings of the Conference Empirical Meth- ods in Natural Language Processing, pp. 1-11, 2015a.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing, pp. 1-11, 2015b.
Ramakanth Pasunuru and Mohit Bansal. Reinforced video captioning with entailment rewards. In Proceedings of the Conference Empirical Methods in Natural Language Processing, pp. 979- 985, 2017.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1532- 1523, 2014.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the ACL International Workshop on Semantic Evaluation, pp. 27-35, 2014.
Maria Pontiki, Dimitrios Galanis, Harris Papageorgiou, Suresh Manandhar, and lon Androutsopou- los. Semeval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation, pp. 486 495, 2015.
Maria Pontiki, Dimitrios Galanis, Haris Papageorgiou, lon Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. Semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the International Work- shop on Semantic Evaluation, 2016.
Pengda Qin, Weiran Xu, and William Yang Wang. Robust distant supervision relation extraction via deep reinforcement learning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 2137-2147, 2018.
Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. Effective LSTMs for target-dependent senti- ment classification. In Proceedings of the COLING International Conference on Computational Linguistics, pp. 3298-3307, 2016a.
Duyu Tang, Bing Qin, and Ting Liu. Aspect level sentiment classification with deep memory net- work. arXiv preprint arXiv: 160ivv1605.006 1605.689000 2016b.
10

---

Under review as a conference paper at ICLR 2021
Duyu Tang, Bing Qin, and Ting Liu. Aspect level sentiment classification with deep memory net- work. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 214-224, 2016c.
Hao Tang, Donghong Ji, Chenliang Li, and Qiji Zhou. Dependency graph enhanced dual- transformer structure for aspect-based sentiment classification. In Proceedings of the Annual Meeting of the Association for Computational Linguisties, pp. 6578-6588, 2020.
Tun Thura Thet, Jin-Cheon Na, and Christopher S.G. Khoo. Aspect-based sentiment analysis of movie reviews on discussion boards. Journal of Information Science, 36(6):823-848, 2010.
Duy-Tin Vo and Yue Zhang. Target-dependent twitter sentiment classification with rich automatic features. In Proceedings of the International Joint Conference on Artificial Intelligence, 2015.
Jingjing Wang. Changlong Sun, Shoushan Li, Jiancheng Wang, Luo Si, Min Zhang, Xiaozhong Liu, and Guodong Zhou. Human-like decision making: Document-level aspect sentiment classifica- tion via hicrarchical reinforcement learning. In Proceedings of the Empirical Methods in Natural Language Processing, 2019.
William Yang Wang, Jiwei Li, and Xiaodong He. Deep reinforcement learning for NLP. In Pro- ceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 19-21, 2018.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the Conference Empirical Methods in Natural Language Processing, pp. 606-615, 2016.
Jiawei Wu, Lei Li, and William Yang Wang. Reinforced Co- Training. In NAACL-HLT, 2018.
Wenhan Xiong, Thien Hoang, and William Yang Wang. DeepPath: A reinforcement learning method for knowledge graph reasoning. In Proceedings of the 2017 Conference on Empirical Methodd in Natural Language Processing, pp. 564 573, 2017.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, pp. 2048-2057, 2015.
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7370-7377, 2019.
Qingyu Yin, Yu Zhang, Wei-Nan Zhang, Ting Liu, and William Yang Wang. Deep reinforcement learning for Chinese zero pronoun resolution. In Proceedings of the Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 569-578, 2018.
Kyung Hyan Yoo and Ulrike Gretzel. What motivates consumers write online travel reviews? Information Technology & Tourism, 10(4):283-29) 295, 2008
Chen Zhang, Qiuchi Li, and Dawei Song. Aspect-based sentiment classification with aspect-specific graph convolutional networks. In Proceedings of the Conference on Empirical Methods Natural Language Processing, pp. 4567-4577, 2019.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 2205-2215, Brussels, Belgium, 2018. ACL.
Xianda Zhou and William Yang Wang. MojiTalk: Generating emotional responses at scale. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 1128- 1137, 2018
11

---

Under review as a conference paper at ICLR 2021
A BASELINES
We compare our model with several baselines methods and state-of-the the-art models for a fair and comprchensive comparison. The detailed baseline methods are listed below:
SVM (Kiritchenko al. 2014) deploys traditional feature extraction methods and uses in-house sequence tagger to detect aspect terms. Supervised classifiers are used to do the classification.
LSTT (Tagg et 2016a) 6a) proposes an extension on conventional LSTTM (Hochreiter & Schmiidhuber 1997 model to predict sentiment polarity. It incorporates target information to fully explore the relevant context words.
MemNet (Tang et al. 2016c) uses external memories and a multi-hop architecture to learn the importance of the context words and further help learning the text representation.
AOA (Huang al. 2018 introduces affention-overraattetioon (AOA) in the machine trans- lation field to the sentiment classification. It takes both text-to-aspect and aspect-to-text into consideration.
IAN (Ma al., (017b) applies the attention network both aspects and context learn the representations interactively. The concatenation of text and context feature is both used to obtain the sentimental classification results.
TNet-LF (Li et al. 2018) introduces a Parget-Specific Transformation (TST) component to get the context information of the aspect text. Then it proposes Context-Preserrinn Trans- formation (CPT) layers to further learn the abstract feature which is useful for classification tasks.
ASGCN Zhhag al. 2019) considers the contextual information by utilizing the depen- dency graph as prior knowledge. A GCN associated with an attention mechanism frame- work used extract most related sentimental representations.
B PARAMETER SENSITIVITY
Walk length of the agent is the crucial parameter in our model. In this experiment, we tested the model performance with different walk length (i.c., from 2 to 5) and the result shown in Figure We observed that our model achieves the best performance when walk length 3. We assume that walk length is less than the agent cannot reach the sentiment word and the performance decreases significantly. Meanwhile, if walk length is more than 3, the performance only have slight decline, and we conjecture this due the unnecessary walks which aggregate extra information. The result illustrates the requirement of long enough walk length of our approach, and it also should parameter insensitivity when walk length achieves the threshold.
83
82
Accuracy (96)
81
(96)
72
80
70
79
78
3
4
2
5
Walk Length
Walk Length
Figure 5: Classification performance with different walk length.
12

---

Under review as a conference paper at ICLR 2021
Table 3: Sentiment classification performance based random walk
Lap14
Rest
Rest
Rest16
Twitter
Walk Length
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F]
65.88
60.05
74.37
61.51
74.11
51.59
84.30
61.13
67.34
63.86
2
67.47
63.28
75.44
64.82
74.76
53.15
84.44
63.29
67.34
65.17
3
70.42
65.10
76.03
65.89
74.91
53.85
84.98
64.12
67.93
66.12
4
70.51
65.43
76.15
66.28
75.61
54.04
85.15
65.00
68.14
66.82
Ours
75.55
71.48
82.05
73.40
80.07
65.54
87.82
71.17
73.12
71.64
C
MODEL ANALYSIS
To further analysis the effectiveness of our reinforcement learning module, we intentionally let the agent randomly walk though the dependency graph while keeping the other modules unchanged, and the result is illustrated in Table 3 where the walk length is set from to 4. From Table 3 we observed that when walk length is the performance significantly drops. However, it still higher than random guess, When the walk length increases, the performance cannot increase consistently. From the results, we could conclude that 1) our reinforcement learning approach effective and es- sential for obtaining high performance, and 2) the aspect node does have bias for different sentiment polarity, and walking through the dependency graph could more effectively capture the syntactical knowledge and improve the performance. However, compare with the specifically trained agent based on reinforcement learning, not enough for getting high accuracy predictions.
13


Reviews:
[{"review_id": "zDXzKasOQhC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper addresses aspect-based sentiment analysis by running reinforcement learning on the dependency parse of input sentences. The agent learns a policy network to select the most effective walk along edges in the dependency graphs, starting from the target aspect in the input sentences. The state representation is learned with an LSTM. At the end of the path, a sentiment classifier predicts the distribution of the polarity. The reward is the mean squared error between the class label (i.e. sentiment polarity) and the probability predicted by the model. The paper claims that by limiting the agent's budget, the approach forces the agent to discard irrelevant information and focus on the effective paths, enabling the approach to perform well with a small number of training examples.\n\nStrengths: \n- The intuition behind the approach of an agent learning dependency paths that can be used for sentiment analysis is appealing. The paper evaluates the performance on 5 data sets, and compares against baselines such as MemNet, TNet-Lf, and ASGCN. Among the baselines, ASGCN also leverage on dependency parses of the input sentences.\n\nWeaknesses: \n- The paper failed to compare against a number of publications that also uses dependency parses as input to improve aspect based sentiment analysis, e.g., [1], [2], [3], [4] and [5]. [1] and [2] were referenced in the paper, but their results were not included in the comparison. The results in these 5 papers seem considerably better than in the current submission: e.g., for laptops, a number of these papers obtain over 74% F1 compared to the 71.48% in this submission. \n\n- It seems that there is a big advantage in using dependency parses for sentiment analysis (e.g., [1,2,3,4,5]), so it is unclear how much of the good performance in the current submission is actually from reinforcement learning. While the intuition behind finding dependency paths is appealing,  the ablation that tries to show this is limited to a few examples in the paper. It would be interesting to have a baseline that uses the dependency parse but not reinforcement learning. They could also try and compile some statistics of the relations on the most selected dependency paths, or evaluate a small sample of selected paths with human evaluators.\n\n[1] Syntax-Aware Aspect Level Sentiment Classiﬁcation with Graph Attention Networks\nBinxuan Huang and Kathleen M. Carley. EMNLP 2019\n\n[2] Exploiting Typed Syntactic Dependencies for Targeted Sentiment Classiﬁcation Using Graph Attention Neural Network. Xuefeng Bai, Pengbo Liu, and Yue Zhang. ArXiv, submitted on 22nd Feb 2020\n\n[3] Relational graph attention network for aspect-based sentiment analysis. Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. ACL 2020.\n\n[4] Dependency graph enhanced dual transformer structure for aspect-based sentiment classification. Hao Tang, Donghong Ji, Chenliang Li, and Qiji Zhou. ACL 2020.\n\n[5] Modelling context and syntactical features for aspect based sentiment analysis. Minh Hieu Phan and Philip O. Ogunbona. ACL 2020.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "eqk8D-1TlhJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to aspect-based sentiment classification, which is the task of identifying the sentiment of a specific phrase or entity in a sentence. The paper proposes to do this by first generating a dependency parse of the entire sentence and then using an RL agent to walk the dependency tree starting from the word or phrase to be classified. The state for the RL agent is the LSTM representation of its history. The final state is then used to classify the sentiment of the aspect.\nThe approach is evaluated on multiple benchmark datasets for the task and compared with many previously proposed approaches. Most of the compared methods don’t make use of the dependency graphs but the authors also compare with one recent graph convolution-based approach that also makes use of the dependency graph. They show improvements over these methods on most of the benchmarks.\n\nStrong Points\n- The approach makes use of dependency graph information in an interesting way to learn to find paths that would lead to better aspect-based sentiment classification.\n- The proposed method is well explained and easy to follow.\n- The results show improvements on many datasets.\n\nWeak Points\n- While the paper compares with many baselines, it doesn’t compare with some highly related work [1,2] which uses Transformer text encoders. Ideally, the paper should discuss these and potentially directly compare with these.\n- Due to the use of RL, the proposed approach will be more computationally expensive. It will be good to also discuss this and contrast with other methods.\n\nQuestions for authors:\n- How do you obtain the dependency tree for the sentences? Have you analyzed how errors in dependency parsing affect performance?\n- Can you comment on the computation time for the proposed and competing approaches?\n- Can you compare and contrast your approach to [1] and [2]? It would be useful to see if using BERT token embeddings as input embeddings for the proposed method still provides competitive performance as these approaches[1,2] report better performance than the proposed approach.\n- Most of the datasets used in the experiments are very small. Given the high sample complexity requirement of RL methods, it is a bit surprising that the RL-based approach works at all for this problem. Can you explain if you found this to be an issue in your experiments?\n\nSuggestions:\n- It will be good to also mention the word representations and the text encoder (LSTM or something else) that is used in the compared baselines.\n- For path length comparisons in Appendix, please add a pointer to it in the experiments section. It will also be good to see the performance at path length 1 (which wouldn’t require any agent walk).\n- Many citations are not properly formatted. Instead of “By leveraging textual context and word-level attention learned from deep models Vo & Zhang (2015); Dong et al. (2014)”, it should be cited as “By leveraging textual context and word-level attention learned from deep models (Vo & Zhang, 2015; Dong et al., 2014)”\n\n[1] Target-Dependent Sentiment Classification With BERT https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8864964\n\n[2] Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification https://www.aclweb.org/anthology/2020.acl-main.588.pdf\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OHRCUKnCFGA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a reinforcement-learning-based model for aspect-based sentiment analysis. The RL model trains an agent that tries to walk through the most effective path from the aspect target to determine the final sentiment towards this target. \n\nIn general, the RL model based on the dependency structure could be deemed as a pioneer implementation on aspect-based sentiment analysis. And this approach is able to remove the negative effect brought by irrelevant context. However, there are a few limitations that need to be addressed:\n1. The writing needs to be improved. There are many grammatical mistakes that affect the readability, e.g., \"To effectively contain the impact from task-irrelevant information...\" from the last paragraph in page 1: the misusing of word \"contain\". \"The goal of aspect-based sentiment classification is to predict sentiment polarities (i.e., “positive”, “neutral”, and “negative”) for each given aspect.\": \"sentiment polarities\"->\"sentiment polarity\".\n2. The proposed model is missing many details. For example, when will the policy terminate? It seems from (3) and (4) there could be infinite loop of candidate actions. And what is exactly the policy network in (3)?\n3. The RL model in this case is very limited in terms of exploration. From the construction, the candidate actions only contain connected nodes at each timestamp. This may miss important information, e.g., the second aspect in figure 1 could not traverse to \"do not\". How do you solve this issue? \n4. I am not convinced by the statement that the proposed model is more generalizable compared to baseline models. Where does the generalization ability come from? And I conjecture RL needs even more training data to perform well.\n5. As mentioned in the main text, the reward function (6) could be any form, but no experiments are conducted on other forms. The experimental result in Table 2 misses other recent baselines, e.g., \"Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification\" that actually outperforms this method.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed a reinforcement-learning-based model for aspect-based sentiment analysis. As raised by the reviewers, 1) the writing needs to be improved: e.g., presenting the details of the proposed method clearly, citing the references properly, etc. 2) related methods need to be implemented for comparison, 3) the reported results are not SOTA compared with existing methods. Moreover, some technical claims are not convincing, which need to be stated more carefully.

In summary, based on its current shape, this paper is not ready to be published in ICLR.
Decision: Reject