Introduction
Aspect-Based Sentiment Analysis (ABSA) involves detecting opinion targets and locating opinion indicators in sentences in product review texts BIBREF0 . The first sub-task, called Aspect Term Extraction (ATE), is to identify the phrases targeted by opinion indicators in review sentences. For example, in the sentence “I love the operating system and preloaded software”, the words “operating system” and “preloaded software” should be extracted as aspect terms, and the sentiment on them is conveyed by the opinion word “love”. According to the task definition, for a term/phrase being regarded as an aspect, it should co-occur with some “opinion words” that indicate a sentiment polarity on it BIBREF1 .
Many researchers formulated ATE as a sequence labeling problem or a token-level classification problem. Traditional sequence models such as Conditional Random Fields (CRFs) BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , Long Short-Term Memory Networks (LSTMs) BIBREF6 and classification models such as Support Vector Machine (SVM) BIBREF7 have been applied to tackle the ATE task, and achieved reasonable performance. One drawback of these existing works is that they do not exploit the fact that, according to the task definition, aspect terms should co-occur with opinion-indicating words. Thus, the above methods tend to output false positives on those frequently used aspect terms in non-opinionated sentences, e.g., the word “restaurant” in “the restaurant was packed at first, so we waited for 20 minutes”, which should not be extracted because the sentence does not convey any opinion on it.
There are a few works that consider opinion terms when tackling the ATE task. BIBREF8 proposed Recursive Neural Conditional Random Fields (RNCRF) to explicitly extract aspects and opinions in a single framework. Aspect-opinion relation is modeled via joint extraction and dependency-based representation learning. One assumption of RNCRF is that dependency parsing will capture the relation between aspect terms and opinion words in the same sentence so that the joint extraction can benefit. Such assumption is usually valid for simple sentences, but rather fragile for some complicated structures, such as clauses and parenthesis. Moreover, RNCRF suffers from errors of dependency parsing because its network construction hinges on the dependency tree of inputs. CMLA BIBREF9 models aspect-opinion relation without using syntactic information. Instead, it enables the two tasks to share information via attention mechanism. For example, it exploits the global opinion information by directly computing the association score between the aspect prototype and individual opinion hidden representations and then performing weighted aggregation. However, such aggregation may introduce noise. To some extent, this drawback is inherited from the attention mechanism, as also observed in machine translation BIBREF10 and image captioning BIBREF11 .
To make better use of opinion information to assist aspect term extraction, we distill the opinion information of the whole input sentence into opinion summary, and such distillation is conditioned on a particular current token for aspect prediction. Then, the opinion summary is employed as part of features for the current aspect prediction. Taking the sentence “the restaurant is cute but not upscale” as an example, when our model performs the prediction for the word “restaurant”, it first generates an opinion summary of the entire sentence conditioned on “restaurant”. Due to the strong correlation between “restaurant' and “upscale” (an opinion word), the opinion summary will convey more information of “upscale” so that it will help predict “restaurant” as an aspect with high probability. Note that the opinion summary is built on the initial opinion features coming from an auxiliary opinion detection task, and such initial features already distinguish opinion words to some extent. Moreover, we propose a novel transformation network that helps strengthen the favorable correlations, e.g. between “restaurant' and “upscale”, so that the produced opinion summary involves less noise.
Besides the opinion summary, another useful clue we explore is the aspect prediction history due to the inspiration of two observations: (1) In sequential labeling, the predictions at the previous time steps are useful clues for reducing the error space of the current prediction. For example, in the B-I-O tagging (refer to Section SECREF4 ), if the previous prediction is “O”, then the current prediction cannot be “I”; (2) It is observed that some sentences contain multiple aspect terms. For example, “Apple is unmatched in product quality, aesthetics, craftmanship, and customer service” has a coordinate structure of aspects. Under this structure, the previously predicted commonly-used aspect terms (e.g., “product quality”) can guide the model to find the infrequent aspect terms (e.g., “craftmanship”). To capture the above clues, our model distills the information of the previous aspect detection for making a better prediction on the current state.
Concretely, we propose a framework for more accurate aspect term extraction by exploiting the opinion summary and the aspect detection history. Firstly, we employ two standard Long-Short Term Memory Networks (LSTMs) for building the initial aspect and opinion representations recording the sequential information. To encode the historical information into the initial aspect representations at each time step, we propose truncated history attention to distill useful features from the most recent aspect predictions and generate the history-aware aspect representations. We also design a selective transformation network to obtain the opinion summary at each time step. Specifically, we apply the aspect information to transform the initial opinion representations and apply attention over the transformed representations to generate the opinion summary. Experimental results show that our framework can outperform state-of-the-art methods.
The ATE Task
Given a sequence INLINEFORM0 of INLINEFORM1 words, the ATE task can be formulated as a token/word level sequence labeling problem to predict an aspect label sequence INLINEFORM2 , where each INLINEFORM3 comes from a finite label set INLINEFORM4 which describes the possible aspect labels. As shown in the example below:
INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 denote beginning of, inside and outside of the aspect span respectively. Note that in commonly-used datasets such as BIBREF12 , the gold standard opinions are usually not annotated.
Model Description
As shown in Figure FIGREF3 , our model contains two key components, namely Truncated History-Attention (THA) and Selective Transformation Network (STN), for capturing aspect detection history and opinion summary respectively. THA and STN are built on two LSTMs that generate the initial word representations for the primary ATE task and the auxiliary opinion detection task respectively. THA is designed to integrate the information of aspect detection history into the current aspect feature to generate a new history-aware aspect representation. STN first calculates a new opinion representation conditioned on the current aspect candidate. Then, we employ a bi-linear attention network to calculate the opinion summary as the weighted sum of the new opinion representations, according to their associations with the current aspect representation. Finally, the history-aware aspect representation and the opinion summary are concatenated as features for aspect prediction of the current time step.
As Recurrent Neural Networks can record the sequential information BIBREF13 , we employ two vanilla LSTMs to build the initial token-level contextualized representations for sequence labeling of the ATE task and the auxiliary opinion word detection task respectively. For simplicity, let INLINEFORM0 denote an LSTM unit where INLINEFORM1 is the task indicator. In the following sections, without specification, the symbols with superscript INLINEFORM2 and INLINEFORM3 are the notations used in the ATE task and the opinion detection task respectively. We use Bi-Directional LSTM to generate the initial token-level representations INLINEFORM4 ( INLINEFORM5 is the dimension of hidden states): DISPLAYFORM0
In principle, RNN can memorize the entire history of the predictions BIBREF13 , but there is no mechanism to exploit the relation between previous predictions and the current prediction. As discussed above, such relation could be useful because of two reasons: (1) reducing the model's error space in predicting the current label by considering the definition of B-I-O schema, (2) improving the prediction accuracy for multiple aspects in one coordinate structure.
We propose a Truncated History-Attention (THA) component (the THA block in Figure FIGREF3 ) to explicitly model the aspect-aspect relation. Specifically, THA caches the most recent INLINEFORM0 hidden states. At the current prediction time step INLINEFORM1 , THA calculates the normalized importance score INLINEFORM2 of each cached state INLINEFORM3 ( INLINEFORM4 ) as follows: DISPLAYFORM0
DISPLAYFORM0
INLINEFORM0 denotes the previous history-aware aspect representation (refer to Eq. EQREF12 ). INLINEFORM1 can be learned during training. INLINEFORM2 are parameters associated with previous aspect representations, current aspect representation and previous history-aware aspect representations respectively. Then, the aspect history INLINEFORM3 is obtained as follows: DISPLAYFORM0
To benefit from the previous aspect detection, we consolidate the hidden aspect representation with the distilled aspect history to generate features for the current prediction. Specifically, we adopt a way similar to the residual block BIBREF14 , which is shown to be useful in refining word-level features in Machine Translation BIBREF15 and Part-Of-Speech tagging BIBREF16 , to calculate the history-aware aspect representations INLINEFORM0 at the time step INLINEFORM1 : DISPLAYFORM0
where ReLU is the relu activation function.
Previous works show that modeling aspect-opinion association is helpful to improve the accuracy of ATE, as exemplified in employing attention mechanism for calculating the opinion information BIBREF9 , BIBREF17 . MIN BIBREF17 focuses on a few surrounding opinion representations and computes their importance scores according to the proximity and the opinion salience derived from a given opinion lexicon. However, it is unable to capture the long-range association between aspects and opinions. Besides, the association is not strong because only the distance information is modeled. Although CMLA BIBREF9 can exploit global opinion information for aspect extraction, it may suffer from the noise brought in by attention-based feature aggregation. Taking the aspect term “fish” in “Furthermore, while the fish is unquestionably fresh, rolls tend to be inexplicably bland.” as an example, it might be enough to tell “fish” is an aspect given the appearance of the strongly related opinion “fresh”. However, CMLA employs conventional attention and does not have a mechanism to suppress the noise caused by other terms such as “rolls”. Dependency parsing seems to be a good solution for finding the most related opinion and indeed it was utilized in BIBREF8 , but the parser is prone to generating mistakes when processing the informal online reviews, as discussed in BIBREF17 .
To make use of opinion information and suppress the possible noise, we propose a novel Selective Transformation Network (STN) (the STN block in Figure FIGREF3 ), and insert it before attending to global opinion features so that more important features with respect to a given aspect candidate will be highlighted. Specifically, STN first calculates a new opinion representation INLINEFORM0 given the current aspect feature INLINEFORM1 as follows: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are parameters for history-aware aspect representations and opinion representations respectively. They map INLINEFORM2 and INLINEFORM3 to the same subspace. Here the aspect feature INLINEFORM4 acts as a “filter” to keep more important opinion features. Equation EQREF14 also introduces a residual block to obtain a better opinion representation INLINEFORM5 , which is conditioned on the current aspect feature INLINEFORM6 .
For distilling the global opinion summary, we introduce a bi-linear term to calculate the association score between INLINEFORM0 and each INLINEFORM1 : DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are parameters of the Bi-Linear Attention layer. The improved opinion summary INLINEFORM2 at the time INLINEFORM3 is obtained via the weighted sum of the opinion representations: DISPLAYFORM0
Finally, we concatenate the opinion summary INLINEFORM0 and the history-aware aspect representation INLINEFORM1 and feed it into the top-most fully-connected (FC) layer for aspect prediction: DISPLAYFORM0 DISPLAYFORM1
Note that our framework actually performs a multi-task learning, i.e. predicting both aspects and opinions. We regard the initial token-level representations INLINEFORM0 as the features for opinion prediction: DISPLAYFORM0
INLINEFORM0 and INLINEFORM1 are parameters of the FC layers.
Joint Training
All the components in the proposed framework are differentiable. Thus, our framework can be efficiently trained with gradient methods. We use the token-level cross-entropy error between the predicted distribution INLINEFORM0 ( INLINEFORM1 ) and the gold distribution INLINEFORM2 as the loss function: DISPLAYFORM0
Then, the losses from both tasks are combined to form the training objective of the entire model: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 represent the loss functions for aspect and opinion extractions respectively.
Datasets
To evaluate the effectiveness of the proposed framework for the ATE task, we conduct experiments over four benchmark datasets from the SemEval ABSA challenge BIBREF1 , BIBREF18 , BIBREF12 . Table TABREF24 shows their statistics. INLINEFORM0 (SemEval 2014) contains reviews of the laptop domain and those of INLINEFORM1 (SemEval 2014), INLINEFORM2 (SemEval 2015) and INLINEFORM3 (SemEval 2016) are for the restaurant domain. In these datasets, aspect terms have been labeled by the task organizer.
Gold standard annotations for opinion words are not provided. Thus, we choose words with strong subjectivity from MPQA to provide the distant supervision BIBREF19 . To compare with the best SemEval systems and the current state-of-the-art methods, we use the standard train-test split in SemEval challenge as shown in Table TABREF24 .
Comparisons
We compare our framework with the following methods:
CRF-1: Conditional Random Fields with basic feature templates.
CRF-2: Conditional Random Fields with basic feature templates and word embeddings.
Semi-CRF: First-order Semi-Markov Conditional Random Fields BIBREF20 and the feature templates in BIBREF21 are adopted.
LSTM: Vanilla bi-directional LSTM with pre-trained word embeddings.
IHS_RD BIBREF2 , DLIREC BIBREF3 , EliXa BIBREF22 , NLANGP BIBREF4 : The winning systems in the ATE subtask in SemEval ABSA challenge BIBREF1 , BIBREF18 , BIBREF12 .
WDEmb BIBREF5 : Enhanced CRF with word embeddings, dependency path embeddings and linear context embeddings.
MIN BIBREF17 : MIN consists of three LSTMs. Two LSTMs are employed to model the memory interactions between ATE and opinion detection. The last one is a vanilla LSTM used to predict the subjectivity of the sentence as additional guidance.
RNCRF BIBREF8 : CRF with high-level representations learned from Dependency Tree based Recursive Neural Network.
CMLA BIBREF9 : CMLA is a multi-layer architecture where each layer consists of two coupled GRUs to model the relation between aspect terms and opinion words.
To clarify, our framework aims at extracting aspect terms where the opinion information is employed as auxiliary, while RNCRF and CMLA perform joint extraction of aspects and opinions. Nevertheless, the comparison between our framework and RNCRF/CMLA is still fair, because we do not use manually annotated opinions as used by RNCRF and CMLA, instead, we employ an existing opinion lexicon to provide weak opinion supervision.
Settings
We pre-processed each dataset by lowercasing all words and replace all punctuations with PUNCT. We use pre-trained GloVe 840B vectors BIBREF23 to initialize the word embeddings and the dimension (i.e., INLINEFORM0 ) is 300. For out-of-vocabulary words, we randomly sample their embeddings from the uniform distribution INLINEFORM1 as done in BIBREF24 . All of the weight matrices except those in LSTMs are initialized from the uniform distribution INLINEFORM2 . For the initialization of the matrices in LSTMs, we adopt Glorot Uniform strategy BIBREF25 . Besides, all biases are initialized as 0's.
The model is trained with SGD. We apply dropout over the ultimate aspect/opinion features and the input word embeddings of LSTMs. The dropout rates are empirically set as 0.5. With 5-fold cross-validation on the training data of INLINEFORM0 , other hyper-parameters are set as follows: INLINEFORM1 , INLINEFORM2 ; the number of cached historical aspect representations INLINEFORM3 is 5; the learning rate of SGD is 0.07.
Main Results
As shown in Table TABREF39 , the proposed framework consistently obtains the best scores on all of the four datasets. Compared with the winning systems of SemEval ABSA, our framework achieves 5.0%, 1.6%, 1.4%, 1.3% absolute gains on INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 respectively.
Our framework can outperform RNCRF, a state-of-the-art model based on dependency parsing, on all datasets. We also notice that RNCRF does not perform well on INLINEFORM0 and INLINEFORM1 (3.7% and 3.9% inferior than ours). We find that INLINEFORM2 and INLINEFORM3 contain many informal reviews, thus RNCRF's performance degradation is probably due to the errors from the dependency parser when processing such informal texts.
CMLA and MIN do not rely on dependency parsing, instead, they employ attention mechanism to distill opinion information to help aspect extraction. Our framework consistently performs better than them. The gains presumably come from two perspectives: (1) In our model, the opinion summary is exploited after performing the selective transformation conditioned on the current aspect features, thus the summary can to some extent avoid the noise due to directly applying conventional attention. (2) Our model can discover some uncommon aspects under the guidance of some commonly-used aspects in coordinate structures by the history attention.
CRF with basic feature template is not strong, therefore, we add CRF-2 as another baseline. As shown in Table TABREF39 , CRF-2 with word embeddings achieves much better results than CRF-1 on all datasets. WDEmb, which is also an enhanced CRF-based method using additional dependency context embeddings, obtains superior performances than CRF-2. Therefore, the above comparison shows that word embeddings are useful and the embeddings incorporating structure information can further improve the performance.
Ablation Study
To further investigate the efficacy of the key components in our framework, namely, THA and STN, we perform ablation study as shown in the second block of Table TABREF39 . The results show that each of THA and STN is helpful for improving the performance, and the contribution of STN is slightly larger than THA. “OURS w/o THA & STN” only keeps the basic bi-linear attention. Although it performs not bad, it is still less competitive compared with the strongest baseline (i.e., CMLA), suggesting that only using attention mechanism to distill opinion summary is not enough. After inserting the STN component before the bi-linear attention, i.e. “OURS w/o THA”, we get about 1% absolute gains on each dataset, and then the performance is comparable to CMLA. By adding THA, i.e. “OURS”, the performance is further improved, and all state-of-the-art methods are surpassed.
Attention Visualization and Case Study
In Figure FIGREF41 , we visualize the opinion attention scores of the words in two example sentences with the candidate aspects “maitre-D” and “bathroom”. The scores in Figures FIGREF41 and FIGREF41 show that our full model captures the related opinion words very accurately with significantly larger scores, i.e. “incredibly”, “unwelcoming” and “arrogant” for “maitre-D”, and “unfriendly” and “filthy” for “bathroom”. “OURS w/o STN” directly applies attention over the opinion hidden states INLINEFORM0 's, similar to what CMLA does. As shown in Figure FIGREF41 , it captures some unrelated opinion words (e.g. “fine”) and even some non-opinionated words. As a result, it brings in some noise into the global opinion summary, and consequently the final prediction accuracy will be affected. This example demonstrates that the proposed STN works pretty well to help attend to more related opinion words given a particular aspect.
Some predictions of our model and those of LSTM and OURS w/o THA & STN are given in Table TABREF43 . The models incorporating attention-based opinion summary (i.e., OURS and OURS w/o THA & STN) can better determine if the commonly-used nouns are aspect terms or not (e.g. “device” in the first input), since they make decisions based on the global opinion information. Besides, they are able to extract some infrequent or even misspelled aspect terms (e.g. “survice” in the second input) based on the indicative clues provided by opinion words. For the last three cases, having aspects in coordinate structures (i.e. the third and the fourth) or long aspects (i.e. the fifth), our model can give precise predictions owing to the previous detection clues captured by THA. Without using these clues, the baseline models fail.
Related Work
Some initial works BIBREF26 developed a bootstrapping framework for tackling Aspect Term Extraction (ATE) based on the observation that opinion words are usually located around the aspects. BIBREF27 and BIBREF28 performed co-extraction of aspect terms and opinion words based on sophisticated syntactic patterns. However, relying on syntactic patterns suffers from parsing errors when processing informal online reviews. To avoid this drawback, BIBREF29 , BIBREF30 employed word-based translation models. Specifically, these models formulated the ATE task as a monolingual word alignment process and aspect-opinion relation is captured by alignment links rather than word dependencies. The ATE task can also be formulated as a token-level sequence labeling problem. The winning systems BIBREF2 , BIBREF22 , BIBREF4 of SemEval ABSA challenges employed traditional sequence models, such as Conditional Random Fields (CRFs) and Maximum Entropy (ME), to detect aspects. Besides heavy feature engineering, they also ignored the consideration of opinions.
Recently, neural network based models, such as LSTM-based BIBREF6 and CNN-based BIBREF31 methods, become the mainstream approach. Later on, some neural models jointly extracting aspect and opinion were proposed. BIBREF8 performs the two task in a single Tree-Based Recursive Neural Network. Their network structure depends on dependency parsing, which is prone to error on informal reviews. CMLA BIBREF9 consists of multiple attention layers on top of standard GRUs to extract the aspects and opinion words. Similarly, MIN BIBREF17 employs multiple LSTMs to interactively perform aspect term extraction and opinion word extraction in a multi-task learning framework. Our framework is different from them in two perspectives: (1) It filters the opinion summary by incorporating the aspect features at each time step into the original opinion representations; (2) It exploits history information of aspect detection to capture the coordinate structures and previous aspect features.
Concluding Discussions
For more accurate aspect term extraction, we explored two important types of information, namely aspect detection history, and opinion summary. We design two components, i.e. truncated history attention, and selective transformation network. Experimental results show that our model dominates those joint extraction works such as RNCRF and CMLA on the performance of ATE. It suggests that the joint extraction sacrifices the accuracy of aspect prediction, although the ground-truth opinion words were annotated by these authors. Moreover, one should notice that those joint extraction methods do not care about the correspondence between the extracted aspect terms and opinion words. Therefore, the necessity of such joint extraction should be obelized, given the experimental findings in this paper.

Question: How do they determine the opinion summary?
Answer: the weighted sum of the new opinion representations, according to their associations with the current aspect representation

Question: Do they explore how useful is the detection history and opinion summary?
Answer: Yes

Question: Which dataset(s) do they use to train the model?
Answer: INLINEFORM0 (SemEval 2014) contains reviews of the laptop domain and those of INLINEFORM1 (SemEval 2014), INLINEFORM2 (SemEval 2015) and INLINEFORM3 (SemEval 2016) are for the restaurant domain.

Question: By how much do they outperform state-of-the-art methods?
Answer: Compared with the winning systems of SemEval ABSA, our framework achieves 5.0%, 1.6%, 1.4%, 1.3% absolute gains on INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 respectively.