Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence

Abstract: Aspect-based sentiment analysis (ABSA), which aims to identify fine-grained opinion polarity towards a specific aspect, is a challenging subtask of sentiment analysis (SA). In this paper, we construct an auxiliary sentence from the aspect and convert ABSA to a sentence-pair classification task, such as question answering (QA) and natural language inference (NLI). We fine-tune the pre-trained model from BERT and achieve new state-of-the-art results on SentiHood and SemEval-2014 Task 4 datasets.

Sentiment analysis (SA) is an important task in natural language processing. It solves the computational processing of opinions, emotions, and subjectivity - sentiment is collected, analyzed and summarized. It has received much attention not only in academia but also in industry, providing real-time feedback through online reviews on websites such as Amazon, which can take advantage of customers' opinions on specific products or services. The underlying assumption of this task is that the entire text has an overall polarity.
However, the users' comments may contain different aspects, such as: “This book is a hardcover version, but the price is a bit high." The polarity in `appearance' is positive, and the polarity regarding `price' is negative. Aspect-based sentiment analysis (ABSA) BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 aims to identify fine-grained polarity towards a specific aspect. This task allows users to evaluate aggregated sentiments for each aspect of a given product or service and gain a more granular understanding of their quality.
Both SA and ABSA are sentence-level or document-level tasks, but one comment may refer to more than one object, and sentence-level tasks cannot handle sentences with multiple targets. Therefore, BIBREF4 introduce the task of targeted aspect-based sentiment analysis (TABSA), which aims to identify fine-grained opinion polarity towards a specific aspect associated with a given target. The task can be divided into two steps: (1) the first step is to determine the aspects associated with each target; (2) the second step is to resolve the polarity of aspects to a given target.
The earliest work on (T)ABSA relied heavily on feature engineering BIBREF5 , BIBREF6 , and subsequent neural network-based methods BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 achieved higher accuracy. Recently, BIBREF12 incorporate useful commonsense knowledge into a deep neural network to further enhance the result of the model. BIBREF13 optimize the memory network and apply it to their model to better capture linguistic structure.
More recently, the pre-trained language models, such as ELMo BIBREF14 , OpenAI GPT BIBREF15 , and BERT BIBREF16 , have shown their effectiveness to alleviate the effort of feature engineering. Especially, BERT has achieved excellent results in QA and NLI. However, there is not much improvement in (T)ABSA task with the direct use of the pre-trained BERT model (see Table TABREF19 ). We think this is due to the inappropriate use of the pre-trained BERT model.
Since the input representation of BERT can represent both a single text sentence and a pair of text sentences, we can convert (T)ABSA into a sentence-pair classification task and fine-tune the pre-trained BERT.
In this paper, we investigate several methods of constructing an auxiliary sentence and transform (T)ABSA into a sentence-pair classification task. We fine-tune the pre-trained model from BERT and achieve new state-of-the-art results on (T)ABSA task. We also conduct a comparative experiment to verify that the classification based on a sentence-pair is better than the single-sentence classification with fine-tuned BERT, which means that the improvement is not only from BERT but also from our method. In particular, our contribution is two-fold:
1. We propose a new solution of (T)ABSA by converting it to a sentence-pair classification task.
2. We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on SentiHood and SemEval-2014 Task 4 datasets.
In this section, we describe our method in detail.
In TABSA, a sentence INLINEFORM0 usually consists of a series of words: INLINEFORM1 , and some of the words INLINEFORM2 are pre-identified targets INLINEFORM3 , following BIBREF4 , we set the task as a 3-class classification problem: given the sentence INLINEFORM4 , a set of target entities INLINEFORM5 and a fixed aspect set INLINEFORM6 , predict the sentiment polarity INLINEFORM7 over the full set of the target-aspect pairs INLINEFORM8 . As we can see in Table TABREF6 , the gold standard polarity of (LOCATION2, price) is negative, while the polarity of (LOCATION1, price) is none.
In ABSA, the target-aspect pairs INLINEFORM0 become only aspects INLINEFORM1 . This setting is equivalent to learning subtasks 3 (Aspect Category Detection) and subtask 4 (Aspect Category Polarity) of SemEval-2014 Task 4 at the same time.
For simplicity, we mainly describe our method with TABSA as an example.
We consider the following four methods to convert the TABSA task into a sentence pair classification task:
The sentence we want to generate from the target-aspect pair is a question, and the format needs to be the same. For example, for the set of a target-aspect pair (LOCATION1, safety), the sentence we generate is “what do you think of the safety of location - 1 ?"
For the NLI task, the conditions we set when generating sentences are less strict, and the form is much simpler. The sentence created at this time is not a standard sentence, but a simple pseudo-sentence, with (LOCATION1, safety) pair as an example: the auxiliary sentence is: “location - 1 - safety".
For QA-B, we add the label information and temporarily convert TABSA into a binary classification problem ( INLINEFORM0 ) to obtain the probability distribution. At this time, each target-aspect pair will generate three sequences such as “the polarity of the aspect safety of location - 1 is positive", “the polarity of the aspect safety of location - 1 is negative", “the polarity of the aspect safety of location - 1 is none". We use the probability value of INLINEFORM1 as the matching score. For a target-aspect pair which generates three sequences ( INLINEFORM2 ), we take the class of the sequence with the highest matching score for the predicted category.
The difference between NLI-B and QA-B is that the auxiliary sentence changes from a question to a pseudo-sentence. The auxiliary sentences are: “location - 1 - safety - positive", “location - 1 - safety - negative", and “location - 1 - safety - none".
After we construct the auxiliary sentence, we can transform the TABSA task from a single sentence classification task to a sentence pair classification task. As shown in Table TABREF19 , this is a necessary operation that can significantly improve the experimental results of the TABSA task.
BERT BIBREF16 is a new language representation model, which uses bidirectional transformers to pre-train a large corpus, and fine-tunes the pre-trained model on other tasks. We fine-tune the pre-trained BERT model on TABSA task. Let's take a brief look at the input representation and the fine-tuning procedure.
The input representation of the BERT can explicitly represent a pair of text sentences in a sequence of tokens. For a given token, its input representation is constructed by summing the corresponding token, segment, and position embeddings. For classification tasks, the first word of each sequence is a unique classification embedding ([CLS]).
BERT fine-tuning is straightforward. To obtain a fixed-dimensional pooled representation of the input sequence, we use the final hidden state (i.e., the output of the transformer) of the first token as the input. We denote the vector as INLINEFORM0 . Then we add a classification layer whose parameter matrix is INLINEFORM1 , where INLINEFORM2 is the number of categories. Finally, the probability of each category INLINEFORM3 is calculated by the softmax function INLINEFORM4 .
BERT for single sentence classification tasks. Suppose the number of target categories are INLINEFORM0 and aspect categories are INLINEFORM1 . We consider TABSA as a combination of INLINEFORM2 target-aspect-related sentiment classification problems, first classifying each sentiment classification problem, and then summarizing the results obtained. For ABSA, We fine-tune pre-trained BERT model to train INLINEFORM3 classifiers for all aspects and then summarize the results.
BERT for sentence pair classification tasks. Based on the auxiliary sentence constructed in Section SECREF7 , we use the sentence-pair classification approach to solve (T)ABSA. Corresponding to the four ways of constructing sentences, we name the models: BERT-pair-QA-M, BERT-pair-NLI-M, BERT-pair-QA-B, and BERT-pair-NLI-B.
We evaluate our method on the SentiHood BIBREF4 dataset, which consists of 5,215 sentences, 3,862 of which contain a single target, and the remainder multiple targets. Each sentence contains a list of target-aspect pairs INLINEFORM0 with the sentiment polarity INLINEFORM1 . Ultimately, given a sentence INLINEFORM2 and the target INLINEFORM3 in the sentence, we need to:
(1) detect the mention of an aspect INLINEFORM0 for the target INLINEFORM1 ;
(2) determine the positive or negative sentiment polarity INLINEFORM0 for detected target-aspect pairs.
We also evaluate our method on SemEval-2014 Task 4 BIBREF1 dataset for aspect-based sentiment analysis. The only difference from the SentiHood is that the target-aspect pairs INLINEFORM0 become only aspects INLINEFORM1 . This setting allows us to jointly evaluate subtask 3 (Aspect Category Detection) and subtask 4 (Aspect Category Polarity).
We use the pre-trained uncased BERT-base model for fine-tuning. The number of Transformer blocks is 12, the hidden layer size is 768, the number of self-attention heads is 12, and the total number of parameters for the pre-trained model is 110M. When fine-tuning, we keep the dropout probability at 0.1, set the number of epochs to 4. The initial learning rate is 2e-5, and the batch size is 24.
We compare our model with the following models:
LR BIBREF4 : a logistic regression classifier with n-gram and pos-tag features.
LSTM-Final BIBREF4 : a biLSTM model with the final state as a representation.
LSTM-Loc BIBREF4 : a biLSTM model with the state associated with the target position as a representation.
LSTM+TA+SA BIBREF12 : a biLSTM model which introduces complex target-level and sentence-level attention mechanisms.
SenticLSTM BIBREF12 : an upgraded version of the LSTM+TA+SA model which introduces external information from SenticNet BIBREF17 .
Dmu-Entnet BIBREF13 : a bi-directional EntNet BIBREF18 with external “memory chains” with a delayed memory update mechanism to track entities.
During the evaluation of SentiHood, following BIBREF4 , we only consider the four most frequently seen aspects (general, price, transit-location, safety). When evaluating the aspect detection, following BIBREF12 , we use strict accuracy and Macro-F1, and we also report AUC. In sentiment classification, we use accuracy and macro-average AUC as the evaluation indices.
Results on SentiHood are presented in Table TABREF19 . The results of the BERT-single model on aspect detection are better than Dmu-Entnet, but the accuracy of sentiment classification is much lower than that of both SenticLstm and Dmu-Entnet, with a difference of 3.8 and 5.5 respectively.
However, BERT-pair outperforms other models on aspect detection and sentiment analysis by a substantial margin, obtaining 9.4 macro-average F1 and 2.6 accuracies improvement over Dmu-Entnet. Overall, the performance of the four BERT-pair models is close. It is worth noting that BERT-pair-NLI models perform relatively better on aspect detection, while BERT-pair-QA models perform better on sentiment classification. Also, the BERT-pair-QA-B and BERT-pair-NLI-B models can achieve better AUC values on sentiment classification than the other models.
The benchmarks for SemEval-2014 Task 4 are the two best performing systems in BIBREF1 and ATAE-LSTM BIBREF8 . When evaluating SemEval-2014 Task 4 subtask 3 and subtask 4, following BIBREF1 , we use Micro-F1 and accuracy respectively.
Results on SemEval-2014 are presented in Table TABREF35 and Table TABREF36 . We find that BERT-single has achieved better results on these two subtasks, and BERT-pair has achieved further improvements over BERT-single. The BERT-pair-NLI-B model achieves the best performance for aspect category detection. For aspect category polarity, BERT-pair-QA-B performs best on all 4-way, 3-way, and binary settings.
Why is the experimental result of the BERT-pair model so much better? On the one hand, we convert the target and aspect information into an auxiliary sentence, which is equivalent to exponentially expanding the corpus. A sentence INLINEFORM0 in the original data set will be expanded into INLINEFORM1 in the sentence pair classification task. On the other hand, it can be seen from the amazing improvement of the BERT model on the QA and NLI tasks BIBREF16 that the BERT model has an advantage in dealing with sentence pair classification tasks. This advantage comes from both unsupervised masked language model and next sentence prediction tasks.
TABSA is more complicated than SA due to additional target and aspect information. Directly fine-tuning the pre-trained BERT on TABSA does not achieve performance growth. However, when we separate the target and the aspect to form an auxiliary sentence and transform the TABSA into a sentence pair classification task, the scenario is similar to QA and NLI, and then the advantage of the pre-trained BERT model can be fully utilized. Our approach is not limited to TABSA, and this construction method can be used for other similar tasks. For ABSA, we can use the same approach to construct the auxiliary sentence with only aspects.
In BERT-pair models, BERT-pair-QA-B and BERT-pair-NLI-B achieve better AUC values on sentiment classification, probably because of the modeling of label information.
In this paper, we constructed an auxiliary sentence to transform (T)ABSA from a single sentence classification task to a sentence pair classification task. We fine-tuned the pre-trained BERT model on the sentence pair classification task and obtained the new state-of-the-art results. We compared the experimental results of single sentence classification and sentence pair classification based on BERT fine-tuning, analyzed the advantages of sentence pair classification, and verified the validity of our conversion method. In the future, we will apply this conversion method to other similar tasks.
We would like to thank the anonymous reviewers for their valuable comments. The research work is supported by Shanghai Municipal Science and Technology Commission (No. 16JC1420401 and 17JC1404100), National Key Research and Development Program of China (No. 2017YFB1002104), and National Natural Science Foundation of China (No. 61672162 and 61751201).

Figure and table captions:
Table 1: An example of SentiHood dataset.
Table 2: The construction methods. Due to limited space, we use the following abbreviations: S.P. for sentiment polarity, w/o for without, and w/ for with.
Table 3: Performance on SentiHood dataset. We boldface the score with the best performance across all models. We use the results reported in Saeidi et al. (2016), Ma et al. (2018) and Liu et al. (2018). “-” means not reported.
Table 4: Test set results for Semeval-2014 task 4 Subtask 3: Aspect Category Detection. We use the results reported in XRCE (Brun et al., 2014) and NRC-Canada (Kiritchenko et al., 2014).
Table 5: Test set accuracy (%) for Semeval-2014 task 4 Subtask 4: Aspect Category Polarity. We use the results reported in XRCE (Brun et al., 2014), NRCCanada (Kiritchenko et al., 2014) and ATAE-LSTM (Wang et al., 2016). “-” means not reported.

Questions about this paper:
How much do they outperform previous state-of-the-art?

Answers:
On subtask 3 best proposed model has F1 score of 92.18 compared to best previous F1 score of 88.58.
On subtask 4 best proposed model has 85.9, 89.9 and 95.6 compared to best previous results of 82.9, 84.0 and 89.9 on 4-way, 3-way and binary aspect polarity.