Venue: NAACL 2021

# ASAP: A Chinese Review Dataset Towards Aspect Category Sentiment Analysis and Rating Prediction

Jiahao Bu1, Lei Ren1; Shuang Zheng12; Yang Yang1, Jinggang Wang1, Fuzheng Zhang1, Wei Wu1

1Meituan, Beijing, China

Equal contribution.Corresponding author. 1Meituan, Beijing, China

###### Abstract

Sentiment analysis has attracted increasing attention in e-commerce. The sentiment polarities underlying user reviews are of great value for business intelligence. Aspect category sentiment analysis (ACSA) and review rating prediction (RP) are two essential tasks to detect the fine-to-coarse sentiment polarities. ACSA and RP are highly correlated and usually employed jointly in real-world e-commerce scenarios. While most public datasets are constructed for ACSA and RP separately, which may limit the further exploitations of both tasks. To address the problem and advance related researches, we present a large-scale Chinese restaurant review dataset **ASAP** including \(46,730\) genuine reviews from a leading online-to-offline (O2O) e-commerce platform in China. Besides a \(5\)-star scale rating, each review is manually annotated according to its sentiment polarities towards \(18\) pre-defined aspect categories. We hope the release of the dataset could shed some light on the field of sentiment analysis. Moreover, we propose an intuitive yet effective joint model for ACSA and RP. Experimental results demonstrate that the joint model outperforms state-of-the-art baselines on both tasks.

1
Footnote 1: [https://www.tacbao.com/](https://www.tacbao.com/)

2
Footnote 2: [https://www.dianping.com/](https://www.dianping.com/)

3
Footnote 3: [https://www.koubei.com/](https://www.koubei.com/)

## 1 Introduction

With the rapid development of e-commerce, massive user reviews available on e-commerce platforms are becoming valuable resources for both customers and merchants. Aspect-based sentiment analysis(ABSA) on user reviews is a fundamental and challenging task which attracts interests from both academia and industries Hu and Liu (2004); Ganu et al. (2009); Jo and Oh (2011); Kiritchenko et al. (2014). According to whether the aspect terms are explicitly mentioned in texts, ABSA can be further classified into aspect term sentiment analysis (ATSA) and aspect category sentiment analysis (ACSA), we focus on the latter which is more widely used in industries. Specifically, given a review "Although the fish is delicious, the waiter is horrible!", the ACSA task aims to infer the sentiment polarity over aspect category _food_ is positive while the opinion over the aspect category _service_ is negative.

The user interfaces of e-commerce platforms are more intelligent than ever before with the help of ACSA techniques. For example, Figure 1 presents the detail page of a coffee shop on a popular e-commerce platform in China. The upper aspect-based sentiment text-boxes display the aspect categories (e.g., _food_, _sanitation_) mentioned frequently in user reviews and the aggregated sentiment polarities on these aspect categories (the orange ones represent positive and the blue ones represent negative). Customers can focus on corresponding reviews effectively by clicking the aspect-based sentiment text-boxes they care about (e.g., the orange filled text-box "\(\texttt{\tiny{\textregistered}}\)\(\texttt{\tiny{

diction(RP) [30, 21, 4] which aims to predict the "seeing stars" of reviews also has wide applications. For example, to promise the aspect-based sentiment text-boxes accurate, unreliable reviews should be removed before ACSA algorithms are performed. Given a piece of user review, we can predict a rating for it based on the overall sentiment polarity underlying the text. We assume the predicted rating of the review should be consistent with its ground-truth rating as long as the review is reliable. If the predicted rating and the user rating of a review disagree with each other explicitly, the reliability of the review is doubtful. Figure 2 demonstrates an example review of low-reliability. In summary, RP can help merchants to detect unreliable reviews.

Therefore, both ACSA and RP are of great importance for business intelligence in e-commerce, and they are highly correlated and complementary. ACSA focuses on predicting its underlying sentiment polarities on different aspect categories, while RP focuses on predicting the user's overall feelings from the review content. We reckon these two tasks are highly correlated and better performance could be achieved by considering them jointly.

As far as we know, current public datasets are constructed for ACSA and RP separately, which limits further joint explorations of ACSA and RP. To address the problem and advance the related researches, this paper presents a large-scale Chinese restaurant review dataset for **A**spect category **S**entiment **A**nalysis and rating **P**rediction, denotes as **ASAP** for short. All the reviews in ASAP are collected from the aforementioned e-commerce platform. There are \(46,730\) restaurant reviews attached with \(5\)-star scale ratings. Each review is manually annotated according to its sentiment polarities towards \(18\) fine-grained aspect categories. To the best of our knowledge, ASAP is the largest Chinese large-scale review dataset towards both ACSA and RP tasks.

We implement several state-of-the-art (SOTA) baselines for ACSA and RP and evaluate their performance on ASAP. To make a fair comparison, we also perform ACSA experiments on a widely used SemEval-2014 restaurant review dataset [9]. Since BERT [4] has achieved great success in several natural language understanding tasks including sentiment analysis [29, 30, 31], we propose a joint model that employs the fine-to-coarse semantic capability of BERT. Our joint model outperforms the competing baselines on both tasks.

Our main contributions can be summarized as follows. (1) We present a large-scale Chinese review dataset towards aspect category sentiment analysis and rating prediction, named as ASAP, including as many as \(46,730\) real-world restaurant reviews annotated from \(18\) pre-defined aspect categories. Our dataset has been released at [https://github.com/Meituan-Dianping/asap](https://github.com/Meituan-Dianping/asap). (2) We explore the performance of widely used models for ACSA and RP on ASAP. (3) We propose a joint learning model for ACSA and RP tasks. Our model achieves the best results both on ASAP and SemEval Restaurant datasets.

## 2 Related Work and Datasets

**Aspect Category Sentiment Analysis.** ACSA [32, 28, 29, 10] aims to predict sentiment polarities on all aspect categories mentioned in the text. The series of SemEval datasets consisting of user reviews from e-commerce websites have been widely used and

Figure 1: The user interface of a coffee shop on a popular e-commerce App. The top aspect-based sentiment text-boxes display aspect categories and sentiment polarities. The orange text-boxes are positive, while the blue ones are negative. The reviews mentioning the clicked aspect category (e.g., _good sanitation_) with ratings are shown below. The text spans mentioning the aspect categories are also highlighted.

Figure 2: A content-rating disagreement case. The review holds a \(2\)-star rating while all the mentioned aspects are super positive.

 

pushed forward related research (Wang et al., 2016; Ma et al., 2017; Xu et al., 2019; Sun et al., 2019; Jiang et al., 2019). The SemEval-2014 task-4 dataset (SE-ABSA14) (Pontiki et al., 2014) is composed of laptop and restaurant reviews. The restaurant subset includes \(5\) aspect categories (i.e., _Food_, _Service_, _Price_, _Ambience_ and _Anecdotes/Miscellaneous_) and \(4\) polarity labels (i.e., _Positive_, _Negative_, _Conflict_ and _Neutral_). The laptop subset is not suitable for ACSA. The SemEval-2015 task-12 dataset (SE-ABSA15) (Pontiki et al., 2015) builds upon SE-ABSA14 and defines its aspect category as a combination of an entity type and an attribute type(e.g., _Food#StyleOptions_). The SemEval-2016 task-5 dataset (SE-ABSA16) (Pontiki et al., 2016) extends SE-ABSA15 to new domains and new languages other than English. MAMS (Jiang et al., 2019) tailors SE-ABSA14 to make it more challenging, in which each sentence contains at least two aspects with different sentiment polarities.

Compared with the prosperity of English resources, high-quality Chinese datasets are not rich enough. "ChnSentiCorp" (Tan and Zhang, 2008), "IT168TEST" (Zagibalov and Carroll, 2008), "Weibo"4, "CTB" (Li et al., 2014) are \(4\) popular Chinese datasets for general sentiment analysis. However, aspect category information is not annotated in these datasets. Zhao et al. (2014) presents two Chinese ABSA datasets for consumer electronics (mobile phones and cameras). Nevertheless, the two datasets only contain \(400\) documents (\(\sim 4000\) sentences), in which each sentence only mentions one aspect category at most. BDCI5 automobile opinion mining and sentiment analysis dataset Dai et al. (2019) contains \(8,290\) user reviews in automobile industry with \(10\) pre-defined categories. Peng et al. (2017) summarizes available Chinese ABSA datasets. While most of them are constructed through rule-based or machine learning-based approaches, which inevitably introduce additional noise into the datasets. Our ASAP excels above Chinese datasets both on quantity and quality.

Footnote 4: [http://tcci.ccf.org.cn/conference/2014/pages/page0d_dg.html](http://tcci.ccf.org.cn/conference/2014/pages/page0d_dg.html)

Footnote 5: [https://www.datafoundain.cn/competings/310](https://www.datafoundain.cn/competings/310)

**Rating Prediction.** Rating prediction (RP) aims to predict the "seeing stars" of reviews, which represent the overall ratings of reviews. In comparison to fine-grained aspect sentiment, the overall review rating is usually a coarse-grained synthesis of the opinions on multiple aspects. Ganu et al. (2009); Li et al. (2011); Chen et al. (2018) form this task as a text classification or regression problem. Considering the importance of opinions on multiple aspects in reviews, recent years have seen numerous work Jin et al. (2016); Cheng et al. (2018); Li et al. (2018); Wu et al. (2019) utilizing the information of the aspects to improve the rating prediction performance. This trending also inspires the motivation of ASAP.

Most RP datasets are crawled from real-world review websites and created for RP specifically. Amazon Product Review English dataset McAuley and Leskovec (2013) containing product reviews and metadata from Amazon has been widely used for RP Cheng et al. (2018); McAuley and Leskovec (2013). Another popular English dataset comes from Yelp Dataset Challenge 20176, which includes reviews of local businesses in \(12\) metropolitan areas across \(4\) countries. Openrice7 is a Chinese RP dataset composed of \(168,142\) reviews. Both the English and Chinese datasets don't annotate fine-grained aspect category sentiment polarities.

Footnote 6: [http://www.yelp.com/dataset_challenge/](http://www.yelp.com/dataset_challenge/)

Footnote 7: [https://www.openrice.com](https://www.openrice.com)

## 3 Dataset Collection and Analysis

### Data Construction & Curation

We collect reviews from one of the most popular O2O e-commerce platforms in China, which allows users to publish coarse-grained star ratings and writing fine-grained reviews to restaurants (or places of interest) they have visited. In the reviews, users comment on multiple aspects either explicitly or implicitly, including _ambience_price_, _food_, _service_, and so on.

First, we retrieve a large volume of user reviews from popular restaurants holding more than \(50\) user reviews randomly. Then, \(4\) pre-processing steps are performed to promise the ethics, quality, and reliability of the reviews. (1) User information (e.g., user-ids, usernames, avatars, and post-times) are removed due to privacy considerations. (2) Short reviews with less than \(50\) Chinese characters, as well as lengthy reviews with more than \(1000\) Chinese characters are filtered out. (3) If the ratio of non-Chinese characters within a review is over \(70\)%, the review is discarded. (4) To detect the low 

quality reviews (e.g., advertising texts), we build a BERT-based classifier with an accuracy of \(97\%\) in a leave-out test-set. The reviews detected as low-quality by the classifier are discarded too.

### Aspect Categories

Since the reviews already hold users' star ratings, this section mainly introduces our annotation details for ACSA. In SE-ABSA14 restaurant dataset (denoted as Restaurant for simplicity), there are \(5\) coarse-grained aspect categories, including _food_, _service_, _price_, _ambience_ and _miscellaneous_. After an in-depth analysis of the collected reviews, we find the aspect categories mentioned by users are rather diverse and fine-grained. Take the text "...The restaurant holds a high-end decoration but is quite noisy since a wedding ceremony was being held in the main hall... (...) in Table 3 for example, the reviewer actually expresses opposite sentiment polarities on two fine-grained aspect categories related to _ambience_. The restaurant's decoration is very high-end (_Positive_), while it's very noisy due to an ongoing ceremony (_Negative_). Therefore, we summarize the frequently mentioned aspects and refine the \(5\) coarse-grained categories into \(18\) fine-grained categories. We replace _miscellaneous_ with _location_ since we find users usually review the restaurants' location (e.g., whether the restaurant is easy to reach by public transportation.). We denote the aspect category as the form of "_Coarse-grained Category#Fine-grained Category_", such as "_Food#Taste_" and "_Ambience#Decoration_". The full list of aspect categories and definitions are listed in Table 1.

### Annotation Guidelines & Process

Bearing in mind the pre-defined \(18\) aspects, assessors are asked to annotate sentiment polarities towards the mentioned aspect categories of each review. Given a review, when an aspect category is mentioned within the review either explicitly and implicitly, the sentiment polarity over the aspect category is labeled as \(1\) (_Positive_), \(0\) (_Neutral_) or \(-1\) (_Negative_) as shown in Table 3.

We hire \(20\) vendor assessors, \(2\) project managers, and \(1\) expert reviewer to perform annotations. Each assessor needs to attend a training to ensure their intact understanding of the annotation guidelines. Three rounds of annotation are conducted sequentially. First, we randomly split the whole dataset into \(10\) groups, and every group is assigned to \(2\) assessors to annotate independently. Second, each group is split into \(2\) subsets according to the annotation results, denoted as **Sub-Agree** and **Sub-Disagree**. **Sub-Agree** comprises the data examples with agreement annotation, and **Sub-Disagree** comprises the data examples with disagreement annotation. **Sub-Agree** will be reviewed by assessors from other groups. The controversial examples during the review are considered as difficult cases. **Sub-Disagree** will be reviewed by the \(2\) project managers independently and then discuss to reach an agreement annotation. The examples that could not be addressed after discussions are also considered as difficult cases. Third, for each group, the difficult examples from two subsets are delivered to the expert reviewer to make a final decision. More details of difficult cases and annotation guidelines during annotation are demonstrated in Table 2.

Finally, ASAP corpus consists of \(46,730\) pieces of real-world user reviews, and we split it into a training set (\(36,850\)), a validation set (\(4,940\)) and a test set (\(4,940\)) randomly. Table 3 presents an example review of ASAP and corresponding annotations on the \(18\) aspect categories.

### Dataset Analysis

Figure 3 presents the distribution of \(18\) aspect categories in ASAP. Because ASAP concentrates on the domain of restaurant, \(94.7\)% reviews mention _Food#Taste_ as expected. Users also pay great attention to aspect categories such as _Service#Hospitality_, _Price#Level_ and _Amphence#Decoration_. The distribution proves the advantages of ASAP, as users' fine-grained preferences could reflect the pros and cons of restaurants more precisely.

The statistics of ASAP are presented in Table 4. We also include a tailored SE-ABSA14 Restaurant dataset for reference. Please note that we remove the reviews holding aspect categories with sentiment polarity of "conflict" from the original Restaurant dataset.

Compared with Restaurant, ASAP excels in the quantities of training instances, which supports the exploration of recent data-intensive deep neural models. ASAP is a review-level dataset, while Restaurant is a sentence-level dataset. The average length of reviews in ASAP is much longer, thus the reviews tend to contain richer aspect information. In ASAP, the reviews contain 

\(5.8\) aspect categories in average, which is \(4.7\) times of Restaurant. Both review-level ACSA and RP are more challenging than their sentence-level counterparts. Take the review in Table 3 for example, the review contains several sentiment polarities towards multiple aspect categories. In addition to aspect category sentiment annotations, ASAP also includes overall user ratings for reviews. With the help of ASAP, ACSA and RP can be further optimized either separately or jointly.

## 4 Methodology

### Problem Formulation

We use \(D\) to denote the collection of user review corpus in the training data. Given a review \(R\) which consists of a series of words: \(\{w_{1},w_{2},...,w_{Z}\}\), ACSA aims to predict the sentiment polarity \(y_{i}\in\{Positive,Neutral,Negative\}\) of review \(R\) with respect to the mentioned aspect category \(a_{i}\), \(i\in\{1,2,...,N\}\). \(Z\) denotes the length of review \(R\). \(N\) is the number of pre-defined aspect categories (i.e., \(18\) in this paper). Suppose there are \(K\) mentioned aspect categories in \(R\). We define a mask vector \([p_{1},p_{2},...,p_{N}]\) to indicate the occurrence of aspect categories. When the aspect category \(a_{i}\) is mentioned in \(R\), \(p_{i}=1\), otherwise \(p_{i}=0\). So we have \(\sum_{i=1}^{N}p_{i}=K\). In terms of RP, it aims to predict the \(5\)-star rating score of \(g\), which represents the overall rating of the given review \(R\).

### Joint Model

Given a user review, ACSA focuses on predicting its underlying sentiment polarities on different aspect categories, while RP focuses on predicting the user's overall feelings from the review content. We reckon these two tasks are highly correlated and better performance could be achieved by considering them jointly.

The advent of BERT has established the success

Figure 3: The distribution of \(18\) fine-grained aspect categories in ASAP.

 

of the "pre-training and then fine-tuning" paradigm for NLP tasks. BERT-based models have achieved impressive results in ACSA Xu et al. (2019); Sun et al. (2019); Jiang et al. (2019). Review rating prediction can be deemed as a single-sentence classification (regression) task, which could also be addressed with BERT. Therefore, we propose a joint learning model to address ACSA and RP in a multi-task learning manner. Our joint model employs the fine-to-coarse semantic representation capability of the BERT encoder. Figure 4 illustrates the framework of our joint model.

**ACSA** As shown in Figure 4, the token embeddings of the input review are generated through a shared BERT encoder. Briefly, let \(H\in\mathbb{R}^{d*Z}\) be the matrix consisting of token embedding vectors \(\{h_{1},...,h_{Z}\}\) that BERT produces, where \(d\) is the size of hidden layers and \(Z\) is the length of the given review. Since different aspect category information is dispersed across the content of \(R\), we add an attention-pooling layer Wang et al. (2016) to aggregate the related token embeddings dynamically for every aspect category. The attention-pooling layer helps the model focus on the tokens most related to the target aspect categories.

\[M_{i}^{a}=\tanh(W_{i}^{a}*H) \tag{1}\]

\[\alpha_{i}=softmax(\omega_{i}^{T}*M_{i}^{a}) \tag{2}\]

\[r_{i}=\tanh(W_{i}^{p}*H*\alpha_{i}^{T}) \tag{3}\]

Where \(W_{i}^{a}\in\mathbb{R}^{d*d}\), \(M_{i}^{a}\in\mathbb{R}^{d*Z}\), \(\omega_{i}\in\mathbb{R}^{d}\), \(\alpha_{i}\in\mathbb{R}^{Z},W_{i}^{p}\in\mathbb{R}^{d*d}\), and \(r_{i}\in\mathbb{R}^{d}\). \(\alpha_{i}\) is a vector consisting of attention weights of all tokens which can selectively attend the regions of the aspect category related tokens, and \(r_{i}\) is the attentive representation of review with respect to the \(i_{th}\) aspect category \(a_{i}\), \(i\in\{1,2,...,N\}\). Then we have

\[\hat{y}_{i}=softmax(W_{i}^{q}*r_{i}+b_{i}^{q}) \tag{4}\]

Where \(W_{i}^{q}\in\mathbb{R}^{C*d}\) and \(b_{i}^{q}\in\mathbb{R}^{C}\) are trainable parameters of the softmax layer. \(C\) is the number of labels (i.e, 3 in our task). Hence, the ACSA loss for a given review \(R\) is defined as follows,

\[loss_{ACSA}=\frac{1}{K}\sum_{i=1}^{N}p_{i}\sum_{C}y_{i}*\log\hat{y}_{i} \tag{5}\]

If the aspect category \(a_{i}\) is not mentioned in \(S\), \(y_{i}\) is set as a random value. The \(p_{i}\) serves as a gate function, which filters out the random \(y_{i}\) and ensures only the mentioned aspect categories can participate in the calculation of the loss function.

\begin{table}
\begin{tabular}{c c l l l} \hline Category & Example & Example (Translation) & Guideline & Annotation \\ \hline Change of & & I used to like the food of & When there existed a & _(Food#Taste, \(-1\))_ \\ sentiment over & & This restaurant, but the & sentinel drifting over & _(Food\#Taste, \(-1\))_ \\ time & & taste is not as expected & time in the review, the & \\  & & today. & most recent sentiment & \\ Implicit sentiment & & The restaurant was far & Some reviewers express & \\ polarity & i¿¿ T & & worse than the dinning & their polarities in an \\  & & hill¿ R¿ P¿ A¿ & h¿ M¿ & malt¿ & implicit manner instead \\  & & & considering that the meal & of expressing their \\  & & for two people only cost & feelings directly. The \\  & & & \(500\) CNY in a five-star & implicit sentiment \\  & & hotel. & polarity is adopted to & \\ Conflict opinions & ii¿¿ If¿ A¿¿, & & complete the annotation. & \\  & & hit¿ R¿ & This dish was a bit salty, & When there existed \\  & & but it tasted great. & multiple sentiment & \\  & & & & \\  & & & & same aspect-category, the \\  & & dominant sentiment is & chosen. & \\ Mild sentiment & & The food was okay, but & The “neutral” label & \\  & & nothing great. & applies to mildly positive & \\  & & or mildly negative & \\ Irrelevant & & The food of the shop & The revive mentions & \\ restaurant & & which I went to last time & restaurant that the user & \\  & & was very bad. Today I & has visited in the past. & \\  & & came to this new one. I & We only focus on the & \\  & & felt very good. & restaurant being reviewed & \\ \hline \end{tabular}
\end{table}
Table 2: Difficult cases and annotation guidelines.

 



**Rating Prediction** Since the objective of RP is to predict the review rating based on the review content, we adopt the [CLS] embedding \(h_{[cls]}\in\mathbb{R}^{d}\) BERT produces as the representation of the input review, where \(d\) is the size of hidden layers in the BERT encoder.

\[\hat{g}=\beta^{T}*\tanh(W^{r}*h_{[cls]}+b^{r}) \tag{6}\]

Hence the RP loss for a given review \(R\) is defined as follows,

\[loss_{RP}=|g-\hat{g}| \tag{7}\]

Where \(W^{r}\in\mathbb{R}^{d*d}\),\(b^{r}\in\mathbb{R}^{d}\), \(\beta\in\mathbb{R}^{d}\) are trainable parameters.

The final loss of our joint model becomes as follows.

\[loss=loss_{ACSA}+loss_{RP} \tag{8}\]

## 5 Experiments

We perform an extensive set of experiments to evaluate the performance of our joint model on ASAP

Figure 4: The framework of the proposed joint learning model. The right part of the dotted vertical line is used to predict multiple aspect category sentiment polarities, while the left part is used to predict the review rating.



and Restaurant(Pontiki et al., 2014). Ablation studies are also conducted to probe the interactive influence between ACSA and RP.

### Acsa

**Baseline Models** We implement several ACSA baselines for comparison. According to the different structures of their encoders, these models are classified into Non-BERT based models or BERT-based models. Non-BERT based models include TextCNN (Kim, 2014), BiLSTM+Attn Zhou et al. (2016), ATAE-LSTM Wang et al. (2016) and CapsNet Sabour et al. (2017). BERT-based models include vanilla BERT Devlin et al. (2018), QA-BERT Sun et al. (2019) and CapsNet-BERT Jiang et al. (2019).

**Implementation Details of Experimental Models** In terms of non-BERT-based models, we initialize their inputs with pre-trained embeddings. For Chinese ASAP, we utilize Jieba8 to segment Chinese texts and adopt Tencent Chinese word embeddings Song et al. (2018) composed of \(8,000,000\) words. For English Restaurant, we adopt \(300\)-dimensional word embeddings pre-trained by Glove Pennington et al. (2014).

Footnote 8: [https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba)

In terms of BERT-based models, we adopt the \(12\)-layer Google BERT Base9 to encode the inputs.

Footnote 9: [https://github.com/google-research/bert](https://github.com/google-research/bert)

The batch sizes are set as \(32\) and \(16\) for non-BERT-based models and BERT-based models respectively. Adam optimizer Kingma and Ba (2014) is employed with \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\). The maximum sequence length is set as \(512\). The number of epochs is set as \(3\). The learning rates are set as \(0.001\) and \(0.00005\) for non-BERT-based models and BERT-based models respectively. All the models are trained on a single NVIDIA Tesla \(32\)G V\(100\) Volta GPU.

**Evaluation Metrics** Following the settings of Restaurant, we adopt Macro-F1 and Accuracy (Acc) as evaluation metrics.

**Experimental Results & Analysis** We report the performance of aforementioned models on ASAP and Restaurant in Table 5. Generally, BERT-based models outperform Non-BERT based models on both datasets. The two variants of our joint model perform better than vanilla-BERT, QA-BERT, and CapsNet-BERT, which proves the advantages of our joint learning model. Given a user review, vanilla-BERT, QA-BERT, and CapsNet-BERT treat the pre-defined aspect categories independently, while our joint model combines them together with a multi-task learning framework. On one hand, the encoder-sharing setting enables knowledge transferring among different aspect categories. On the other hand, our joint model is more efficient than other competitors, especially when the number of aspect categories is large. The ablation of RP (i.e., joint model(w/o RP)) still outperforms all other baselines. The introduction of RP to ACSA brings marginal improvement. This is reasonable considering that the essential objective of RP is to estimate the overall sentiment polarity instead of fine-grained sentiment polarities.

We visualize the attention weights produced by our joint model on the example of Table 3 in Figure 5. Since different aspect category information is dispersed across the review of \(R\), we add an attention-pooling layer Wang et al. (2016) to aggregate the related token embeddings dynamically for every aspect category. The attention-pooling layer helps the model focus on the tokens most related to the target aspect categories. Figure 5 visualizes attention weights of \(3\) given aspect categories. The intensity of the color represents the magnitude of attention weight, which means the relatedness of tokens to the given aspect category. It's obvious that our joint model focus on the tokens most related to the aspect categories across the review of \(R\).

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline Dataset & Split & Reviews & Average sentences & Average papers & Negative length & Positive Negative & Neutral & \(1\)-st & \(2\)-st & \(3\)-stur & \(4\)-stur & \(5\)-star \\  & & & per review & per review & & & & & & & & \\ \hline \multirow{3}{*}{ASAP} & Train & \(36,850\) & \(8.6\) & \(5.8\) & \(319.7\) & \(183,721\) & \(27r,425\) & \(52,225\) & \(1,219\) & \(1,258\) & \(5,241\) & \(13,392\) & \(15,770\) \\  & Dev & \(4,940\) & \(8.7\) & \(5.9\) & \(319.9\) & \(18,176\) & \(3,733\) & \(7,192\) & \(151\) & \(166\) & \(784\) & \(1,734\) & \(2,105\) \\ \cline{1-1}  & Test & \(4,940\) & \(8.3\) & \(5.7\) & \(317.1\) & \(17,523\) & \(3,813\) & \(7,026\) & \(165\) & \(173\) & \(717\) & \(1,867\) & \(2,018\) \\ \hline \multirow{3}{*}{Restaurant} & Train & \(2,855\) & \(1\) & \(1.2\) & \(15.2\) & \(2150\) & \(822\) & \(498\) & - & - & - & - \\ \cline{1-1}  & Test & \(7.49\) & \(1\) & \(1.3\) & \(15.6\) & \(645\) & \(215\) & \(94\) & - & - & - & - \\ \hline \end{tabular}
\end{table}
Table 4: The statistics and label/rating distribution of ASAP and Restaurant. The review length are counted by Chinese characters and English words respectively. The sentences are segmented with periods in ASAP, while Restaurant is a sentence-level dataset.

 



### Rating Prediction

We compare several RP models on ASAP, including TextCNN Kim (2014), BiLSTM+Attn Zhou et al. (2016) and ARP Wu et al. (2019b). The data pre-processing and implementation details are identical with ACSA experiments.

**Evaluation Metrics.** We adopt Mean Absolute Error (MAE) and Accuracy (by mapping the predicted rating score to the nearest category) as evaluation metrics.

**Experimental Results & Analysis** The experimental results of comparative RP models are illustrated in Table 6.

Our joint model which combines ACSA and RP outperforms other models considerably. On one hand, the performance improvement is expected since our joint model is built upon BERT. On the other hand, the ablation of ACSA (i.e., joint model(w/o ACSA)) brings performance degradation of RP on both metrics. We can conclude that the fine-grained aspect category sentiment prediction of the review indeed helps the model predict its overall rating more accurately.

This section conducts preliminary experiments to evaluate classical ACSA and RP models on our proposed ASAP dataset. We believe there still exists much room for improvements to both tasks, and we will leave them for future work.

## 6 Conclusion

This paper presents **ASAP**, a large-scale Chinese restaurant review dataset towards aspect category sentiment analysis (ACSA) and rating prediction (RP). ASAP consists of \(46,730\) restaurant user reviews with star ratings from a leading e-commerce platform in China. Each review is manually annotated according to its sentiment polarities on \(18\) fine-grained aspect categories. Besides evaluations of ACSA and RP models on ASAP separately, we also propose a joint model to address ACSA and RP synthetically, which outperforms other state-of-the-art baselines considerably. we hope the release of ASAP could push forward related researches and applications.

Figure 5: Attention visualization example. We only show attention weights of \(3\) aspect categories for beauty. The red text span ”_With convenient traffic...{X}@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@[]@][]@[]@[]@[]@[]@[]@[]@[]@][]@[]@[][]@[]@[]@[][]@][]@[][]@[][]@[][]@][][]@[][][]@][][][][][][][][][][][][]][][][][][][][][][][]][][][][][][][][][][]][][][][][][][][][][]][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][][]][][][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][][]][][][][][][][][][][][][][]][][][][][][][][][][][][][]][][][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][][][]][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][][][][]][][][][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][][]][][][][][][][][][][][][]][][][][][][][][][][]][][][][][][][][][]][][][][][][][][][]][][][][][][][][][]][][][][][][][][][]][][][][][][][]][][][][][][]][][][][][][][]][][][][][][][][]][][][][][][][]][][][][][][][]][][][][][][]][][][][][][][]][][][][][][]][]



## References

* Chen et al. (2018) Chong Chen, Min Zhang, Yiqun Liu, and Shaoping Ma. 2018. Neural attentional rating regression with review-level explanations. In _Proceedings of the 2018 World Wide Web Conference_, pages 1583-1592.
* Cheng et al. (2018) Zhiyong Cheng, Ying Ding, Lei Zhu, and Mohan Kankanhalli. 2018. Aspect-aware latent factor model: Rating prediction with ratings and reviews. In _Proceedings of the 2018 world wide web conference_, pages 639-648.
* Dai et al. (2019) Zehui Dai, Wei Dai, Zhenhua Liu, Fengyun Rao, Huajie Chen, Guangpeng Zhang, Yadong Ding, and Jiyang Liu. 2019. Multi-task multi-head attention memory network for fine-grained sentiment analysis. In _CCF International Conference on Natural Language Processing and Chinese Computing_, pages 609-620. Springer.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _CoRR_, abs/1810.04805.
* Ganu et al. (2009) Gayatree Ganu, Noemie Elhadad, and Amelie Marian. 2009. Beyond the stars: improving rating predictions using review text content. In _WebDB_, volume 9, pages 1-6. Citeseer.
* Hu et al. (2018) Mengting Hu, Shiwan Zhao, Li Zhang, Keke Cai, Zhong Su, Renhong Cheng, and Xiaowei Shen. 2018. CAN: constrained attention networks for multi-aspect sentiment analysis. _CoRR_, abs/1812.10735.
* Hu and Liu (2004) Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In _Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 168-177.
* Jiang et al. (2019) Qingnan Jiang, Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang. 2019. A challenge dataset and effective models for aspect-based sentiment analysis. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6281-6286.
* Jin et al. (2016) Zhipeng Jin, Qiudan Li, Daniel D Zeng, YongCheng Zhan, Ruron Liu, Lei Wang, and Hongyuan Ma. 2016. Jointly modeling review content and aspect ratings for review rating prediction. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_, pages 893-896.
* Jo and Oh (2011) Yohan Jo and Alice H Oh. 2011. Aspect and sentiment unification model for online review analysis. In _Proceedings of the fourth ACM international conference on Web search and data mining_, pages 815-824. ACM.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. _arXiv preprint arXiv:1408.5882_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Kiritchenko et al. (2014) Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif Mohammad. 2014. Nrc-canada-2014: Detecting aspects and sentiment in customer reviews. In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_, pages 437-442.
* Li et al. (2014) Changliang Li, Bo Xu, Gaowei Wu, Saike He, Guanhua Tian, and Hongwei Hao. 2014. Recursive deep learning for sentiment analysis over social data. In _2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT)_, volume 2, pages 180-185. IEEE.
* Li et al. (2011) Fangtao Li, Nathan Nan Liu, Hongwei Jin, Kai Zhao, Qiang Yang, and Xiaoyan Zhu. 2011. Incorporating reviewer and product information for review rating prediction. In _Twenty-second international joint conference on artificial intelligence_.
* Li et al. (2018) Junje Li, Haitong Yang, and Chengqing Zong. 2018. Document-level multi-aspect sentiment classification by jointly modeling users, aspects, and overall ratings. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 925-936, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Ma et al. (2017) Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In _International Joint Conference on Artificial Intelligence_, pages 4068-4074.
* McAuley and Leskovec (2013) Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In _Proceedings of the 7th ACM conference on Recommender systems_, pages 165-172.
* Movahedi et al. (2019) Sajd Movahedi, Erfan Ghadery, Heshaam Faili, and Azadeh Shakery. 2019. Aspect category detection via topic-attention network.
* Peng et al. (2017) Haiyu Peng, Erik Cambria, and Amir Hussain. 2017. A review of sentiment analysis research in chinese language. _Cognitive Computation_, 9(4):423-435.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Pontiki et al. (2015) Maria Pontiki, Dimitrios Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment

analysis. In _International Workshop on Semantic Evaluation (SemEval 2015)_.
* Pontiki et al. [2014] Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Haris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In _International Workshop on Semantic Evaluation (SemEval 2015)_.
* Pontiki et al. [2016] Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, ALSmadi Mohammad, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphee De Clercq, et al. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In _Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016)_, pages 19-30.
* Ruder et al. [2016] Sebastian Ruder, Parsa Ghaffari, and John G. Breslin. 2016. A hierarchical model of reviews for aspect-based sentiment analysis.
* Sabour et al. [2017] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In _Advances in neural information processing systems_, pages 3856-3866.
* Song et al. [2018] Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional skip-gram: Explicitly distinguishing left and right context for word embeddings. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 175-180.
* Sun et al. [2019] Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. _arXiv preprint arXiv:1903.09588_.
* Tan and Zhang [2008] Songbo Tan and Jin Zhang. 2008. An empirical study of sentiment analysis for chinese documents. _Expert Systems with applications_, 34(4):2622-2629.
* Wang et al. [2016] Yequan Wang, Minlie Huang, and Li Zhao. 2016. Attention-based lstm for aspect-level sentiment classification. In _Empirical Methods in Natural Language Processing_, pages 606-615.
* Wu et al. [2019a] Chuhan Wu, Fangzhao Wu, Junxin Liu, Yongfeng Huang, and Xing Xie. 2019a. Arp: Aspect-aware neural review rating prediction. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_, pages 2169-2172.
* Wu et al. [2019b] Chuhan Wu, Fangzhao Wu, Junxin Liu, Yongfeng Huang, and Xing Xie. 2019b. Arp: Aspect-aware neural review rating prediction. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_, pages 2169-2172.
* Xu et al. [2019] Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. _arXiv preprint arXiv:1904.02232_.
* Zagigilov and Carroll [2008] Taras Zagigilov and John A Carroll. 2008. Unsupervised classification of sentiment and objectivity in chinese text. In _Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I_.
* Zhao et al. [2014] Yangyan Zhao, Bing Qin, and Ting Liu. 2014. Creating a fine-grained corpus for chinese sentiment analysis. _IEEE Intelligent Systems_, 30(1):36-43.
* Zhou et al. [2016] Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 207-212.
* Zhou et al. [2015] Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2015. Representation learning for aspect category detection in online reviews. In _Twenty-ninth Aaai Conference on Artificial Intelligence_.