SlowGuess/NAACL_2021_OCR huggingface dataset:
Where  $W_{i}^{q} \in \mathbb{R}^{C*d}$  and  $b_{i}^{q} \in \mathbb{R}^{C}$  are trainable parameters of the softmax layer.  $C$  is the number of labels (i.e., 3 in our task). Hence, the ACSA loss for a given review  $R$  is defined as follows,
$$
\operatorname {l o s s} _ {A C S A} = \frac {1}{K} \sum_ {i = 1} ^ {N} p _ {i} \sum_ {C} y _ {i} * \log \hat {y} _ {i} \tag {5}
$$
If the aspect category  $a_i$  is not mentioned in  $S$ ,  $y_i$  is set as a random value. The  $p_i$  serves as a gate function, which filters out the random  $y_i$  and ensures only the mentioned aspect categories can participate in the calculation of the loss function.
Table 3: A review example in ASAP, with overall star rating and aspect category sentiment polarity annotations.
Review	Rating	Aspect Category	Label	Aspect Category	Label
Location#Transportation1(交通方便)	Price#Discount(折扣力度)	-
With convenient traffic, the restaurant holds a high-end decoration, but quite noisy because a wedding ceremony was being held in the main hall. Impressed by its delicate decoration and grand appearance though, we had to wait for a while at the weekend time. However, considering its high price level, the taste is unexpected. We ordered the Kung Pao Prawn, the taste was acceptable and the serving size is enough, but the shrimp is not fresh. In terms of service, you could not expect too much due to the massive customers there. By the way, the free-served fruit cup was nice. Generally speaking, it was a typical wedding banquet restaurant rather than a comfortable place to date with friends.	3-Star	Location#Downtown(位于商圈附近)	-	Ambience#Decoration(装修)	1
Location#Easy_to_find(是否容易寻找)	-	Ambience#Noise(嘈杂情况)	-1
Service#Queue(排队时间)	-	Ambience#Space(就餐空间)	1
Service#Hospitality(服务人员态度)	-	Ambience#Sanitary(卫生情况)	-
交通还挺方便的，环境看起来很高大上的样子，但是因为主厅在举办婚礼非常混乱，特别吵感觉，但是装修的还不错，感觉很精致的装修，门面很气派，周末去的时候还需要等位。味道的话我觉得还可以但是跟价格比起来就很一般了，性价比挺低的，为了去吃宫保虾球的，但是我觉得也就那样吧虾不是特别新鲜，不过虾球很大，味道还行。服务的话由于人很多所以也顾不过来上菜的速度不快，但是有送水果杯还挺好吃的。总之就是典型的婚宴餐厅不是适合普通朋友吃饭的地方了。	Service#Parking(停车方便)	-	Food#Portion(分量)	1
Service#Timely(点菜/上菜速度)	-1	Food#Taste(口味)	1
Price#Level(价格水平)	0	Food#Appearance(外观)	-
Price#Cost-effective(性价比)	-1	Food#Recommend(推荐程度)	-
Figure 4: The framework of the proposed joint learning model. The right part of the dotted vertical line is used to predict multiple aspect category sentiment polarities, while the left part is used to predict the review rating.
Rating Prediction Since the objective of RP is to predict the review rating based on the review content, we adopt the [CLS] embedding  $h_{[cls]} \in \mathbb{R}^d$
BERT produces as the representation of the input review, where  $d$  is the size of hidden layers in the BERT encoder.
$$
\hat {g} = \beta^ {T} * \tanh  \left(W ^ {r} * h _ {[ c l s ]} + b ^ {r}\right) \tag {6}
$$
Hence the RP loss for a given review  $R$  is defined as follows,
$$
\operatorname {l o s s} _ {R P} = | g - \hat {g} | \tag {7}
$$
Where  $W^{r} \in \mathbb{R}^{d*d}, b^{r} \in \mathbb{R}^{d}, \beta \in \mathbb{R}^{d}$  are trainable parameters.
The final loss of our joint model becomes as follows.
$$
l o s s = l o s s _ {A C S A} + l o s s _ {R P} \tag {8}
$$
We perform an extensive set of experiments to evaluate the performance of our joint model on ASAP
Table 4: The statistics and label/rating distribution of ASAP and RESTAURANT. The review length are counted by Chinese characters and English words respectively. The sentences are segmented with periods in ASAP, while RESTAURANT is a sentence-level dataset.
Dataset	Split	Reviews	Average sentences per review	Average aspects per review	Average length	Positive	Negative	Neutral	1-star	2-star	3-star	4-star	5-star
ASAP	Train	36,850	8.6	5.8	319.7	133,721	27,425	52,225	1,219	1,258	5,241	13,362	15,770
Dev	4,940	8.7	5.9	319.9	18,176	3,733	7,192	151	166	784	1,734	2,105
Test	4,940	8.3	5.7	317.1	17,523	3,813	7,026	165	173	717	1,867	2,018
RESTAURANT	Train	2,855	1	1.2	15.2	2150	822	498	-	-	-	-	-
Test	749	1	1.3	15.6	645	215	94	-	-	-	-	-
and RESTAURANT (Pontiki et al., 2014). Ablation studies are also conducted to probe the interactive influence between ACSA and RP.
5.1 ACSA
Baseline Models We implement several ACSA baselines for comparison. According to the different structures of their encoders, these models are classified into Non-BERT based models or BERT-based models. Non-BERT based models include TextCNN (Kim, 2014), BiLSTM+Attn (Zhou et al., 2016), ATAE-LSTM (Wang et al., 2016) and CapsNet (Sabour et al., 2017). BERT-based models include vanilla BERT (Devlin et al., 2018), QABERT (Sun et al., 2019) and CapsNet-BERT (Jiang et al., 2019).
Implementation Details of Experimental Models In terms of non-BERT-based models, we initialize their inputs with pre-trained embeddings. For Chinese ASAP, we utilize Jieba to segment Chinese texts and adopt Tencent Chinese word embeddings (Song et al., 2018) composed of 8,000,000 words. For English RESTAURANT, we adopt 300-dimensional word embeddings pre-trained by Glove (Pennington et al., 2014).
In terms of BERT-based models, we adopt the 12-layer Google BERT Base to encode the inputs.
The batch sizes are set as 32 and 16 for non-BERT-based models and BERT-based models respectively. Adam optimizer (Kingma and Ba, 2014) is employed with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ . The maximum sequence length is set as 512. The number of epochs is set as 3. The learning rates are set as 0.001 and 0.00005 for non-BERT-based models and BERT-based models respectively. All the models are trained on a single NVIDIA Tesla 32G V100 Volta GPU.
Evaluation Metrics Following the settings of RESTAURANT, we adopt Macro-F1 and Accuracy
(Acc) as evaluation metrics.
Experimental Results & Analysis We report the performance of aforementioned models on ASAP and RESTAURANT in Table 5. Generally, BERT-based models outperform Non-BERT based models on both datasets. The two variants of our joint model perform better than vanilla-BERT, QABERT, and CapsNet-BERT, which proves the advantages of our joint learning model. Given a user review, vanilla-BERT, QA-BERT, and CapsNet-BERT treat the pre-defined aspect categories independently, while our joint model combines them together with a multi-task learning framework. On one hand, the encoder-sharing setting enables knowledge transferring among different aspect categories. On the other hand, our joint model is more efficient than other competitors, especially when the number of aspect categories is large. The ablation of RP (i.e., joint model(w/o RP)) still outperforms all other baselines. The introduction of RP to ACSA brings marginal improvement. This is reasonable considering that the essential objective of RP is to estimate the overall sentiment polarity instead of fine-grained sentiment polarities.
We visualize the attention weights produced by our joint model on the example of Table 3 in Figure 5. Since different aspect category information is dispersed across the review of  $R$ , we add an attention-pooling layer (Wang et al., 2016) to aggregate the related token embeddings dynamically for every aspect category. The attention-pooling layer helps the model focus on the tokens most related to the target aspect categories. Figure 5 visualizes attention weights of 3 given aspect categories. The intensity of the color represents the magnitude of attention weight, which means the relatedness of tokens to the given aspect category. It's obvious that our joint model focuses on the tokens most related to the aspect categories across the review of  $R$ .
https://github.com/fxsjy/jieba  
https://github.com/google-research/bert
Table 5: The experimental results of ACSA models on ASAP and RESTAURANT. Best scores are boldfaced.
Category	Model	ASAP	RESTAURANT
Macro-F1	Acc.	Macro-F1	Acc.
Non-BERT-based models	TextCNN (Kim, 2014)	60.41%	71.10%	70.56%	82.29%
BiLSTM+Attn (Zhou et al., 2016)	70.53%	77.78%	70.85%	81.97%
ATAE-LSTM (Wang et al., 2016)	76.60%	81.94%	70.15%	82.12%
CapsNet (Sabour et al., 2017)	75.54%	81.66%	71.84%	82.63%
BERT-based models	Vanilla-BERT (Devlin et al., 2018)	79.18%	84.09%	79.22%	87.63%
QA-BERT (Sun et al., 2019)	79.44%	83.92%	80.89%	88.89%
CapsNet-BERT (Jiang et al., 2019)	78.92%	83.74%	80.94%	89.00%
Joint Model (w/o RP)	80.75%	85.15%	82.01%	89.62%
Joint Model	80.78%	85.19%	-	-
Review:
交通还挺方便的，环境看起来很高大上的样子，但是因为主厅在举办婚礼非常混乱，特别吵感觉，但是装修的还不错，感觉很精致的装修，门面很气派，周末去的时候还需要等位。味道的话我觉得还可以但是跟价格比起来就很一般了，性价比挺低的，为了去吃宫保虾球的，但是我觉得也就那样吧虾不是特别新鲜，不过虾球很大，味道还行。服务的话由于人很多所以也顾不过来上菜的速度不快，但是有送水果杯还挺好吃的。总之就是典型的婚宴餐厅不是适合普通朋友吃饭的地方了。
With convenient traffic, the restaurant holds a high-end decoration, but quite noisy because a wedding ceremony was being held in the main hall. Impressed by its delicate decoration and grand appearance though, we had to wait for a while at the weekend time. However, considering its high price level, the taste is unexpected. We ordered the Kung Pao Prawn, the taste was acceptable and the serving size is enough, but the shrimp is not fresh. In terms of service, you could not expect too much due to the massive customers there. By the way, the free-served fruit cup was nice. Generally speaking, it was a typical wedding banquet restaurant rather than a comfortable place to date with friends.
Figure 5: Attention visualization example. We only show attention weights of 3 aspect categories for beauty. The red text span ".With convenient traffic.(交通还挺方便的.)" is related to Location#Transportation. The blue text span ".the restaurant holds a high-end decoration..Impressed by its delicate decoration and grand appearance though.(环境看起来高大上的样子..装修还不错，很精致的装修.)" is related to Ambience#Decoration. The green text span ".but quite noisy.(特别吵感觉.)" is related to Ambience#Noise. The intensity of the color represents the magnitude of attention weight.
5.2 Rating Prediction
We compare several RP models on ASAP, including TextCNN (Kim, 2014), BiLSTM+Attn (Zhou et al., 2016) and ARP (Wu et al., 2019b). The data pre-processing and implementation details are identical with ACSA experiments.
Evaluation Metrics. We adopt Mean Absolute Error (MAE) and Accuracy (by mapping the predicted rating score to the nearest category) as evaluation metrics.
Experimental Results & Analysis The experimental results of comparative RP models are illustrated in Table 6.
Table 6: Experimental results of RP models on ASAP. Best scores are boldfaced.
Model	MAE	Acc.
TextCNN (Kim, 2014)	.5814	52.99%
BiLSTM+Attn (Zhou et al., 2016)	.5737	54.38%
ARP (Wu et al., 2019b)	.5620	54.76%
Joint Model (w/o ACSA)	.4421	60.08%
Joint Model	.4266	61.26%
Our joint model which combines ACSA and RP outperforms other models considerably. On one hand, the performance improvement is expected since our joint model is built upon BERT. On the other hand, the ablation of ACSA (i.e., joint model(w/o ACSA)) brings performance degradation of RP on both metrics. We can conclude that the fine-grained aspect category sentiment prediction of the review indeed helps the model predict its overall rating more accurately.
This section conducts preliminary experiments to evaluate classical ACSA and RP models on our proposed ASAP dataset. We believe there still exists much room for improvements to both tasks, and we will leave them for future work.
This paper presents ASAP, a large-scale Chinese restaurant review dataset towards aspect category sentiment analysis (ACSA) and rating prediction (RP). ASAP consists of 46,730 restaurant user reviews with star ratings from a leading e-commerce platform in China. Each review is manually annotated according to its sentiment polarities on 18 fine-grained aspect categories. Besides evaluations of ACSA and RP models on ASAP separately, we also propose a joint model to address ACSA and RP synthetically, which outperforms other state-of-the-art baselines considerably. We hope the release of ASAP could push forward related researches and applications.
Chong Chen, Min Zhang, Yiqun Liu, and Shaoping Ma. 2018. Neural attentional rating regression with review-level explanations. In Proceedings of the 2018 World Wide Web Conference, pages 1583-1592.
Zhiyong Cheng, Ying Ding, Lei Zhu, and Mohan Kankanhalli. 2018. Aspect-aware latent factor model: Rating prediction with ratings and reviews. In Proceedings of the 2018 world wide web conference, pages 639-648.
Zehui Dai, Wei Dai, Zhenhua Liu, Fengyun Rao, Huajie Chen, Guangpeng Zhang, Yadong Ding, and Jiyang Liu. 2019. Multi-task multi-head attention memory network for fine-grained sentiment analysis. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 609-620. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Gayatree Ganu, Noemie Elhadad, and Amelie Marian. 2009. Beyond the stars: improving rating predictions using review text content. In WebDB, volume 9, pages 1-6. CiteSeer.
Mengting Hu, Shiwan Zhao, Li Zhang, Keke Cai, Zhong Su, Renhong Cheng, and Xiaowei Shen. 2018. CAN: constrained attention networks for multi-aspect sentiment analysis. CoRR, abs/1812.10735.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177.
Qingnan Jiang, Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang. 2019. A challenge dataset and effective models for aspect-based sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6281-6286.
Zhipeng Jin, Qiudan Li, Daniel D Zeng, YongCheng Zhan, Ruoran Liu, Lei Wang, and Hongyuan Ma. 2016. Jointly modeling review content and aspect ratings for review rating prediction. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 893-896.
Yohan Jo and Alice H Oh. 2011. Aspect and sentiment unification model for online review analysis. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 815-824. ACM.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif Mohammad. 2014. Nrc-canada-2014: Detecting aspects and sentiment in customer reviews. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 437-442.
Changliang Li, Bo Xu, Gaowei Wu, Saike He, Guanhua Tian, and Hongwei Hao. 2014. Recursive deep learning for sentiment analysis over social data. In 2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), volume 2, pages 180-185. IEEE.
Fangtao Li, Nathan Nan Liu, Hongwei Jin, Kai Zhao, Qiang Yang, and Xiaoyan Zhu. 2011. Incorporating reviewer and product information for review rating prediction. In Twenty-second international joint conference on artificial intelligence.
Junjie Li, Haitong Yang, and Chengqing Zong. 2018. Document-level multi-aspect sentiment classification by jointly modeling users, aspects, and overall ratings. In Proceedings of the 27th International Conference on Computational Linguistics, pages 925-936, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In International Joint Conference on Artificial Intelligence, pages 4068-4074.
Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165-172.
Sajad Movahedi, Erfan Ghadery, Heshaam Faili, and Azadeh Shakery. 2019. Aspect category detection via topic-attention network.
Haiyun Peng, Erik Cambria, and Amir Hussain. 2017. A review of sentiment analysis research in chinese language. Cognitive Computation, 9(4):423-435.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Maria Pontiki, Dimitrios Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment
analysis. In International Workshop on Semantic Evaluation (SemEval 2015).
Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Haris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In International Workshop on Semantic Evaluation (SemEval 2015).
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, AL-Smadi Mohammad, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 19-30.
Sebastian Ruder, Parsa Ghaffari, and John G. Breslin. 2016. A hierarchical model of reviews for aspect-based sentiment analysis.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In Advances in neural information processing systems, pages 3856-3866.
Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional skip-gram: Explicitly distinguishing left and right context for word embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 175–180.
Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. arXiv preprint arXiv:1903.09588.
Songbo Tan and Jin Zhang. 2008. An empirical study of sentiment analysis for chinese documents. Expert Systems with applications, 34(4):2622-2629.
Yequan Wang, Minlie Huang, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Empirical Methods in Natural Language Processing, pages 606-615.
Chuhan Wu, Fangzhao Wu, Junxin Liu, Yongfeng Huang, and Xing Xie. 2019a. Arp: Aspect-aware neural review rating prediction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2169-2172.
Chuhan Wu, Fangzhao Wu, Junxin Liu, Yongfeng Huang, and Xing Xie. 2019b. Arp: Aspect-aware neural review rating prediction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2169-2172.
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. arXiv preprint arXiv:1904.02232.
Taras Zagibalov and John A Carroll. 2008. Unsupervised classification of sentiment and objectivity in chinese text. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I.
Yanyan Zhao, Bing Qin, and Ting Liu. 2014. Creating a fine-grained corpus for chinese sentiment analysis. IEEE Intelligent Systems, 30(1):36-43.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 207-212.
Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2015. Representation learning for aspect category detection in online reviews. In Twenty-ninth Aaai Conference on Artificial Intelligence.