SlowGuess/ACL_2020_OCR huggingface dataset:
outperforms RoBERTa-AE model in defining the boundary of multi-word aspect terms. Using a contextualized RoBERTa feature, the RoBERTa-AE is only able to identify the noun "cocktail" in a noun phrase, suggesting a RoBERTa representation fails to capture rich syntactical structure in a contextual sentence. In the universal dependencies schema, "Times" and "Square" are a PROPN (proper noun) tag which is part of the name of specific place, and have compound relation with the noun "cocktail". Being given explicit information about special syntactical properties of an example, CSAE successfully identifies a compound noun as an aspect term even though an aspect term "Time Square cocktail" does not appear in a training set. Additionally, even though RoBERTa-AE can identify individual aspect terms "espresso cup filled with" and "chocolate mousse" in example 2, it fails to group them together to form a complete multi-word term. CSAE, on the other hand, is able to model the role of the preposition "with" and detect the true boundary of the aspect term.
6 Results & Analysis
6.1 Aspect Extraction
6.1.1 Main Results
Table 3 summarizes the results of our proposed models compared with the baseline models. When compared with the word-embedding-based models, our CSAE model performs better than the BiLSTM and DTBCSNN models with gains of 3.93 percentage points (p.p), 1.99p.p and 5.23p.p, 2.68p.p in laptop and restaurant datasets respectively. The performance of our model is close to IMN's in laptop domain and outperforms other integrated approaches in both settings. Especially, our CSAE model has F1-score at least 3.32 p.p higher than other integrated approaches in the restaurant domain, suggesting that single-task models can significantly outperform integrated solutions with sophisticated architecture by simply improving the quality of feature representations.
6.1.2 Ablation Study
To investigate the effects of different designed components in a CSAE, we start with a base model using just a RoBERTa representation for aspect extraction and add other components one at a time. We found that our base model always gives superior performance compared to the BERT-based model. The performance is improved when we introduce
the POS embedding and dependency-based embedding to capture rich syntactical information. The POS embeddings solely represent the POS of each individual word and leave the feature extraction job for the attention layer, while the dependency-based embeddings directly infuse the grammatical interaction between words into the word representation. Hence, it is expected that RoBERTa with dependency-based features has slightly higher F1-score than RoBERTa with POS features. Overall, CSAE with full complement of both components gained significant improvement. It suggests that the RoBERTa model has not entirely "comprehended" the grammatical aspects of natural language and there is room for improvements in contextualized LM by further leveraging syntactical information of sentences.
Table 3: Comparison of our best performing AE model variants in terms of F1 scores  $(\%)$  with the state-of-the-art methods
Domain	Laptop	Rest
Model	F1	F1
Single-task	BiLSTM	73.72	81.42
DTBCSNN	75.66	83.97
BERT-AE	73.92	82.56
Integrated	MNN	76.94	83.05
UABSA	77.34	83.92
IMN	77.96	83.33
Proposed	RoBERTa-AE	75.22	85.12
RoBERTa-POS	76.01	85.56
RoBERTa-Dep	76.88	86.25
CSAE	77.65	86.65
Note: The best result in each dataset is highlighted in bold
6.2 Aspect Sentiment Classification
6.2.1 Main Results
Table 4 demonstrates that our proposed LCFS-ASC using Syntactic Relative Distance to localize the context features has the best performance in both Laptop and Restaurant dataset. The single-task, integrated and our proposed approach are displayed in the first, second and third parts, respectively. Our proposed model outperforms the BERT-PT by a large margin without utilizing additional knowledge from a larger corpus to train domain-specific embeddings. All BERT-based single-task models outperform the integrated models, suggesting that the unified tagging schema imposed overheads to the ASC tasks by introducing extra classes. As discussed in Section 3.2.1, the removal of the next-sentence-pair task in RoBERTa makes the RoBERTa representation less suitable to the ASC
Table 4: Comparison results of our best performing ASC model variants in terms of F1 scores and accuracy  $(\%)$  with the state-of-the-art methods
Domain	Laptop	Rest
Model	F1	Acc	F1	Acc
AOA	-	74.5	-	81.2
MGAN	72.47	75.39	71.94	81.25
BERT-ASC *	72.68	76.25	76.98	84.46
BERT-PT	75.08	78.07	76.96	84.95
AEN-BERT	76.31	79.93	73.76	83.12
LCF-BERT-CDW *	76.20	80.21	79.12	85.91
LCF-BERT-CDM *	75.76	79.65	78.74	85.73
MNN	65.98	70.40	68.45	77.17
UABSA	68.24	72.30	68.38	79.68
IMN	72.02	75.36	75.66	83.89
RoBERTa-ASC	70.52	74.12	75.12	82.82
LCFS-ASC-CDW	77.13	80.52	80.31	86.71
LCFS-ASC-CDM	76.45	80.34	80.10	86.13
Note: The best result in each dataset is highlighted in bold. The results of models we reproduced by following the methodology published in the paper are indicated by asterisk  $(^{*})$
task leading to the underperformance of RoBERTa-ASC.
The proposed LCFS-ASC has a slightly improved performance compared with the LCF-BERT when using either CDM or CDW. The result demonstrates the effectiveness of Syntactical Relative Distance in encoding syntactical information. CDW helps to boost the performance of LCFS-ASC model more than the CDM. Since CDM completely blocks the signals of the contexts being identified unimportant, it may falsely disregard useful signals. On the other hand, CDW emphasizes flexibility and allows further signals to contribute small weights corresponding to its relatedness with the aspect terms in the dependency-based tree.
6.2.2 Analysis of SRD's Effects by Visualizing Attention Scores
Figure 6: Attention scores of LCF-BERT-CDW (left) and LCFS-ASC-CDW (right)
Fig. 6 visualizes the attention score for the best
performing LCFS-ASC-CDW and LCF-BERT-CDW models. For a given input sentence, LCFS-ASC assigns a correct positive polarity to the aspect term "cuisine", while LCF-BERT gives a wrong prediction as negative. Since LCF-BERT uses Semantic Relative Distance, the sentiment term "without a doubt" has been paid the most focus due to its close distance to the aspect term "cuisine" based on word counts metrics. On the other hand, the signal of a key sentiment word "delicious" is mistakenly down-weighted because it is far away from the aspect term "cuisine". Nevertheless, the LCFS-ASC retains the importance of the word "delicious" because Syntactical Relative Distance accounts for the direct interaction between the adjective "delicious" and the aspect term "cuisine" in a dependency-based tree.
We proposed an end-to-end ABSA solution which pipelined an aspect extractor and an aspect sentiment classifier. The results indicate that exploitation of syntactical structures of sentences empowers the contextualized models to improve on current works in both ASC and AE tasks. Our proposed aspect sentiment classifier outperformed post-training ASC model and enabled the creation of a domain-independent solution. The proposed SRD allows the aspect sentiment classifier to focus on critical sentiment words which modify the target aspect term through dependency-based structure. The substantial improvements highlight the under-performance of recent contextualized embedding models in "understanding" syntactical features and suggests future directions in developing more syntax-learning contextualized embeddings. One can try to adapt our proposed CSAE architecture for an integrated approach by applying the unified tagging scheme; thereby, aspect extraction and sentiment classification can be achieved simultaneously.
Thanks to Vinh Hung Ngo, who has provided insightful advice to improve my writings and experimental results.
Aminu Da'u and Naomie Salim. 2019. Aspect extraction on user textual reviews using multi-channel con
volitional neural network. PeerJ Computer Science, 5:e191.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Feifan Fan, Yansong Feng, and Dongyan Zhao. 2018. Multi-grained attention network for aspect-level sentiment classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3433-3442.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. arXiv preprint arXiv:1906.06906.
Binxuan Huang, Yanglan Ou, and Kathleen M Carley. 2018. Aspect level sentiment classification with attention-over-attention neural networks. arxiv preprint arXiv:1804.06536.
Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302-308.
Xin Li, Lidong Bing, Piji Li, and Wai Lam. 2019. A unified model for opinion target extraction and target sentiment prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6714-6721.
Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Finegrained opinion mining with recurrent neural networks and word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1433-1443.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect-based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27-35.
Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2016. Aspect extraction for opinion mining with a deep convolutional neural network. Knowledge-Based Systems, 108:42-49.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper.pdf.
Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. 2019. Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314.
Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2015. Effective lstms for target-dependent sentiment classification. arXiv preprint arXiv:1512.01100.
Tun Thura Thet, Jin-Cheon Na, and Christopher SG Khoo. 2010. Aspect-based sentiment analysis of movie reviews on discussion boards. Journal of information science, 36(6):823-848.
Feixiang Wang, Man Lan, and Wenting Wang. 2018. Towards a one-stop solution to both aspect extraction and sentiment analysis tasks with neural multitask learning. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 606-615.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2018. Double embeddings and CNN-based sequence labeling for aspect extraction. arXiv preprint arXiv:1805.04601.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics.
Hai Ye, Zichao Yan, Zhunchen Luo, and Wenhan Chao. 2017. Dependency-tree based convolutional neural networks for aspect term extraction. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 350-362. Springer.
Biqing Zeng, Heng Yang, Ruyang Xu, Wu Zhou, and Xuli Han. 2019. Lcf: A local context focus mechanism for aspect-based sentiment classification. Applied Sciences, 9(16):3389.