SlowGuess/ACL_2020_OCR huggingface dataset:
els. Both base and large versions of RoBERTa are used. RoBERTa<sub>base</sub> and RoBERTalarge contain 12 and 24 transformer layers respectively. As the pre-training method is quite costly in terms of GPU resources, most of the experiments are done on RoBERTa<sub>base</sub>, and only the main results report the performance on RoBERTalarge.
For SKEP, the transformer encoder is first initialized with RoBERTa, then is pre-trained on sentiment unlabeled data. An input sequence is truncated to 512 tokens. Learning rate is kept as  $5e - 5$ , and batch-size is 8192. The number of epochs is set to 3. For the fine-tuning of each dataset, we run 3 times with random seeds for each combination of parameters (Table 2), and choose the medium checkpoint for testing according to the performance on the development set.
We compare our SKEP method with the strong pretraining baseline RoBERTa and previous SOTA. The result is shown in Table 3.
Comparing with RoBERTa, SKEP significantly and consistently improves the performance on both
From	Model	Sentence Samples	Prediction
SST-2	RoBERTa	altogether, this is successful as a film, while at the same time being a most touching reconsideration of the familiar masterpiece.	positive
SKEP	altogether, this is successful as a film, while at the same time being a most touching reconsideration of the familiar masterpiece.	positive
Sem-L	RoBERTa	I got this at an amazing price from Amazon and it arrived just in time.	negative
SKEP	I got this at an amazing price from Amazon and it arrived just in time.	positive
Table 5: Visualization of chosen samples. Words above wavy underline are mean sentiment words, and words above double underlines mean aspects. Color depth denotes importance for classification. The deeper color means more importance. The color depth is calculated by the attention weights with the classification token [CLS].
Model	SST-2 dev	Sem-L	Sem-R
Sent-Vector	96.87	81.32	87.92
Pair-Vector	96.91	81.38	87.95
Table 6: Comparison of vector used for aspect-sentiment pair prediction. Sent-Vector uses sentence representation (output vector of [CLS]) for prediction, while pair-vector uses the concatenation of output vectors of the two words in a pair.
base and large settings. Even on RoBERTa $_{large}$ , SKEP achieves an improvement of up to 2.4 points. According to the task types, SKEP achieves larger improvements on fine-grained tasks, aspect-level classification and opinion role labeling, which are supposed to be more difficult than sentence-level classification. We think this owes to the aspect-sentiment knowledge that is more effective for these tasks. Interestingly, "RoBERTa $_{base}$  + SKEP" always outperforms RoBERTa $_{large}$ , except on Amazon-2. As the large version of RoBERTa is computationally expensive, the base version of SKEP provides an efficient model for application. Compared with previous SOTA, SKEP achieves new state-of-the-art results on almost all datasets, with a less satisfactory result only on SST-2.
Overall, through comparisons of various sentiment tasks, the results strongly verify the necessity of incorporating sentiment knowledge for pretraining methods, and also the effectiveness of our proposed sentiment pre-training method.
5.4 Detailed Analysis
Effect of Sentiment Knowledge SKEP uses an additional sentiment data for further pre-training and utilizes three objectives to incorporate three types of knowledge. Table 4 compares the contributions of these factors. Further pre-training with random sub-word masking of Amazon, Roberta<sub>base</sub> obtains some improvements. This proves the value
of large-size task-specific unlabeled data. However, the improvement is less evident compared with sentiment word masking. This indicates that the importance of sentiment word knowledge. Further improvements are obtained when word polarity and aspect-sentiment pair objectives are added, confirming the contribution of both types of knowledge. Compare “+SW+WP+AP” with “+Random Token”, the improvements are consistently significant in all evaluated data and is up to about 1.5 points.
Overall, from the comparison of objectives, we conclude that sentiment knowledge is helpful, and more diverse knowledge results in better performance. This also encourages us to use more types of knowledge and use better mining methods in the future.
Effect of Multi-label Optimization Multi-label classification is proposed to deal with the dependency in an aspect-sentiment pair. To confirm the necessity of capturing the dependency of words in the aspect-sentiment pair, we also compare it with the method where the token is predicted independently, which is denoted by AP-I. AP-I uses softmax for normalization, and independently predicts each word of a pair as the sentiment word prediction. According to the last line that contains AP-I in Table 4, predicting words of a pair independently do not hurt the performance of sentence-level classification. This is reasonable as the sentence-level task mainly relies on sentiment words. In contrast, in aspect-level classification and opinion role labeling, multi-label classification is efficient and yields improvement of up to 0.6 points. This denotes that multi-label classification does capture better dependency between aspect and sentiment, and also the necessity of dealing with such dependency.
Comparison of Vector for Aspect-Sentiment  
Pair Prediction SKEP utilizes the sentence rep
resentation, which is the final state of classification token [CLS], for aspect-sentiment pair prediction. We call this Sent-Vector methods. Another way is to use the concatenation of the final vectors of the two words in a pair, which we call Pair-Vector. As shown in Table 6, the performances of these two decisions are very close. We suppose this due to the robustness of the pre-training approach. As using a single vector for prediction is more efficient, we use final state of token [CLS] in SKEP.
Attention Visualization Table 5 shows the attention distribution of final layer for the [CLS] token when we adopt our SKEP model to classify the input sentences. On the SST-2 example, despite RoBERTa gives a correct prediction, its attention about sentiment is inaccurate. On the Sem-L case, RoBERTa fails to attend to the word "amazing", and produces a wrong prediction. In contrast, SKEP produces correct predictions and appropriate attention of sentiment information in both cases. This indicates that SKEP has better interpretability.
Sentiment Analysis with Knowledge Various types of sentiment knowledge, including sentiment words, word polarity, aspect-sentiment pairs, have been proved to be useful for a wide range of sentiment analysis tasks.
Sentiment words with their polarity are widely used for sentiment analysis, including sentence-level sentiment classification (Taboada et al., 2011; Shin et al., 2017; Lei et al., 2018; Barnes et al., 2019), aspect-level sentiment classification (Vo and Zhang, 2015), opinion extraction (Li and Lam, 2017), emotion analysis (Gui et al., 2017; Fan et al., 2019) and so on. Lexicon-based method (Turney, 2002; Taboada et al., 2011) directly utilizes polarity of sentiment words for classification. Traditional feature-based approaches encode sentiment word information in manually-designed features to improve the supervised models (Pang et al., 2008; Agarwal et al., 2011). In contrast, deep learning approaches enhance the embedding representation with the help of sentiment words (Shin et al., 2017), or absorb the sentiment knowledge through linguistic regularization (Qian et al., 2017; Fan et al., 2019).
Aspect-sentiment pair knowledge is also useful for aspect-level classification and opinion extraction. Previous works often provide weak supervision by this type of knowledge, either for aspect
level classification (Zeng et al., 2019) or for opinion extraction (Yang et al., 2017; Ding et al., 2017).
Although studies of exploiting sentiment knowledge have been made throughout the years, most of them tend to build a specific mechanism for each sentiment analysis task, so different knowledge is adopted to support different tasks. Whereas our method incorporates diverse knowledge in pretraining to provide a unified sentiment representation for sentiment analysis tasks.
Pre-training Approaches Pre-training methods have remarkably improved natural language processing, using self-supervised training with large scale unlabeled data. This line of research is dramatically advanced very recently, and various types of methods are proposed, including ELMO (Peters et al., 2018), GPT (Radford et al., 2018), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019) and so on. Among them, BERT pre-trains a bidirectional transformer by randomly masked word prediction, and have shown strong performance gains. RoBERTa (Liu et al., 2019) further improves BERT by robust optimization, and become one of the best pre-training methods.
Inspired by BERT, some works propose fine-grained objectives beyond random word masking. SpanBERT (Joshi et al., 2019) masks the span of words at the same time. ERNIE (Sun et al., 2019) proposes to mask entity words. On the other hand, pre-training for specific tasks is also studied. GlossBERT (Huang et al., 2019) exploits gloss knowledge to improve word sense disambiguation. SenseBERT (Levine et al., 2019) uses WordNet super-senses to improve word-in-context tasks. A different ERNIE (Zhang et al., 2019) exploits entity knowledge for entity-linking and relation classification.
In this paper, we propose Sentiment Knowledge Enhanced Pre-training for sentiment analysis. Sentiment masking and three sentiment pre-training objectives are designed to incorporate various types of knowledge for pre-training model. Thought conceptually simple, SKEP is empirically highly effective. SKEP significantly outperforms strong pre-training baseline RoBERTa, and achieves new state-of-the-art on most datasets of three typical specific sentiment analysis tasks. Our work verifies the necessity of utilizing sentiment knowledge for pre-training models, and provides a unified semi-
ment representation for a wide range of sentiment analysis tasks.
In the future, we hope to apply SKEP on more sentiment analysis tasks, to further see the generalization of SKEP, and we are also interested in exploiting more types of sentiment knowledge and more fine-grained sentiment mining methods.
We thanks Qinfei Li for her valuable comments. We also thank the anonymous reviewers for their insightful comments. This work was supported by the National Key Research and Development Project of China (No. 2018AAA0101900).
Apoory Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of twitter data. In Proceedings of the Workshop on Language in Social Media (LSM 2011).
Jeremy Barnes, Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2019. Lexicon information in neural sentiment analysis: a multi-task learning approach. In Proceedings of the 22nd Nordic Conference on Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL 2019.
Ying Ding, Jianfei Yu, and Jing Jiang. 2017. Recurrent neural networks with auxiliary labels for cross-domain opinion target extraction. In AAAI 2017.
Chuang Fan, Hongyu Yan, Jiachen Du, Lin Gui, Li-dong Bing, Min Yang, Ruifeng Xu, and Ruibin Mao. 2019. A knowledge regularized hierarchical approach for emotion cause analysis. In EMNLP 2019.
Lin Gui, Jiannan Hu, Yulan He, Ruifeng Xu, Qin Lu, and Jiachen Du. 2017. A question answering approach for emotion cause extraction. In EMNLP 2017.
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In EMNLP 2019.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In ACL 2019.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
Zhen-Zhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942.
Zeyang Lei, Yujiu Yang, Min Yang, and Yi Liu. 2018. A multi-sentiment-resource enhanced attention network for sentiment classification. In ACL 2018.
Yoav Levine, Barak Lenz, Or Dagan, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2019. Sensebert: Driving some sense into bert.
Xin Li and Wai Lam. 2017. Deep multi-task learning for aspect term extraction with memory interaction. In EMNLP 2017.
Bing Liu. 2012. Sentiment analysis and opinion mining. In Synthesis Lectures on Human Language Technologies 5.1 (2012): 1-167.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ana Marasović and Anette Frank. 2018. SRL4ORL: Improving opinion role labeling using multi-task learning with semantic role labeling. In *NAACL* 2018.
Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2):1-135.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In SemEval 2014.
Qiao Qian, Minlie Huang, Jinhao Lei, and Xiaoyan Zhu. 2017. Linguistically regularized LSTM for sentiment classification. In ACL 2017.
Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer.
Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2019. Adapt or get left behind: Domain adaptation through bert language model finetuning for aspect-target sentiment classification. ArXiv, abs/1908.11860.
Bonggun Shin, Timothy Lee, and Jinho D. Choi. 2017. Lexicon integrated CNN models with attention for sentiment analysis. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP 2013.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational linguistics, 37(2):267-307.
Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In ACL 2014.
Peter D Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In ACL 2002.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS 2017.
Duy-Tin Vo and Yue Zhang. 2015. Target-dependent twitter sentiment classification with rich automatic features. In *IJCAI* 2015.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation.
Theresa Ann Wilson. 2008. Fine-grained subjectivity and sentiment analysis: Recognizing the intensity, polarity, and attitudes of private states.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation. CoRR, abs/1904.12848.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding.
Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. ArXiv, abs/1703.06345.
Ziqian Zeng, Wenxuan Zhou, Xin Liu, and Yangqiu Song. 2019. A variational approach to weakly supervised document-level multi-aspect sentiment classification. In *NAACL* 2019.
Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS 2015.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In ACL 2019.
Pinlong Zhao, Linlin Hou, and Ou Wu. 2019. Modeling sentiment dependencies with graph convolutional networks for aspect-level sentiment classification. CoRR, abs/1906.04501.
For sentiment knowledge mining, we construct 46 sentiment seed words as follows. We first count the 9,750 items of Qian et al. (2017) on training data of Amazon-2, and get 50 most frequent sentiment words. Then, we manually filter out inappropriate words from these 50 words in a few minutes and finally get 46 sentiment words with polarities (Table 7). The filtered words are need, fun, plot and fine respectively, which are all negative words.
positive word	great, good, like, just, will, well, even, love, best, better, back, want, recommend, worth, easy, sound, right, excellent, nice, real, fun, sure, pretty, interesting, stars
negative word	too, little, bad, game, down, long, hard, waste, disappointed, problem, try, poor, less, boring, worst, trying, wrong, least, although, problems, cheap
Table 7: Sentiment seed words used in our experiment.
4076