SlowGuess/ACL_2020_OCR huggingface dataset:
Hao Tian\*,†, Can Gao†, Xinyan Xiao†, Hao Liu†, Bolei He†, Hua Wu†, Haifeng Wang†, Feng Wu‡
†Baidu Inc., Beijing, China ‡University of Science and Technology of China
{tianhao,gaocan01,xiaoxinyan,liuhao24,hebolei,wu_hua, wanghaifeng}@baidu.com, fengwu@ustc.edu.cn
Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowledge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta.
Sentiment analysis refers to the identification of sentiment and opinion contained in the input texts that are often user-generated comments. In practice, sentiment analysis involves a wide range of specific tasks (Liu, 2012), such as sentence-level sentiment classification, aspect-level sentiment classification, opinion extraction and so on. Traditional methods often study these tasks separately and design specific models for each task, based on manually-designed features (Liu, 2012) or deep learning (Zhang et al., 2018).
Recently, pre-training methods (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019;
Yang et al., 2019) have shown their powerfulness in learning general semantic representations, and have remarkably improved most natural language processing (NLP) tasks like sentiment analysis. These methods build unsupervised objectives at word-level, such as masking strategy (Devlin et al., 2019), next-word prediction (Radford et al., 2018) or permutation (Yang et al., 2019). Such word-prediction-based objectives have shown great abilities to capture dependency between words and syntactic structures (Jawahar et al., 2019). However, as the sentiment information of a text is seldom explicitly studied, it is hard to expect such pre-trained general representations to deliver optimal results for sentiment analysis (Tang et al., 2014).
Sentiment analysis differs from other NLP tasks in that it deals mainly with user reviews other than news texts. There are many specific sentiment tasks, and these tasks usually depend on different types of sentiment knowledge including sentiment words, word polarity and aspect-sentiment pairs. The importance of these knowledge has been verified by tasks at different level, for instance, sentence-level sentiment classification (Taboada et al., 2011; Shin et al., 2017; Lei et al., 2018), aspect-level sentiment classification (Vo and Zhang, 2015; Zeng et al., 2019), opinion extraction (Li and Lam, 2017; Gui et al., 2017; Fan et al., 2019) and so on. Therefore, we assume that, by integrating these knowledge into the pre-training process, the learned representation would be more sentiment-specific and appropriate for sentiment analysis.
In order to learn a unified sentiment representation for multiple sentiment analysis tasks, we propose Sentiment Knowledge Enhanced Pre-training (SKEP), where sentiment knowledge about words, polarity, and aspect-sentiment pairs are included to guide the process of pre-training. The sentiment knowledge is first automatically mined from unlabeled data (Section 3.1). With the knowledge
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4067-4076 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Sentiment Knowledge Enhanced Pre-training (SKEP). SKEP contains two parts: (1) Sentiment masking recognizes the sentiment information of an input sequence based on automatically-mined sentiment knowledge, and produces a corrupted version by removing these informations. (2) Sentiment pre-training objectives require the transformer to recover the removed information from the corrupted version. The three prediction objectives on top are jointly optimized: Sentiment Word (SW) prediction (on  $\mathbf{x}_9$ ), Word Polarity (SP) prediction (on  $\mathbf{x}_6$  and  $\mathbf{x}_9$ ), Aspect-Sentiment pairs (AP) prediction (on  $\mathbf{x}_1$ ). Here, the smiley denotes positive polarity. Notably, on  $\mathbf{x}_6$ , only SP is calculated without SW, as its original word has been predicted in the pair prediction on  $\mathbf{x}_1$ .
mined, sentiment masking (Section 3.2) removes sentiment information from input texts. Then, the pre-training model is trained to recover the sentiment information with three sentiment objectives (Section 3.3).
SKEP integrates different types of sentiment knowledge together and provides a unified sentiment representation for various sentiment analysis tasks. This is quite different from traditional sentiment analysis approaches, where different types of sentiment knowledge are often studied separately for specific sentiment tasks. To the best of our knowledge, this is the first work that has tackled sentiment-specific representation during pretraining. Overall, our contributions are as follows:
- We propose sentiment knowledge enhanced pre-training for sentiment analysis, which provides a unified sentiment representation for multiple sentiment analysis tasks.
- Three sentiment knowledge prediction objectives are jointly optimized during pre-training so as to embed sentiment words, polarity, aspect-sentiment pairs into the representation. In particular, the pair prediction is converted into multi-label classification to capture the dependency between aspect and sentiment.
- SKEP significantly outperforms the strong pre-training methods RoBERTa (Liu et al., 2019) on three typical sentiment tasks, and achieves new state-of-the-art results on most of the test datasets.
2 Background: BERT and RoBERTa
BERT (Devlin et al., 2019) is a self-supervised representation learning approach for pre-training a deep transformer encoder (Vaswani et al., 2017). BERT constructs a self-supervised objective called masked language modeling (MLM) to pre-train the transformer encoder, and relies only on large-size unlabeled data. With the help of pre-trained transformer, downstream tasks have been substantially improved by fine-tuning on task-specific labeled data. We follow the method of BERT to construct masking objectives for pre-training.
BERT learns a transformer encoder that can produce a contextual representation for each token of input sequences. In reality, the first token of an input sequence is a special classification token [CLS]. In fine-tuning step, the final hidden state of [CLS] is often used as the overall semantic representation of the input sequence.
In order to train the transformer encoder, MLM is proposed. Similar to doing a cloze test, MLM predicts the masked token in a sequence from their placeholder. Specifically, parts of input tokens are randomly sampled and substituted. BERT uniformly selects  $15\%$  of input tokens. Of these sampled tokens,  $80\%$  are replaced with a special masked token [MASK],  $10\%$  are replaced with a random token,  $10\%$  are left unchanged. After the construction of this noisy version, the MLM aims to predict the original tokens in the masked positions using the corresponding final states.
Most recently, RoBERTa (Liu et al., 2019) significantly outperforms BERT by robust opti
mization without the change of neural structure, and becomes one of the best pre-training models. RoBERTa also removes the next sentence prediction objective from standard BERT. To verify the effectiveness of our approach, this paper uses RoBERTa as a strong baseline.
3 SKEP: Sentiment Knowledge Enhanced Pre-training
We propose SKEP, Sentiment Knowledge Enhanced Pre-training, which incorporates sentiment knowledge by self-supervised training. As shown in Figure 1, SKEP contains sentiment masking and sentiment pre-training objectives. Sentiment masking (Section 3.2) recognizes the sentiment information of an input sequence based on automatically-mined sentiment knowledge (Section 3.1), and produces a corrupted version by removing this information. Three sentiment pre-training objectives (Section 3.3) require the transformer to recover the sentiment information for the corrupted version.
Formally, sentiment masking constructs a corrupted version  $\widetilde{X}$  for an input sequence  $X$  guided by sentiment knowledge  $\mathcal{G}$ .  $x_{i}$  and  $\widetilde{x}_i$  denote the  $i$ -th token of  $X$  and  $\widetilde{X}$  respectively. After masking, a parallel data  $(\widetilde{X}, X)$  is obtained. Thus, the transformer encoder can be trained with sentiment pre-training objectives that are supervised by recovering sentiment information using the final states of encoder  $\widetilde{\mathbf{x}}_1, \ldots, \widetilde{\mathbf{x}}_n$ .
3.1 Unsupervised Sentiment Knowledge Mining
SKEP mines the sentiment knowledge from unlabeled data. As sentiment knowledge has been the central subject of extensive research, SKEP finds a way to integrate former technique of knowledge mining with pre-training. This paper uses a simple and effective mining method based on Pointwise Mutual Information (PMI) (Turney, 2002).
PMI method depends only on a small number of sentiment seed words and the word polarity  $\mathrm{WP}(s)$  of each seed word  $s$  is given. It first builds a collection of candidate word-pairs where each word-pair contains a seed word, and meet with pre-defined part-of-speech patterns as Turney (2002). Then, the co-occurrence of a word-pair is calculated by PMI as follows:
$$
\operatorname {P M I} \left(w _ {1}, w _ {2}\right) = \log \frac {p \left(w _ {1} , w _ {2}\right)}{p \left(w _ {1}\right) p \left(w _ {2}\right)} \tag {1}
$$
Here,  $p(.)$  denotes probability estimated by count. Finally, the polarity of a word is determined by the difference between its PMI scores with all positive seeds and that with all negative seeds.
$$
\begin{array}{l} \mathrm {W P} (w) = \sum_ {\mathrm {W P} (s) = +} \mathrm {P M I} (w, s) \tag {2} \\ - \sum_ {\mathrm {W P} (s) = -} \mathrm {P M I} (w, s) \\ \end{array}
$$
If  $\mathrm{WP}(w)$  of a candidate word  $w$  is larger than 0, then  $w$  is a positive word, otherwise it is negative.
After mining sentiment words, aspect-sentiment pairs are extracted by simple constraints. An aspect-sentiment pair refers to the mention of an aspect and its corresponding sentiment word. Thus, a sentiment word with its nearest noun will be considered as an aspect-sentiment pair. The maximum distance between the aspect word and the sentiment word of a pair is empirically limited to no more than 3 tokens.
Consequently, the mined sentiment knowledge  $\mathcal{G}$  contains a collection of sentiment words with their polarity along with a set of aspect-sentiment pairs. Our research focuses for now the necessity of integrating sentiment knowledge in pre-training by virtue of a relatively common mining method. We believe that a more fine-grained method would further improve the quality of knowledge, and this is something we will be exploring in the nearest future.
3.2 Sentiment Masking
Sentiment masking aims to construct a corrupted version for each input sequence where sentiment information is masked. Our sentiment masking is directed by sentiment knowledge, which is quite different from previous random word masking. This process contains sentiment detection and hybrid sentiment masking that are as follows.
Sentiment Detection with Knowledge Sentiment detection recognizes both sentiment words and aspect-sentiment pairs by matching input sequences with the mined sentiment knowledge  $\mathcal{G}$ .
1. Sentiment Word Detection. The word detection is straightforward. If a word of an input sequence also occurs in the knowledge base  $\mathcal{G}$ , then this word is seen as a sentiment word.
2. Aspect-Sentiment Pair Detection. The detection of an aspect-sentiment pair is similar to
its mining described before. A detected sentiment word and its nearby noun word are considered as an aspect-sentiment pair candidate, and the maximum distance of these two words is limited to 3. Thus, if such a candidate is also found in mined knowledge  $\mathcal{G}$ , then it is considered as an aspect-sentiment pair.
Hybrid Sentiment Masking Sentiment detection results in three types of tokens for an input sequence: aspect-sentiment pairs, sentiment words and common tokens. The process of masking a sequence runs in following steps:
1. Aspect-sentiment Pair Masking. At most 2 aspect-sentiment pairs are randomly selected to mask. All tokens of a pair are replaced by [MASK] simultaneously. This masking provides a way for capturing the combination of an aspect word and a sentiment word.
2. Sentiment Word Masking. For those un-masked sentiment words, some of them are randomly selected and all the tokens of them are substituted with [MASK] at the same time. The total number of tokens masked in this step is limited to be less than  $10\%$ .
3. Common Token Masking. If the number of tokens in step 2 is insufficient, say less than  $10\%$ , this would be filled during this step with randomly-selected tokens. Here, random token masking is the same as RoBERTa. $^{1}$
3.3 Sentiment Pre-training Objectives
Sentiment masking produces corrupted token sequences  $\widetilde{X}$ , where their sentiment information is substituted with masked tokens. Three sentiment objectives are defined to tell the transformer encoder to recover the replaced sentiment information. The three objectives, Sentiment Word (SW) prediction  $L_{sw}$ , Word Polarity (WP) prediction  $L_{wp}$  and Aspect-sentiment Pair (AP) prediction  $L_{ap}$  are jointly optimized. Thus, the overall pretraining objective  $L$  is:
$$
L = L _ {s w} + L _ {w p} + L _ {a p} \tag {3}
$$
Sentiment Word Prediction Sentiment word prediction is to recover the masked tokens of sentiment words using the output vector  $\widetilde{\mathbf{x}}_i$  from transformer encoder.  $\widetilde{\mathbf{x}}_i$  is fed into an output softmax layer, which produces a normalized probability vector  $\hat{y}_i$  over the entire vocabulary. In this way, the sentiment word prediction objective  $L_{sw}$  is to maximize the probability of original sentiment word  $x_i$  as follows:
$$
\hat {\mathbf {y}} _ {i} = \operatorname {s o f t m a x} \left(\widetilde {\mathbf {x}} _ {i} \mathbf {W} + \mathbf {b}\right) \tag {4}
$$
$$
L _ {s w} = - \sum_ {i = 1} ^ {i = n} m _ {i} \times \mathbf {y} _ {i} \log \hat {\mathbf {y}} _ {i} \tag {5}
$$
Here,  $\mathbf{W}$  and  $\mathbf{b}$  are the parameters of the output layer.  $m_{i} = 1$  if  $i$ -th position of a sequence is masked sentiment word $^2$ , otherwise it equals to 0.  $\mathbf{y}_{i}$  is the one-hot representation of the original token  $x_{i}$ .
Regardless of a certain similarity to MLM of BERT, our sentiment word prediction has a different purpose. Instead of predicting randomly masking tokens, this sentiment objective selects those sentiment words for self-supervision. As sentiment words play a key role in sentiment analysis, the representation learned here is expected to be more suitable for sentiment analysis.
Word Polarity Prediction Word polarity is crucial for sentiment analysis. For example, traditional lexicon-based model (Turney, 2002) directly utilizes word polarity to classify the sentiment of texts. To incorporate this knowledge into the encoder, an objective called word polarity prediction  $L_{wp}$  is further introduced.  $L_{wp}$  is similar to  $L_{sw}$ . For each masked sentiment token  $\widetilde{x}_i$ ,  $L_{wp}$  calculated its polarity (positive or negative) using final state  $\widetilde{\mathbf{x}}_i$ . Then the polarity of target corresponds to the polarity of the original sentiment word, which can be found from the mined knowledge.
Aspect-sentiment Pair Prediction Aspect sentiment pairs reveal more information than sentiment words do. Therefore, in order to capture the dependency between aspect and sentiment, an aspect-sentiment pair objective is proposed. Especially, words in a pair are not mutually exclusive. This is quite different from BERT, which assumes tokens can be independently predicted.
2In sentiment masking, we add common tokens to make up for the deficiency of masked tokens of sentiment words.  $L_{sw}$  also calculates these common tokens, while  $L_{wp}$  does not includes them.
For each sentence, we would always in total mask  $10\%$  of its tokens at step 2 and 3. Among these masked tokens,  $79.9\%$  are sentiment words (during step 2) and  $20.1\%$  are common words (during step 3) in our experiment.
We thus conduct aspect-sentiment pair prediction with multi-label classification. We use the final state of classification token [CLS], which denotes representation of the entire sequence, to predict pairs. sigmoid activation function is utilized, which allows multiple tokens to occur in the output at the same time. The aspect-sentiment pair objective  $L_{ap}$  is denoted as follows:
$$
\begin{array}{l} \hat {\mathbf {y}} _ {a} = \mathrm {s i g m o i d} (\widetilde {\mathbf {x}} _ {1} \mathbf {W} _ {a p} + \mathbf {b} _ {a p}) \qquad (6) \\ L _ {a p} = - \sum_ {a = 1} ^ {a = A} \mathbf {y} _ {a} \log \hat {\mathbf {y}} _ {a} \tag {7} \\ \end{array}
$$
Here,  $\mathbf{x}_1$  denotes the output vector of [CLS].  $A$  is the number of masked aspect-sentiment pairs in a corrupted sequence.  $\hat{\mathbf{y}}_a$  is the word probability normalized by sigmoid.  $\mathbf{y}_a$  is the sparse representation of a target aspect-sentiment pair. Each element of  $\mathbf{y}_a$  corresponds to one token of the vocabulary, and equals to 1 if the target aspect-sentiment pair contains the corresponding token. As there are multiple elements of  $\mathbf{y}_a$  equals to 1, the predication here is multi-label classification.
4 Fine-tuning for Sentiment Analysis
We verify the effectiveness of SKEP on three typical sentiment analysis tasks: sentence-level sentiment classification, aspect-level sentiment classification, and opinion role labeling. On top of the pre-trained transformer encoder, an output layer is added to perform task-specific prediction. The neural network is then fine-tuned on task-specific labeled data.
Sentence-level Sentiment Classification This task is to classify the sentiment polarity of an input sentence. The final state vector of classification token [CLS] is used as the overall representation of an input sentence. On top of the transformer encoder, a classification layer is added to calculate the sentiment probability based on the overall representation.
Aspect-level Sentiment Classification This task aims to analyze fine-grained sentiment for an aspect when given a contextual text. Thus, there are two parts in the input: aspect description and
Dataset	Train	Dev	Test
SST-2	67k	872	1821
Amazon-2	3.2m	400k	400k
Sem-R	3608	-	1120
Sem-L	2328	-	638
MPQA2.0	287	100	95
Table 1: Numbers of samples for each dataset. Sem-R and Sem-L refer to restaurant and laptop parts of SemEval 2014 Task 4.
Dataset	Learning Rate	Batch	Epoch
SST-2	1e-5, 2e-5, 3e-5	16, 32	10
Amazon-2	2e-5, 5e-5	4	3
Sem-R	3e-5	16	5
Sem-L	3e-5	16	5
MPQA2.0	3e-5	16	5
Table 2: Hyper-parameters for fine-tuning on each dataset. Batch and Epoch indicate batch size and maximum epoch respectively.
contextual text. These two parts are combined with a separator [SEP], and fed into the transformer encoder. This task also utilizes the final state of the first token [CLS] for classification.
Opinion Role Labeling This task is to detect fine-grained opinion, such as holder and target, from input texts. Following SRL4ORL (Marasovic and Frank, 2018), this task is converted into sequence labeling, which uses BIOS scheme for labeling, and a CRF-layer is added to predict the labels.
5.1 Dataset and Evaluation
A variety of English sentiment analysis datasets are used in this paper. Table 1 summarizes the statistics of the datasets used in the experiments. These datasets contain three types of tasks: (1) For sentence-level sentiment classification, Standford Sentiment Treebank (SST-2) (Socher et al., 2013) and Amazon-2 (Zhang et al., 2015) are used. In Amazon-2,  $400k$  of the original training data are reserved for development. The performance is evaluated in terms of accuracy. (2) Aspect-level sentiment classification is evaluated on Semantic Eval
5 All the pretraining models, including our SKEP and baselines use CRF-Layer here, thus their performances are comparable.
This means that the dimension of  $\mathbf{y}_a$  equals to the vocabulary size of pre-training method, which is 50265 in our experiment.
<sup>4</sup>It is possible to predict masked pairs with CRF-layer. However, it is more than 10-times slower than multi-label classification, thus could not be used in pre-training.
Model	Sentence-Level	Aspect-Level	Opinion Role
SST-2	Amazon-2	Sem-L	Sem-R	MPQA-Holder	MPQA-Target
Previous SOTA	97.11*	97.372	81.353	87.894	83.67/77.125	81.59/73.165
RoBERTa(base)	94.9	96.61	78.11	84.93	81.89/77.34	80.23/72.19
RoBERTa(base) + SKEP	96.7	96.94	81.32	87.92	84.25/79.03	82.77/74.82
RoBERTaLarge	96.5	97.33	79.22	85.88	83.52/78.59	81.74/75.87
RoBERTaLarge + SKEP	97.0	97.56	81.47	88.01	85.77/80.99	83.59/77.41
Table 3: Comparison with RoBERTa and previous SOTA. For MPQA, here reports both binary-F1 and prop-F1 as (Marasović and Frank, 2018), which are split by a slash. The scores of previous SOTA come from:  ${}^{1}$  (Raffel et al., 2019; Lan et al., 2019);  ${}^{2}$  (Xie et al., 2019);  ${}^{3}$  (Zhao et al., 2019);  ${}^{4}$  (Rietzler et al., 2019);  ${}^{5}$  (Marasović and Frank, 2018). The SOTA score of SST-2 is from GLUE leaderboard (Wang et al., 2018) on December 1, 2019, and the system is based on ensemble-model.
Model	Sentence-Level	Aspect-Level	Opinion Role
SST-2 dev	Amazon-2	Sem-L	Sem-R	MPQA-Holder	MPQA-Target
RoBERTa base	95.21	96.61	78.11	84.93	81.89/77.34	80.23/72.19
+ Random Token	95.57	96.73	78.89	85.77	82.71/77.71	80.86/73.01
+ SW	96.38	96.82	80.13	86.92	82.95/77.63	81.18/73.15
+ SW + WP	96.51	96.87	80.32	87.25	82.97/77.82	81.09/73.24
+ SW + WP + AP	96.87	96.94	81.32	87.92	84.25/79.03	82.77/74.82
+ SW + WP + AP-I	96.89	96.93	81.19	87.71	84.01/78.36	82.69/74.36
Table 4: Effectiveness of objectives. SW, WP, AP refers to pre-training objectives: Sentiment Word prediction, Word Polarity prediction and Aspect-sentiment Pair prediction. "Random Token" denotes random token masking used in RoBERTa. AP-I denotes predicting words in an Aspect-sentiment Pair Independently.
2014 Task4 (Pontiki et al., 2014). This task contains both restaurant domain and laptop domain, whose accuracy is evaluated separately. (3) For opinion role labeling, MPQA 2.0 dataset (Wiebe et al., 2005; Wilson, 2008) is used. MPQA aims to extract the targets or the holders of the opinions. Here we follow the method of evaluation in SRL4ORL (Marasovic and Frank, 2018), which is released and available online. 4 folder cross-validation is performed, and the F-1 scores of both holder and target are reported.
To perform sentiment pre-training of SKEP, the training part of Amazon-2 is used, which is the largest dataset among the list in Table 1. Notably, the pre-training only uses raw texts without any sentiment annotation. To reduce the dependency on manually-constructed knowledge and provide SKEP with the least supervision, we only use 46 sentiment seed words. Please refers to the appendix for more details about seed words.
5.2 Experiment Setting
We use RoBERTa (Liu et al., 2019) as our baseline, which is one of the best pre-training mod