SlowGuess/ACL_2020_OCR huggingface dataset:
Da Yin\*, Tao Meng\*, Kai-Wei Chang\*
$\clubsuit$  Peking University
wade_yin9712@pku.edu.cn, tmeng@cs.ucla.edu, kw@kwchang.net
We propose SentiBERT, a variant of BERT that effectively captures compositional sentiment semantics. The model incorporates contextualized representation with binary constituency parse tree to capture semantic composition. Comprehensive experiments demonstrate that SentiBERT achieves competitive performance on phrase-level sentiment classification. We further demonstrate that the sentiment composition learned from the phrase-level annotations on SST can be transferred to other sentiment analysis tasks as well as related tasks, such as emotion classification tasks. Moreover, we conduct ablation studies and design visualization methods to understand SentiBERT. We show that SentiBERT is better than baseline approaches in capturing negation and the contrastive relation and model the compositional sentiment semantics.
Sentiment analysis is an important language processing task (Pang et al., 2002, 2008; Liu, 2012). One of the key challenges in sentiment analysis is to model compositional sentiment semantics. Take the sentence "Frenetic but not really funny." in Figure 1 as an example. The two parts of the sentence are connected by "but", which reveals the change of sentiment. Besides, the word "not" changes the sentiment of "really funny". These types of negation and contrast are often difficult to handle when the sentences are complex (Socher et al., 2013; Tay et al., 2018; Xu et al., 2019).
In general, the sentiment of an expression is determined by the meaning of tokens and phrases and the way how they are syntactically combined. Prior studies consider explicitly modeling compositional sentiment semantics over constituency structure with recursive neural networks (Socher et al., 2012,
Figure 1: Illustration of the challenges of learning sentiment semantic compositionality. The blue nodes represent token nodes. The colors of phrase nodes in the binary constituency tree represent the sentiment of phrases. The red boxes show that the sentiment changes from the child node to the parent node due to negation and contrast.
2013). However, these models that generate representation of a parent node by aggregating the local information from child nodes, overlook the rich association in context.
In this paper, we propose SentiBERT to incorporate recently developed contextualized representation models (Devlin et al., 2019; Liu et al., 2019) with the recursive constituency tree structure to better capture compositional sentiment semantics. Specifically, we build a simple yet effective attention network for composing sentiment semantics on top of BERT (Devlin et al., 2019). During training, we follow BERT to capture contextual information by masked language modeling. In addition, we instruct the model to learn composition of meaning by predicting sentiment labels of the phrase nodes.
Results on phrase-level sentiment classification on Stanford Sentiment Treebank (SST) (Socher et al., 2013) indicate that SentiBERT improves significantly over recursive networks and the base
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3695-3706 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: The architecture of SentiBERT. Module I is the BERT encoder; Module II denotes the semantic composition module based on an attention mechanism; Module III is a predictor for phrase-level sentiment. The semantic composition module is a two layer attention-based network (see Section 3.1) The first layer (Attention to Tokens) generates representation for each phrase based on the token it covers and the second layer (Attention to Children) refines the phrase representation obtained from the first layer based on its children.
line BERT model. As phrase-level sentiment labels are expensive to obtain, we further explore if the compositional sentiment semantics learned from one task can be transferred to others. In particular, we find that SentiBERT trained on SST can be transferred well to other related tasks such as twitter sentiment analysis (Rosenthal et al., 2017) and emotion intensity classification (Mohammad et al., 2018) and contextual emotion detection (Chatterjee et al., 2019). Furthermore, we conduct comprehensive quantitative and qualitative analyses to evaluate the effectiveness of SentiBERT under various situations and to demonstrate the semantic compositionality captured by the model. The source code is available at https://github.com/WadeYin9712/SentiBERT.
Sentiment Analysis Various approaches have been applied to build a sentiment classifier, including feature-based methods (Hu and Liu, 2004; Pang and Lee, 2004), recursive neural networks (Socher et al., 2012, 2013; Tai et al., 2015), convolution neural networks (Kim, 2014) and recurrent neural networks (Liu et al., 2015). Recently, pretrained language models such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019) and SentiLR (Ke et al., 2019) achieve high performance in sentiment analysis by constructing contextualized representation. Inspired by these prior studies, we design a transformer-based neural network model to capture compositional sentience semantics by
leveraging binary constituency parse tree.
Semantic Compositionality Semantic composition (Pelletier, 1994) has been widely studied in NLP literature. For example, Mitchell and Lapata (2008) introduce operations such as addition or element-wise product to model compositional semantics. The idea of modeling semantic composition is applied to various areas such as sentiment analysis (Socher et al., 2013; Zhu et al., 2016), semantic relatedness (Marelli et al., 2014) and capturing sememe knowledge (Qi et al., 2019). In this paper, we demonstrate that the syntactic structure can be combined with contextualized representation such that the semantic compositionality can be better captured. Our approach resembles to a few recent attempts (Harer et al., 2019; Wang et al., 2019) to integrate tree structures into self-attention. However, our design is specific for the semantic composition in sentiment analysis.
We introduce SentiBERT, a model that captures compositional sentiment semantics based on constituency structures of sentences. SentiBERT consists of three modules: 1) BERT; 2) a semantic composition module based on an attention network; 3) phrase and sentence sentiment predictors. The three modules are illustrated in Figure 2 and we provide an overview in below.
BERT We incorporate BERT (Devlin et al., 2019) as the backbone to generate contextualized
representation of input sentence.
Semantic Composition Module This module aims to obtain effective phrase representation guided by the contextualized representation and constituency parsing tree. To refine phrase representation based on the structural information and its constituencies, we design a two-level attention mechanism: 1) Attention to Tokens and 2) Attention to Children.
Phrase Node Prediction SentiBERT is supervised by phrase-level sentiment labels. We use cross-entropy as the loss function for learning the sentiment predictor.
3.1 Attention Networks for Sentiment Semantic Composition
In this section, we describe the attention networks for sentiment semantic composition in detail.
We first introduce the notations.  $s = [w_1, w_2, \dots, w_n]$  denotes a sentence which consists of  $n$  words.  $\mathrm{phr} = [\mathrm{phr}_1, \mathrm{phr}_2, \dots, \mathrm{phr}_m]$  denotes the phrases on the binary constituency tree of sentence s.  $\mathbf{h} = [\mathbf{h}_1, \mathbf{h}_2, \dots, \mathbf{h}_n]$  is the contextualized representation of tokens after forwarding to a fully-connected layer, where  $\mathbf{h}_t \in \mathbb{R}^d$ . Suppose  $st_i$  and  $en_i$  are beginning and end indices of the  $i$ -th phrase where  $w_{st_i}, w_{st_i+1}, \dots, w_{en_i}$  are constituent tokens of the  $i$ -th phrase. The corresponding token representation is  $[\mathbf{h}_{st_i}, \mathbf{h}_{st_i+1}, \dots, \mathbf{h}_{en_i}]$ .  $\mathbf{p}_i$  is the phrase representation of the  $i$ -th phrase.
Attention to Tokens Given the contextualized representations of the tokens covered by a phrase. We first generate phrase representation  $\mathbf{v}_i$  for a phrase  $i$  by the following attention network.
$$
\begin{array}{l} \mathbf {q} _ {i} = \frac {1}{e n _ {i} - s t _ {i} + 1} \sum_ {j = s t _ {i}} ^ {e n _ {i}} \mathbf {h} _ {j}, \\ t _ {j} = \operatorname {A t t e n t i o n} \left(\mathbf {q} _ {i}, \mathbf {h} _ {j}\right), s t _ {i} \leq j \leq e n _ {i}, \\ a _ {j} = \frac {\exp \left(t _ {j}\right)}{\sum_ {k = s t _ {i}} ^ {e n _ {i}} \exp \left(t _ {k}\right)}, \tag {1} \\ \mathbf {o} _ {i} = \sum_ {j = s t _ {i}} ^ {e n _ {i}} a _ {j} \cdot \mathbf {h} _ {j}. \\ \end{array}
$$
In Eq. (1), we first treat the averaged representation for each token as the query, and then allocate attention weights according to the correlation with each token.  $a_{j}$  represents the weight distributed to the  $j$ -th token. We concatenate the weighted sum  $\mathbf{o}_i$  and  $\mathbf{q}_i$  and feed it to forward networks.
Lastly, we obtain the initial representation for the phrase  $\mathbf{v}_i\in \mathbb{R}^d$  based on the representation of constituent tokens. The detailed computation of attention mechanism is shown in Appendix A.1.
Attention to Children Furthermore, we refine phrase representations in the second layer based on constituency parsing tree and the representations obtained in the first layer. To aggregate information based on hierarchical structure, we develop the following network. For each phrase, the attention network computes correlation with its children in the binary constituency parse tree and itself.
Assume that the indices of child nodes of the  $i$ -th phrase are  $lson$  and  $rson$ . Their representations generated from the first layer are  $\mathbf{v}_i$ ,  $\mathbf{v}_{lson}$ , and  $\mathbf{v}_{rson}$ , respectively. We generate the attention weights  $r_{lson}$ ,  $r_{rson}$  and  $r_i$  over the  $i$ -th phrase and its left and right children by the following.
$$
\begin{array}{l} c _ {l s o n} = \operatorname {A t t e n t i o n} (\mathbf {v} _ {i}, \mathbf {v} _ {l s o n}), \\ c _ {r s o n} = \operatorname {A t t e n t i o n} \left(\mathbf {v} _ {i}, \mathbf {v} _ {r s o n}\right), \tag {2} \\ c _ {i} = \operatorname {A t t e n t i o n} (\mathbf {v} _ {i}, \mathbf {v} _ {i}), \\ r _ {l s o n}, r _ {r s o n}, r _ {i} = \operatorname {S o f t m a x} \left(c _ {l s o n}, c _ {r s o n}, c _ {i}\right). \\ \end{array}
$$
Then the refined representation of phrase  $i$  is computed by
$$
\mathbf {f} _ {i} = r _ {l s o n} \cdot \mathbf {v} _ {l s o n} + r _ {r s o n} \cdot \mathbf {v} _ {r s o n} + r _ {i} \cdot \mathbf {v} _ {i}.
$$
Finally, we concatenate the weighted sum  $\mathbf{f}_i$  and  $\mathbf{v}_i$  and feed it to forward networks with SeLU (Klambauer et al., 2017) and GeLU activations (Hendrycks and Gimpel, 2017) and layer normalization (Ba et al., 2016), similar to Joshi et al. (2020) to generate the final phrase representation  $\mathbf{p}_i\in \mathbb{R}^d$ . Note that when the child of  $i$ -th phrase is token node, the attention mechanism will attend to the representation of all the subtokens the token node covers.
3.2 Training Objective of SentiBERT
Inspired by BERT, the training objective of SentiBERT consists of two parts: 1) Masked Language Modeling. Some texts are masked and the model learn to predict them. This objective allows the model learn to capture the contextual information as in the original BERT model. 2) Phrase Node Prediction. We further consider training the model to predict the phrase-level sentiment label based on the aforementioned phrase representations. This allows SentiBERT lean to capture the compositional sentiment semantics. Similar to BERT, in the
transfer learning setting, pre-trained SentiBERT model can be used to initialize the model parameters of a downstream model.
We evaluate SentiBERT on the SST dataset. We then evaluate SentiBERT in a transfer learning setting and demonstrate that the compositional sentiment semantics learned on SST can be transferred to other related tasks.
We evaluate how effective SentiBERT captures the compositional sentiment semantics on SST dataset (Socher et al., 2013).
The SST dataset has several variants.
- SST-phrase is a 5-class classification task that requires to predict the sentiment of all phrases on a binary constituency tree. Different from Socher et al. (2013), we test the model only on phrases (non-terminal constituents) and ignore its performance on tokens.
- SST-5 is a 5-class sentiment classification task that aims at predicting the sentiment of a sentence. We use it to test if SentiBERT learns a better sentence representation through capturing compositional sentiment semantics.
- Similar to SST-5, SST-2 and SST-3 are 2-class and 3-class sentiment classification tasks. However, the granularity of the sentiment classes is different.
Besides, to test the transferability of SentiBERT, we consider several related datasets, including Twitter Sentiment Analysis (Rosenthal et al., 2017), Emotion Intensity Classification (Mohammad et al., 2018) and Contextual Emotion Detection (EmoContext) (Chatterjee et al., 2019). Details are shown in Appendix A.2.
We build SentiBERT on the HuggingFace library and initialize the model parameters using pre-trained BERT-base and RoBERTa-base models whose maximum length is 128, layer number is 12, and embedding dimension is 768. For the training on SST-phrase, the learning rate is  $2 \times 10^{-5}$ , batch size is 32 and the number of training epochs is 3. For masking mechanism, to put emphasis on
modeling sentiments, the probability of masking opinion words which can be retrieved from SentiWordNet (Baccianella et al., 2010) is set to  $20\%$ , and for the other words, the probability is  $15\%$ . For fine-tuning on downstream tasks, the learning rate is  $\{1\times 10^{-5} - 1\times 10^{-4}\}$ , batch size is  $\{16,32\}$  and the number of training epochs is  $1 - 5$ . We use Stanford CoreNLP API (Manning et al., 2014) to obtain binary constituency trees for the sentences of these tasks to keep consistent with the settings on SST-phrase. Note that when fine-tuning on sentence-level sentiment and emotion classification tasks, the objective is to correctly label the root of tree, instead of targeting at the [CLS] token representation as in the original BERT.
4.2 Effectiveness of SentiBERT
We first compare the proposed attention networks (SentiBERT w/o BERT) with the following baseline models trained on SST-phrase corpus to evaluate the effectiveness of the architecture design: 1) Recursive NN (Socher et al., 2013); 2) GCN (Kipf and Welling, 2017); 3) Tree-LSTM (Tai et al., 2015); 4) BiLSTM (Hochreiter and Schmidhuber, 1997) w/ Tree-LSTM. To further understand the effect of using contextualized representation, we compare SentiBERT with the vanilla pre-trained BERT and its variants which combine the four mentioned baselines and BERT. The training settings remain the same with SentiBERT. We also initialize SentiBERT with pre-trained parameters of RoBERTa (SentiBERT w/ RoBERTa) and further compare it with its variants.
As shown in Table 1, SentiBERT and SentiBERT w/ RoBERTa substantially outperforms their corresponding variants and the networks merely built on the tree. Specifically, we first observe that though our attention network (SentiBERT w/o BERT) is simple, it is competitive with Recursive NN, GCN and TreeLSTM. Besides, SentiBERT largely outperforms SentiBERT w/o BERT by leveraging contextualized representation. Moreover, the results manifest that SentiBERT and SentiBERT w/ RoBERTa outperform the BERT and RoBERTa, indicating the importance of incorporating syntactic guidance.
4.3 Transferability of SentiBERT
Though the designed models are effective, we are curious how beneficial the compositional sentiment semantics learned on SST can be transferred to other tasks. We compare SentiBERT with pub-
'https://github.com/huggingface
Models	SST-phrase	SST-5
Recursive NN	58.33	46.53
GCN	60.89	49.34
Tree-LSTM	61.71	50.07
BiLSTM w/ Tree-LSTM	61.89	50.45
BERT w/ Mean pooling	64.53	50.68
BERT w/ GCN	65.23	54.56
BERT w/ Tree-LSTM	67.39	55.89
RoBERTa w/ Mean pooling	67.73	56.34
SentiBERT w/o BERT	61.04	50.31
SentiBERT	68.31	56.10
SentiBERT w/ RoBERTa	68.98	56.87
Table 1: The averaged accuracies on SST-phrase and SST-5 tasks  $(\%)$  for 5 runs. For baselines vanilla BERT and RoBERTa, we use mean-pooling on token representation of top layer to get phrase and sentence representation.
Models	SST-2 (Dev)	SST-3	Twitter
BERT	92.39	73.78	70.0
BERT w/ Mean pooling	92.33	74.35	69.7
XLNet	93.23	75.89	70.7
RoBERTa	94.31	78.04	71.1
SentiBERT w/o BERT	86.57	68.32	64.9
SentiBERT w/o Masking	92.48	76.95	70.7
SentiBERT w/o Pre-training	92.44	76.78	70.8
SentiBERT	92.78	77.11	70.9
SentiBERT w/ RoBERTa	94.72	78.69	71.5
lished models BERT, XLNet, RoBERTa and their variants on benchmarks mentioned in Section 4.1. Specifically, 'BERT' indicates the model trained on the raw texts of the SST dataset. 'BERT w/ Mean pooling' denotes the model trained on SST, whose phrase and sentence representation is computed by mean pooling on tokens. 'BERT w/ Mean pooling' merely leverages the phrases' range information rather than syntactic structural information.
Sentiment Classification Tasks The evaluation results of sentence-level sentiment classification on the three tasks are shown in Table 2. Despite the difference among tasks and datasets, from experimental results, we find that SentiBERT has competitive performance compared with various baselines. SentiBERT achieves higher performance than the vanilla BERT and XLNet in tasks such as SST-3 and Twitter Sentiment Analysis. Besides, SentiBERT significantly outperform
Table 2: The averaged results on sentence-level sentiment classification  $(\%)$  for 5 runs. For SST-2,3, the metric is accuracy; for Twitter Sentiment Analysis, we use averaged recall value.
Models	Emotion Intensity	EmoContext
BERT	65.2	73.49
RoBERTa	66.4	74.20
SentiBERT w/o Pre-training	66.0	73.81
SentiBERT	66.5	74.23
SentiBERT w/ RoBERTa	67.2	74.67
Table 3: The averaged results on several emotion classification tasks (\%) for 5 runs. For Emotion Intensity Classification task, the metric is averaged Pearson Correlation value of the four subtasks; for EmoContext, we follow the standard metrics used in Chatterjee et al. (2019) and use F1 score as the evaluation metric.
SentiBERT w/o BERT. This demonstrates the importance of leveraging pre-trained BERT model. Moreover, SentiBERT outperforms BERT w/ Mean pooling. This indicates the importance of modeling the compositional structure of sentiment.
Emotion Classification Tasks Emotion detection is different from sentiment classification. However, these two tasks are related. The task aims to classify fine-grained emotions, such as happiness, fearness, anger, sadness, etc. It is challenging compared to sentiment analysis because of various emotion types. We fine-tune SentiBERT and SentiBERT w/ RoBERTa on Emotion Intensity Classification and EmoContext. Table 3 shows the results on the two emotion classification tasks. Similar to the results in sentiment classification tasks, SentiBERT obtains the best results, further justifying the transferability of SentiBERT.
We conduct experiments on SST-phrase using BERT-base model as backbone to demonstrate the effectiveness and interpretability of the SentiBERT architecture in terms of semantic compositionality. We also explore potential of the model when lacking phrase-level sentiment information. In order to simplify the analysis of the change of sentiment polarity, we convert the 5-class labels to 3-class: the classes 'very negative' and 'negative' are converted to be 'negative'; the classes 'very positive' and 'positive' are converted to be 'positive'; the class 'neutral' remains the same. The details of statistical distribution in this part is shown in Appendix A.3.
We consider the following baselines to evaluate the effectiveness of each component in SentiBERT. First we design BERT w/ Mean pooling as a base model, to demonstrate the ne
Figure 3: Evaluation for local difficulty. The figure shows the accuracy difference on phrase node sentiment prediction with BERT w/ Mean pooling for different local difficulty.
cessity of incorporating syntactic guidance and implementing aggregation on it. Then we compare SentiBERT with alternative aggregation approaches, Tree-LSTM, GCN and w/o Attention to Children.
5.1 Semantic Compositionality
We investigate how effectively SentiBERT captures compositional sentiment semantics. We focus on how the representation in SentiBERT captures the sentiments when the children and parent in the constituency tree have different sentiments (i.e., sentiment switch) as shown in the red boxes of Figure 1. Here we focus on the sentiment switches between phrases. We assume that the more the sentiment switches, the harder the prediction is.
We analyze the model under the following two scenarios: local difficulty and global difficulty. Local difficulty is defined as the number of sentiment switches between a phrase and its children. As we consider binary constituency tree, The maximum number of sentiment switches for each phrase is 2. Global difficulty indicates number of sentiment switches in the entire constituency tree. The maximum number of sentiment switches in the test set is 23. The former is a phrase-level analysis and the latter is sentence level.
We compare SentiBERT with aforementioned baselines. We group all the nodes and sentences in the test set by local and global difficulty. Results are shown in Figure 3 and Figure 4. Our model achieves better performance than baselines in all situations. Also, we find that with the increase of difficulty, the gap between our models
Figure 4: Evaluation for global difficulty. The figure shows the accuracy difference on phrase node sentiment prediction with BERT w/ Mean pooling for different global difficulty.
and baselines becomes larger. Especially, when the sentiment labels of both children are different from the parent node (i.e., local difficulty is 2), the performance gap between SentiBERT and BERT w/ Tree-LSTM is about  $7\%$  accuracy. It also outperforms the baseline BERT model with mean pooling by  $15\%$ . This validates the necessity of structural information for semantic composition and the effectiveness of our designed attention networks for leveraging the hierarchical structures.
5.2 Negation and Contrastive Relation
Next, we investigate how SentiBERT deals with negations and contrastive relation.
Negation: Since the negation words such as 'no', 'n't' and 'not' will cause the sentiment switches, the number of negation words also reflects the difficulty of understanding sentence and its constituencies. We first group the sentences by the number of negation words, and then calculate the accuracy of the prediction on their constituencies respectively. In test set, as there are at most six negation words and the amount of sentences with above three negation words is small, we separate all the data into three groups.
Results are provided in Figure 5. We observe SentiBERT performs the best among all the models. Similar to the trend in local and global difficulty experiments, the gap between SentiBERT and other baselines becomes larger with increase of negation words. The results show the ability of SentiBERT when dealing with negations.
Figure 5: Evaluation for negation. We show the accuracy difference with BERT w/ Mean pooling.
Models	Accuracy
BERT w/ Mean pooling	26.1
BERT w/ Tree-LSTM	28.5
BERT w/ GCN	29.4
SentiBERT w/o Attention to Children	29.8
SentiBERT	30.7
Table 4: Evaluation for contrastive relation  $(\%)$ . We show the accuracy for triple-lets ('X but Y', 'X', 'Y'). X and Y must be phrases in our experiments.