SlowGuess/ACL_2020_OCR huggingface dataset:
Contrastive Relation: We evaluate the effectiveness of SentiBERT with regards to tackling contrastive relation problem. Here, we focus on the contrastive conjunction "but". We pick up the sentences containing word 'but' of which the sentiments of left and right parts are different. In our analysis, a 'X but Y' can be counted as correct if and only if the sentiments of all the phrases in triplelet ('X but Y', 'X' and 'Y') are predicted correctly. Table 4 demonstrates the results. SentiBERT outperforms other variants of BERT about  $1\%$ , demonstrating its ability in capturing contrastive relation in sentences.
We showcase several examples to demonstrate how SentiBERT performs sentiment semantic composition. We observe the attention distribution among hierarchical structures. In Figure 7, we demonstrate two sentences of which the sentiments of all the phrases are predicted correctly. We also visualize the attention weights distributed to the child nodes and the phrases themselves to see which part might contribute more to the sentiment of those phrases.
SentiBERT performs well in several aspects.
First, SentiBERT tends to attend to adjectives such as 'frenetic' and 'funny', which contribute to the phrases' sentiment. Secondly, facing negation words, SentiBERT considers them and a switch can be observed between the phrases with and without negation word (e.g., 'not really funny' and 'really funny'). Moreover, SentiBERT can correctly analyze the sentences expressing different sentiments in different parts. For the first case, the model concentrates more on the part after 'but'.
5.4 Amount of Phrase-level Supervision
We are also interested in analyzing how much phrase-level supervision SentiBERT needs in order to capture the semantic compositionality. We vary the amount of phrase-level annotations used in training SentiBERT. Before training, we randomly sample  $0\%$  to  $100\%$  with a step of  $10\%$  of labels from SST training set. After pre-training on them, we fine-tune SentiBERT on tasks SST-5, SST-3 and Twitter Sentiment Analysis. During fine-tuning, for the tasks which use phrase-level annotation, such as SST-5 and SST-3, we use the same phrase-level annotation during pre-training and the sentence-level annotation; for the tasks which do not have phrase-level annotation, we merely use the sentence-level annotation.
Results in Figure 6 show that with about  $30\% - 50\%$  of the phrase labels on SST-5 and SST-3, the model is able to achieve competitive results compared with XLNet. Even without any phrase-level supervision, using  $70\% - 80\%$  of phrase labels in pre-training allows SentiBERT competitive with XLNet on the Twitter Sentiment Analysis dataset.
Furthermore, we find the confidence of about  $40 - 50\%$  of phrase nodes in SST-3 task is above 0.9 and the accuracy of predicting these phrases is above  $90\%$  on the SST dataset. Considering the previous results, we speculate if we produce part of the phrase labels on generic texts, choose the predicted labels with high confidence and add them to the original SST training set during the training process, the results might be further improved.
We proposed SentiBERT, an architecture designed for capturing better compositional sentiment semantics. SentiBERT considers the necessity of contextual information and explicit syntactic guidelines for modeling semantic composition. Experiments show the effectiveness and transferability
3701
(a) SST-5
(b) SST-3
(c) Twitter Sentiment Analysis
Figure 6: The results of SentiBERT trained with part of the phrase-level labels on SST-5, SST-3 and Twitter Sentiment Analysis. We show the averaged results of 5 runs.
Figure 7: Cases for interpretability of compositional sentiment semantics. The three color blocks between parents and children are the attention weights distributed to left child, the phrase itself and right child.
of SentiBERT. Further analysis demonstrates its interpretability and potential with less supervision. For future work, we will extend SentiBERT to other applications involving phrase-level annotations.
We would like to thank the anonymous reviewers for the helpful discussions and suggestions. Also, we would thank Lianian Harold Li, Xiao Liu, Wasi Ahmad and all the members of UCLA NLP Lab for advice about experiments and writing. This material is based upon work supported in part by a gift grant from Taboola.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer Normalization. arXiv preprint arXiv:1607.06450.
Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. Sentiwordnet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. In LREC, volume 10, pages 2200-2204.
Ankush Chatterjee, Kedhar Nath Narahari, Meghan Joshi, and Puneet Agrawal. 2019. Semeval-2019 Task 3: Emocontext Contextual Emotion Detection in Text. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 39-48.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Jacob Harer, Chris Reale, and Peter Chin. 2019. TreeTransformer: A Transformer-Based Method for Correction of Tree-Structured Data. arXiv preprint arXiv:1908.00449.
Dan Hendrycks and Kevin Gimpel. 2017. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. Proceedings of International Conference on Learning Representations.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735-1780.
Minqing Hu and Bing Liu. 2004. Mining and Summarizing Customer Reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 168-177. ACM.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2019. SentiLR: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis. arXiv preprint arXiv:1911.02493.
Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014
Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751. Association for Computational Linguistics.
Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-Normalizing Neural Networks. In Advances in Neural Information Processing Systems, pages 971-980.
Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies, 5(1):1-167.
Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. 2015. Multi-timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2326-2335.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 1-8.
Jeff Mitchell and Mirella Lapata. 2008. Vector-based Models of Semantic Composition. In Proceedings of ACL-08: HLT, pages 236-244.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 Task 1: Affect in Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1-17.
Bo Pang and Lillian Lee. 2004. A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, page 271. Association for Computational Linguistics.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs Up?: Sentiment Classification Using Machine Learning techniques. In Proceedings of the
ACL-02 Conference on Empirical Methods in Natural Language Processing-Volume 10, pages 79-86. Association for Computational Linguistics.
Bo Pang, Lillian Lee, et al. 2008. Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval, 2(1-2):1-135.
Francis Jeffry Pelletier. 1994. The Principle of Semantic Compositionality. Topoi, 13(1):11-24.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237.
Fanchao Qi, Junjie Huang, Chenghao Yang, Zhiyuan Liu, Xiao Chen, Qun Liu, and Maosong Sun. 2019. Modeling Semantic Compositionality with Sememe knowledge. ACL.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 Task 4: Sentiment Analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502-518.
Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic Compositionality through Recursive Matrix-Vector Spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201-1211. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556-1566, Beijing, China. Association for Computational Linguistics.
Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Attentive Gated Lexicon Reader with Contrastive Contextual Co-attention for Sentiment Classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3443-3453.
Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019. Tree Transformer: Integrating Tree Structures into Self-Attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1061-1070, Hong Kong, China. Association for Computational Linguistics.
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019. A Failure of Aspect Sentiment Classifiers and an Adaptive Re-weighting Solution. arXiv preprint arXiv:1911.01460.
Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2016. DAG-Structured Long Short-Term Memory for Semantic Compositionality. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 917-926, San Diego, California. Association for Computational Linguistics.
A.1 Details of Correlation Computation in Attention Networks
For vectors  $\mathbf{a}$  and  $\mathbf{b}$ , the correlation between them is computed as below:
$$
\begin{array}{l} \operatorname {A t t e n t i o n} (\mathbf {a}, \mathbf {b}) = \tanh  \left(\frac {1}{\alpha} \operatorname {S e L U} ((W _ {1} \times \mathbf {a}) ^ {T} \times W _ {3} \right. \\ \times \operatorname {S e L U} \left(W _ {2} \times \mathbf {b}\right)), \tag {3} \\ \end{array}
$$
where SeLU (Klambauer et al., 2017) is an activation function and  $\alpha$  equals 4. The two layers of attention networks do not share the parameters.
A.2 Details of Downstream Tasks
We adopt the following tasks for evaluation of sentence-level sentiment classifications:
SST-2,3 (Socher et al., 2013) These tasks all share with the text of the SST dataset and are single-sentence sentiment classification task, of which the numbers behind indicate the number of classes. Since two of five classes in SST-5 correspond to positive and another two indicate negative, with additional neutral ones, the dataset is separated into three groups in SST-3 task. We convert the 5-class phrase-level labels in SST-5 into three classes and leverage them in the training of SST-3 task.
Twitter Sentiment Analysis (Rosenthal et al., 2017) For Twitter Sentiment Analysis, given a tweet, model needs to decide which sentiment it expresses: positive, negative or neutral.
Emotion Intensity Ordinal Classification (Mohammad et al., 2018) The task is, given a tweet and an emotion, categorizing the tweet into one of four classes of intensity that best represents tweeter's mental state. For Emotion Intensity Classification task, the metric is averaged Pearson Correlation value of the four subtasks, 'happiness', 'sadness', 'anger' and 'feariness'.
Emotions in Textual Conversations (Chatterjee et al., 2019) In a dialogue, given a sentence with two turns of conversation, the models needs to classify the emotion expressed in the last sentence. For EmoContext, we follow the standard metrics used in Chatterjee et al. (2019) and use F1 score on the three classes 'happy', 'sad' and 'angry', except 'others' class, as the evaluation metric.
The statistics of datasets is shown in Table 5.
Dataset	Data Split	# of Classes
SST-phrase	8379 / 2184	5
SST-2	66475 / 859	2
SST-3	8379 / 2184	3
SST-5	8379 / 2184	5
Twitter	50284 / 12273	3
EmoContext	30141 / 2754	3
sad: 1533 / 975
EmoInt	angry: 1701 / 1001	4
fear: 2252 / 986
joy: 1616 / 1105
Table 5: Statistics of benchmarks.
Local Difficulty	0	1	2
Number	28136	10174	1342
Table 6: The distribution of nodes in terms of local difficulty.
Global Difficulty	0-4	5-9	10-14	15-19	20-23
Number	930	861	326	59	8
Table 7: The distribution of nodes in terms of global difficulty.
# of Negation Words	0	1	2-
Number	1825	325	34
Table 8: The distribution of nodes in terms of negation words.
Models	SST-phrase	SST-5
SentiBERT w/ token	68.23	56.02
SentiBERT w/ token and RoBERTa	68.78	56.91
SentiBERT	68.31	56.10
SentiBERT w/ RoBERTa	68.98	56.87
Table 9: The results after incorporating token node prediction. 'Token' denotes token node prediction.
A.3 Details of Analysis Part
The distribution of nodes and sentences in terms of local difficulty, global difficulty and negation words is shown in Table 6, 7 and 8, respectively.
A.4 Incorporating Token Node Prediction
Since the SST dataset also provides token-level sentiment labels, we combine the token node prediction with phrase node prediction learning objective together to model compositional sentiment semantics.
Results are shown in Table 9. We observe that the results drops a bit after additionally incorporating token-level sentiment information. This may be because the phrase sentiment is composed but the token sentiment mainly depends on the meaning of the lexicon itself rather than a kind of compositional sentiment semantics. The inconsistency of the training objectives may result in the performance drop.