Non-Compositionality in Sentiment: New Data and Analyses
Authors: Verna Dankers
Christopher G. Lucas
Venue: EMNLP 2023 Findings (2023)
Keywords: sentiment analysis
compositionality
data annotation
TL;DR: We propose a methodology for obtaining non-compositionality ratings of natural language phrases for sentiment analysis. We use that method to create a resource of ratings, present an analysis of that resource, and use it for model evaluation.

Abstract: When natural language phrases are combined, their meaning is often more than the sum of their parts.
In the context of NLP tasks such as sentiment analysis, where the meaning of a phrase is its sentiment, that still applies.
Many NLP studies on sentiment analysis, however, focus on the fact that sentiment computations are largely compositional.
We, instead, set out to obtain non-compositionality ratings for phrases with respect to their sentiment.
Our contributions are as follows: a) a methodology for obtaining those non-compositionality ratings, b) a resource of ratings for 259 phrases – NonCompSST – along with an analysis of that resource, and c) an evaluation of computational models for sentiment analysis using this new resource.

Non-Compositionality in Sentiment: New Data and Analyses
Verna Dankers and Christopher G. Lucas
Institute for Language, Cognition and Computation University of Edinburgh
vernadankers@@gaill com, c. w..luaasedddc ac.
Abstract
When natural language phrases are combined, their meaning often more than the sum of their parts. In the context of NLP tasks such sentiment analysis, where the meaning of phrase is its sentiment, that still applies. Many NLP studies sentiment analysis, however, fo- cus on the fact that sentiment computations are largely compositional. We, instead, set out to obtain con-conpositoonality ratings for phrases with respect to their sentiment. Our contribu- tions are as follows: a) methodology for ob- taining those non-compositionality ratings, b) a resource of ratings for 259 phrases NON- COMPSST along with an analysis of that resource, and c) an evaluation of computational models for sentiment analysis using this new resource.
conpositional expectation
sentiment observed
composi sentinent observed
expectation
the excitement of eating oatneal
nearly terminal of the cutes
Figure I: Illustration of how the observed sentiment deviates from the expected sentiment when viewing po- larity function of the polarity of the subphrases. These examples were obtained from our newly anno- tated stimuli.
sentiment of the individual parts (e.g. see Zhu et al., 2015; Hwang and Hidey, 2019; Barnes et al., 2019; Tahayna et al., 2022, for work on non- compositional sentiment). These include the case of sarcasm ("life is good, you should get one"), opposing sentiments ("terribly fascinating"), id- iomatic expressions ("break leg") and neutral terms that, when composed, suddenly convey timent ("yeah right"). Adequately capturing senti- ment computationally requires both learning com- positional rules and understanding when such ex- ceptions exist, where most contemporary sentiment models are expected to learn that via mere end-to- end training on examples.
1 Introduction
In NLP, the topics of the compositionality of lan- guage and neural models' capabilities compute meaning compositionally have gained substantial interest in recent years. Yet, the meaning of lin- guistic utterances often does not adhere to strict patterns and can be surprising when looking at the individual words involved. This affects how those utterances behave in downstream tasks, such as sentiment analysis. Given phrase sentence, that task involves predicting the polarity as posi- tive, negative or neutral. Sentiment largely adheres to compositional principles (Moilanen and Pulman, 2007, I): "If the meaning of sentence is a func- tion of the meanings of its parts then the global polarity of sentence is a function of the polarities of its parts." Modelling sentiment as composi- tional process is, therefore, often mentioned design principle for computational sentiment mod- els (e.g. by Socher al., 2013; Sutherland al., 2020; Yin al., 2020).
How can we identify whether the sentiment of phrase non-compositional? We design a proto- col to obtain such non-compositionalliy judgments based on human-annotated sentiment. Our method- ology (elaborated on in $3) utilises phrases from the Stanford Sentiment Treebank (SST) (Socher et al., 2013) and contrasts the sentiment of a phrase with control stimuli, in which one of the two sub- phrases has been replaced. Phrases whose anno- tated sentiment deviates from what expected based on the controls are considered less composi- tional, as is illustrated in Figure 1. We analyse the resulting non-compositionality ranking of phrases
Nonetheless, one can think of examples where the sentiment phrase is unexpected given the

---

($4) and show how the constructed resource can be used to evaluate sentiment models ($5). Our new resource (NONCOMPSST) can further im- prove the understanding of what underlies non- compositionality in sentiment analysis, and can complement existing evaluation protocols for senti- ment analysis models.
Preprocessing
Study
Step 1: phrases arnotate:
Step obtaln the sentinent
Fron the Stanford Santimen Tresbanic (53T)
subphrases.
supplerase
sutphrass
Good agressent
Subchrases length
No naraed entities
control A
control E
sontrol A
contred Eg
Select control sttmull m. for
control Aj
control B
w.bphrass::
Sano grammatical category Same sentiment
control Ai
control BY
2 Related work
Study for
Postprocessing
Over the course of years, sentiment analysis sys- tems went from using rule-based models and sen- timent lexicons (e.g. Moilanen and Pulman, 2007; Taboada et al., 2011) 1) to using recursive neural net- works (Socher et al., 2013; Tai et al., 2015; Zhu et al.. 2015), to abandoning the use of structure altogether by finetuning pretrained large language models (e.g. Pérez al., 2021; Camacho-collados al., 2022: Hartmann et al., 2023), and recently, to abandoning training, via zero-shot generalisation (Wang et al., 2023). Crucial to the development of these systems has been the introduction of bench- marks, such as SST (Socher et al.. 2013) and Se- mEval's ' Twitter benchmarks (e.g. Rosenthal et al., 2015; Nakov et al., 2016; Rosenthal et al., 2017). Although the vast majority of related work fo- cused on simply improving on benchmarks, there have been studies more closely related to ours, ask- ing questions such as: How do phrases with oppos- ing sentiment affect each other (Kiritchenko and Mohammad, 201 ba,b)? What is the role of nega- tions (Zhu et al., 2014), modals and adverbs (Kir- itchenko and Mohammad, 2016c)? Do idioms have non-compositional sentiment (Hwang and Hidey, 2019)? Which linguistic phenomena are still prob- lematic for SOTA sentiment systems (Barnes et al., 2019)? Can we incorporate compositional and non-compositional processing in one system (Zhu et al., 2015)? And can we computationally rank sentences according to their sentiment composition- ality (Dankers and Titov, 2022y? Gaining better understanding of the contexts in which sentiment functions con-compositionally and is challenging predict crucial for the evaluation of sentiment models in an age where sentiment benchmarks may appear saturated (Barnes et al., 2019).!
nol the parase and constructed control parsses.
establish default repping the zenti ment. fifrenncc the default and observed sentiment yields a nonree rosiitonniii tisaaliiality estrie.
control
controll R
observed sentiment
controll
default rapping
contral
controll
Figure The methodology summarised. Steps and 4 consist of data pre- and postprocessing: steps and 3 involve collecting data from participants via Prolific.
Collecting non-compositionality ratings
Compositional processing of sentiment involves applying a function to the polarity of subphrases to obtain the polarity of the phrase. Turning this notion into a quantifiable metric requires to mea- sure the polarities and determine the composition function. Non-compositional phrases are then sim- ply phrases whose sentiment deviates from what is expected. How do we implement this? We first select data ($3.1) and then obtain sentiment labels for phrases through data annotation studies ($3.2). We consider the composition function to be the default mapping from subphrases with spe- cific sentiment to their combined sentiment. We obtain the default mapping by replacing subphrases with control stimuli and annotating sentiment for those modified phrases. Using those results, we can compute the non-compositionaaliy ratings ($3.3). Figure 2 summarises the full procedure.
3.1 Materials
We first select phrases for which obtain the non- compositionality ratings, along with control stim- uli.
We position our work in this latter group of ar- ticles, of which that of Hwang and Hidey is most closely related.
Data selection We obtain our data from the SST dataset, containing 11,855 sentences from movie reviews (Pang and Lee, 2005), and sentiment an- notations from Socher et al. (2013). The dataset provides sentiment labels for all full sentences and
As a concrete example, consider the widely-used binary SST sentiment analysis task contained in the GLUE hench- mark (Wang et al., 2018): at the time of writing, SOTA perfor- mance for this task matches humans' performance.

---

phrases contained in these sentences (all phrases that represent a node in the constituency parse trees of these sentences). We select candidate phrases to include in our dataset by applying the following constraints to the phrases: they consist of two sub- phrases that contain 3-8 tokens each, do not contain
tincl
sentiment composition
0.8
0.9
named entities and had relatively high agreement in the original dataset. In Appendix A, we elaborate the implementation of our constraints.
Selection of control subphrases We assume that if subphrase behaves compositionally, replacing it with control should not affect the overall sen- timent of the phrase. How do we select control stimuli? By taking subphrases with the same senti- ment (based on SST's sentiment labels) and phrase type (c.g. NP, PP, SBAR). For each phrase con- sisting of subphrases A and B we automatically select 32 candidate control subphrases and manu- ally narrow them down eight (A', Py where (1, (2,3,,1) 1)). During the manual annotation, we removed examples for which fewer than eight suitable control stimuli remained. Our final collec- tion contains 500 phrases to be used in the human annotation study.
0.0
0.5
1.0
1.5
2.0
2.5
3.0
(a) Ratings per composition type
all
figurative
0.5
1.5
2.0
2.5
3.0
non-compositionaliy rating
(b) Ratings for all and figurative phrases
Figure 3: MAXABS non-compositionality ratings a) per composition type (-"', "w' and '+' refer to negative, neutral and positive), and b) for figurative examples.
total. The inter-annotator agreement rates obtained were 0.60 and 0.64 for Study and 2, respectively, in terms of Krippendorff's for ordinal data. Ap- pendix B further discusses these studies along with ethical considerations and annotation statistics.
3.2
Data annotation studies
We collect sentiment labels in two rounds using a 7- point scale. In Study 1, we obtain the sentiment for all subphrases involved to ensure that subphrases and their controls have the same sentiment. We then discard phrases for which A and/or B do not have more than three controls each, where we re- strict the controls those whose sentiment is at most 1 point removed from the sentiment of (for Nr or B (for B²,).
3.3 Computing non-compositionallty ratings
We obtain one sentiment label per phrase by aver- aging the annotations from Study 2. Afterwards. the conostopositionalttyy ratings for phrase "A B" are computed separately for A and B. The rating for A is the difference between the sentiment of " "A B" and the mean sentiment of phrases "A', B" (n E 21,2, 3)), and vice versa for B. Together, the two ratings express the non-compositionaaliy of "A B". We compute four variants of the ratings: ALL, ALLABS, MAX, MAXABS, ALL.CEAAN The first two include A and B separately, the second two use rating per phrase (the largest of the two). ALLCLEAN includes A and B separately but ex- cludes any phrases that are considered ungrammat- ical (109 out of 1813 phrases involved in Study 2 were flagged for that).
In Study 2, we collect sentiment labels for all subphrase combinations, namely the remaining 259 phrases and the 1554 phrases in which control subphrase is inserted. For a phrase "A B", there are six controls: three that substitute A, and three that substitute B. Those substitutions could lead to ungrammatical constructions in spite of the data se- lection procedure, and participants can indicate that with checkbox. Figure 4 in Appendix В displays example questions as shown to the participants. Participants were recruited via Prolific and anno- tated sentiment via a Qualtries survey. In Study 1. 57 participants annotate 93 or 94 subphrases each. Study 2, 90 participants annotate 60 or sub- phrase combinations each. That way, every unique phrase and subphrase receives three annotations
Analysis of the ratings
What patterns can we identify in these ratings? We examine sentiment composition types, phrase lengths and syntactic categories of subphrases Appendix C; only the sentiment composition type

---

subphrase subphrase
Rating
Sentiment
HUMAN
ROBERTA
a nearly terminal case of cutes the franchise's ' best years are long past all the excitement of cating oatmeal a pressure cooker fans of the animated wildlife adventure show real human soul
5.33
2.00
-4.11
0.33
1.00
-3.00
2.00
2.33
of horrified awe
-2556
3.67
will be in warthog heaven
1,56
5.67
5.00
buried beneath spellbinding serpent's smirk
1,56
4.67
5.00
everyone involved with moviemaking the modern master of the chase sequence returns with chase to end all chases
is a con artist and
-1.33
1.11
6.00
5.00
Table 1: Examples of non-compositional phrases, with their MAX rating ($3.3 describes how these ratings are computed) and the sentiment assigned by the annotators and by ROBERTA-LARE (details on how the model was trained are contained $5). Red indicates negative sentiment, green indicates positive sentiment.
displays a clear pattern, illustrated by the ratings' distributions in Figure 3. The most compositional are the cases where the subphrases share their pos- itive/negative sentiment, whereas combining op- posites the least compositional. Most phrases have an absolute rating within point of our 7- point scale; only for 67 out of the 259 phrases, the MAXABS non-compositionaliy rating exceeds 1.
"returns with chase to end all chases" functions differently when it concerns the "master of the chase sequence" rather than anyone else. These examples illustrate sentiment compositions are of- ten nuanced and that there is long tail of atypical non-compositional phenomena.
Evaluating sentiment models
What characterises the least compositional ex- amples?2 The most prominent pattern is that figu- rative language is over-represented, which we can quantitatively illustrate by annotating all phrases as figurative and literal; the resulting MAXABS distributions differ substantially (Figure 3). In Ta- ble 1, some examples of figures of speech are the "pressure cooker" (a container metaphor implying stressful situation, Kövecses and Kövecses, 1990), the "neally terminal case of the cutes" (suggesting one can die of cuteness for emphasis), the "ser- pent's smirk" (metaphorically used to invoke con- notations about evil) and the hyperbole of "every- one [...] is a con artist and a liar".
How can we employ the non-compositionaaliy rat- ings better understand the quality of sentiment systems? We illustrate this by recreating the ratings using state-of of-the-art pretrained neural models and comparing them to the humans' ratings.
Experimental setup To obtain the ratings from models, we adapt SST use the 7-point scale and exclude the phrases of interest from the training data. Per model type, we fine-tune three model seeds that we evaluate on the SST test set us- ing Fi-score, and on NONCOMPSST using a) the correlation of the models' and humans' non- compositionalitt ratings (Pearson's r), and b) the Fi Fi-sc of NONCOMPSST phrases, using the hu- mans' sentiment scores as labels. To obtain mod- els' conpopoooiitionllttty ratings, we average sen- timent predictions from the three model seeds and apply the same postprocessing as applied to the human-annotated data (see $3.3).
Other atypical sentiment patterns that we ob- serve require common-sense reasoning about terms that act contextual valence shifters (Polanyi and Zaenen, 2006), e.g. to understand that "are long past" implics something about cur- rent times, or that "cating oatmeal" relates to the blandness of that experience. Lastly, we also ob- serve discourse relations between subphrases that modify the sentiment in a non-compositional man-
We evaluate ROBERTA- HIBERTA-BASE and -LARGE (Liu et al., 2019) along with variants of those models that are further trained on sentiment-laden data: TIMELM (the BASE model pretrained on tweets by Loureiro et al., 2022); the model of Camacho- collados al. (2022), which is TIMELM fine-tuned to predict tweets' sentiment; BERTTWEET (a BASE
In "fans of the animated wildlife adventure show will be in warthog heaven", the parallel be- tween 'wildlife' and 'warthog heaven' amplifies the positive sentiment a way that would not have happened for fans of a fashion show. Similarly,
"The authors published the unprocessed original annota- tions by Amazon Mechanical Turk annotators of Socher et al. (2013): we process this data into SST-7.
We include them in Appendix C, in Table 3.

---

Model name
SST
NONCOMPSST ALLABS
MAX
MAXABS r
ALL
ALLCLEAN
All Fi
Top Fi
Pretrained
ROBERTA-BASE, Liu et al.
.4
.66
.31
.11
.22
.43
.40
.32
RORERTA-LARGE, Liu et al. Pretrained using sentiment-latee laden data
.47
.42
.38
.44
.30
.46
.47
.37
TIMELM, Loureiro et al."
.30
.32
.25
.36
.43
.38
BERT Nguyen et al." #
.46
.22
.20
.25
.15
.27
.43
.30
Finetuned using sentiment-lade ladien data
Camacho-collados et al." R
.45
.33
.36
.36
.22
.38
.49
.43
Pérez
.44
.13
.21
.20
.16
.21
.42
.26
Hartmann 4
.38
.34
.28
.44
.45
.33
IMDB ROBERTA
.44
.37
.31
.41
.24
.44
.47
.43
Table 2: Model evaluation using SST (11) and NONCOMPSST, according correlation (Pearson's 's between the humans' and the models' non-coomositioonaliy ratings, and the Fi of the 259 phrases and the 67 most non- compositional ones, measured using the humans' annotations as labels. We indicate whether models are BASE (B) LARGE (L) and underline the highest performance per column.
model pretrained on tweets by Nguyen et al., 2020); the model of Pérez et al. (2021) (BERTTWEET fine-tuned to predict tweets' sentiment); ROBERTA- LARGE (Hartmann et al., 2021) fine-tuned on sen- timent of comments from social media posts; and finally ROBERTA-BASE fine-tuned sentiment from IMDB movie reviews (Maas et al., 101
Camacho-collados et (2022) achieve the high- est Fi score. These scores emphasise that for the top 67, sentiment is substantially harder to pre- dict: non-compositional examples indeed present a larger challenge to sentiment models than compo- sitional examples do.
Conclusion
Results The results Table 2 suggest that even though the systems have very similar performance on the SST test set, there are differences in terms of the NONCOMPSST ratings: ROBERTA-BASE has the lowest SST Fi, but is the second-best BASE model in terms of only outperformed by IMDB- ROBERTA (i.e. the variant fine-tuned movie re- views, the domain of SST). The ROBERTA-LARGE model outperforms both Hartmann al.'s 's model and the BASE models in terms of the SST F1 and NONCOMPSST correlations. Together, these ob- servations suggest that pretraining or fine-tuning us- ing data from different domain can harm models' ability to capture nuanced sentiment differences required to estimate NONCOMPSST ratings. The SST performance is less sensitive this, suggest- ing that our resource can provide complementary view of sentiment systems.
Sentiment and compositionality go hand-in-hand: success in sentiment analysis is often attributed models' capability "compose' sentiment. In- deed, the sentiment of a phrase reasonably pre- dictable from its subphrases' sentiment, but there are exceptions due to the ambiguity, contextuality and creativity of language. We made this explicit through an experimental design that determines non-compositionality ratings using humans' sen- timent annotations and obtained ratings for 259 phrases ($3). Even though most phrases are fairly compositional. found intriguing exceptions ($4). and have shown how the resource can be used for model evaluation ($5). For future sentiment anal- ysis approaches, we recommend a multi- facttee evaluation setup: to grasp the nuances of sentiment, one needs more than compositionality.
Finally, we also inspect the NONCOMPSST Fi- scores for all 259 phrases and the 67 phrases with the highest non-compositionalliy ratings according to the human annotators, for which results are in- cluded in the final two columns of Table 2. On that subset,
Limitations
Our work makes several limiting assumptions about compositionality in the context of sentiment:
and the model of "See Appendix D for further details on the experimen- tal setup used to fine-tune these models on SST. Visit our repository for the data and code.
We maintain simplistic interpretation of the composition 'function' but are aware that com- positionality is considered vacuous by some (Zadrozny, 1994) since by using a generic

---

notion of 'function', any sentiment compu- tation can be considered compositional. We, therefore, only consider some phrases non- compositional because of the strict interpre- tation of that 'function'. As a result, one might argue that whether a phrase such as "all the excitement of eating oatmeal" is non- compositional in terms of its sentiment is de- batable. We agree with that; if you represent the sentiment with very expressive represen- tation, every sentiment computation is compo- sitional. Our results only apply given a very narrow interpretation of compositionality. 2. In this work, we restrict the notion of mean- ing compositions to the notion of sentiment compositions. Hence, there might be phrases that behave compositionally in terms of sen- timent but are considered non-compositional otherwise. For instance, "rotten apple" carries negative sentiment, both literally and figura- tively, and might thus be considered composi- tional in terms of sentiment.
Lindemann for his comments on draft of this article. VD is supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S02248 1/1) and the University of Edinburgh, School of Informatics and School of Philosophy, Psychology & Language Sciences.
References
Jeremy Claude Barnes, Lilja Øvrelid, and Erik Velldal. 2019. Sentiment analysis is not solved! Assessing and probing sentiment classification. In Association for Computational Linguisties (ACL). Annual Meet- ing Conference Proceedings, pages 12-23, Associa- tion for Computational Linguistics.
Jose Camacho-collados. Kiamehr Rezaee. Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthe- nis Antypas, Joanne Boisson, Luis Espinosa Anke, Fangyu Liu, and Eugenio Martínez Cámara. 2022. TweetNLP: Cutting-edge natural language process- ing for social media". In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, pages 38-49, Association for Computational Linguistics.
In addition to that, the resource we developed has technical limitations:
Verna Dankers and Ivan Titov. 2022. Recursive neural networks with bottlenecks diagnose (non-) composi- tionality. In Findings of the Association for Computa- rional Linguistics: EMNLP 2022, pages 4361-4378.
1. Human annotators can provide unreliable sentiment annotations: they do not neces- sarily agree with one another, may lose focus while performing the task or may misunder- stand the linguistic utterances they annotate. As result, the resource inevitably contains some sentiment ratings that are inaccurate. 2. The resource we develop is small in size, which limits the robustness of the results when using the resource for experimentation. We would like to point out, however, that 259 annotations were involved in obtaining the ratings for these 259 phrases. The results il- lustrate that, in spite of these limitations, the resource can still lead to valuable conclusions.
Jochen Hartmann, Mark Heitmann, Christina Schamp, and Oded Netzer. 2021. The power of brand selfies. Journal of Marketing Research.
Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp. 2023. More than a feel- ing: Accuracy and application of sentiment analy- sis. International Journal of Research in Marketing. 40(1 40(1):75-87.
Dicuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Chris- Christodoolopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, and et al. 2023. A taxonomy and review of generalization research in NLP. Nature Machine Intelligence, 5(10):1161-1122
Finally, the evaluation of the models in $5 is some- what limited, considering that the various mod- els have been fine-tuned suned or pretrained by the men- tioned authors using different experimental setups. Even though we then apply the same setup to fine- tume on SST, these differences need to be kept in mind when interpreting the results.
Alyssa Hwang and Christopher Hidey. 2019. Con- firming the non-compositionalliy of idioms for senti- ment analysis. In Proceedings of the Joins Workshop Multiword Expressions and WordNer (MWE-WN 2019), pages 125-129.
Svetlana Kiritchenko and Saif Mohammad. 2016a. Happy accident: sentiment composition lexicon for opposing polarity phrases. In Proceedings of the Tenth International Conference Language Resources and Evaluation (LREC'16) 16), pages 1157- 1164.
Acknowledgements
We thank Kenny Smith and Ivan Titov for their suggestions throughout this project and Matthias

---

Svetlana Kiritchenko and Saif Mohammad. 2016b. Sen- timent composition of words with opposing polarities. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- lional Linguistics: Human Language Technologies, pages 1102-1108.
Juan Manuel Pérez, Juan Carlos Giudici, and Franco Luque. 2021. pysentimiento: A python toolkit for sentiment analysis and SocialNLP tasks. arXiv preprint arXiv:2106.0942
Livia Polanyi and Annic Zaenen. 2006. Contextual valence shifters. Computing attitude and affect in text: Theory and applications, pages 1-10.
Svetlana Kiritchenko and Saif M. Mohammad. 201 The effect of negators, modals, and degree adverbs on sentiment composition. In Proceedings of NAACL- HLT. pages 43-52.
Peng Qi, Yuhao Zhang. Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 101-108.
Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the s1 annual meeting of the association for computational linguistics, pages 423 430.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval- 2017), pages 502- 518.
Zoltán Kövecses and Zoltán Kövecses. 1990. The con- cept of emotion: The container metaphor. Emotion Concepts, pages 154-159.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint 1907:19907.1192
Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko. Saif Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. Semeval-2015 task 10: Sentiment analysis in twitter. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 451-463.
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. TimeLMs: Diachronic language models from Twitter. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 251-260.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language lechnologies, pages 142-150.
Alexander Sutherland, Suna Bensch, Thomas Hellström, Sven Magg, and Stefan Wermter. 2020. Tell me why you feel that way: Processing compositional depen- dency for Tree-LSTM aspect sentiment triplet extrac- tion (TASTE). In Artificial Neural Networks and Machine Learning-TCANT 2020: 29th International Conference Artificial Neural Networks, Bratislava, Slovakia, September 15-18. 2020, Proceedings. Part 19,, pages 660-671. Springer.
Karo Moilanen and Stephen Pulman. 2007. Sentiment composition. In Proceedings of the Recent Advances in Natural Language Processing International Con- ference, pages 378-382. Association for Computa- tional Linguisties.
Maite Taboada, Julian Brooke, Milan Tofiloski, Kim- berly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational lin- guistics, (7(2):267-307
Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Se- bastiani, and Veselin Stoyanov. 2016. Semeval-201 2016 task 4: Sentiment analysis in twitter. Proceedings of SemEval, pages 1-18.
Bashar MA Tahayna, Ramesh Kumar Ayyasamy, and Rehan Akbar. 2022. Automatic sentiment annotation of idiomatic expressions for sentiment analysis task. IEEE Access, 10: 0222234-122242
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing:
Kai Sheng Tai, Richard Socher, and Christopher D Man- ning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 1556- 1566.
System Demonstrations, pages 9-14.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'OS) pages 1 115-124.

---

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Interna- lioaal Conference on Learning Representations.
Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023. Is ChatGPT good sentiment analyzer? preliminary study. arXiv preprint arXiv:2304.0333.
Da Yin, Tao Meng, and Kai-Wei Chang. 2020. SentiB- ERT: A transferable transformer-bite based architecture for compositional sentiment semantics. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3695-3706.
Wlodek Zadrozny. 1994. From compositional to system- atic semantics. Linguistics and philosophy, 17:329- 342.
Xiaodan Zhu, Hongyu Guo, Saif Mohammad, and Svet- lana Kiritchenko. 2014. An empirical study the effect negation words on sentiment. Proceed- ings of the 52nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 304-313.
Xiaodan Zhu, Hongyu Guo, and Parinaz Sobhani. 2015. Neural networks for integrating compositional and non-compositional sentiment in sentiment composi- tion. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 1-9.

---

A Materials
We require data for which to obtain the non-compositionalty ratings and the control stimuli used to construct the default mapping (step in Figure 2).
Data selection We obtain our data from the Stanford Sentiment Treebank (SST), a resource containing 11,855 sentences from movic reviews collected by Pang and Lce (2005), and annotated with sentiment labels by Socher et al. (2013). The sentences were parsed with the Stanford Parser (Klein and Manning. 2003) to allow for sentiment annotations of phrases in addition to full sentences. The dataset includes sentiment labels for all nodes of those parse trees. The sentiment labels were obtained from Amazon Mechanical Turk annotators, who indicated the sentiment using a multi-stop slider bar with 25 ticks. We select potential phrases to include our dataset, by applying the following constraints to SST constituents. 1) They are marked as constituents by state-of-the-an constituency parser, and have two subphrases." 5 This ensures that the segmentation of the two parts is somewhat reasonable, and excludes compositions of three or more constituents. 2) The constituents' subphrases have 3-8 tokens, excluding punctuation. We exclude longer constituents, cognisant of the higher cognitive load that labelling those phrases would have for our participants. 3) The constituents do not contain named entity, such as movie the name of film. Not all participants may know these names and the associated sentiment, hence we opt for excluding them. 4) The standard deviation of the original SST annotations lies within a range of five (on the original 25- 25-point scale), exclude the most ambiguous phrases. That standard deviation is computed over the three annotations per phrase obtained by Socher et al. (2013).
Selection of control subphrases How do we select the control stimuli? They should have the same polarity as the part they are replacing. In that way, we can obtain the default mapping without altering the 'polarities of the parts" (Moilanen and Pulman, 2007). Each phrase from the data selection step has two subphrases. We distribute those subphrases into groups based on their SST sentiment (on an 11 -pooi scale, from 0.0 to 1.0) and phrase type (NP, VP, PP, SBAR).
For each phrase consisting of subphrases A and B we randomly select 32 candidate control subphrases from those groups and manually narrow those 32 down control subphrases (4 for A, for B). Where needed, small modifications between A', and B, and A and By During the manual annotation, examples for which fewer than 8 suitable control stimuli remain are removed until 500 phrases remain to be used in the annotation study.
made the control stimuli to create grammatical agreement
Question
followiig two phrases...
What sendiment
combined?
Quesitinn 12, Concider the lollowing phrase...
regatiee
nosethee
siihtly nogallit
positle
pos
postiic
strong dranatic and crnoiional pull
What sentiment of tras (eriqe??
Wey
ragadiic
mechad
siigtly
pooduu
Vary
Check fex boxil you sunt pfruee ungrammaaion.
(a) Study
(b) Study 2
Figure Illustration of the question formatting in Study and
B Studies
The data annotation studies have been approved by the local ethics committee of the institute which the authors belong. Prior starting the questionnaire, the participants are presented with Participant
We use s stanza's tokenizer, parser and named entity recognition model for English (Qi et al., 2020), https://stanforddlp. gi thub. stazaa.

---

350
700
175
na tatural cantrol
200
600
Count
Count
125
Count
200
400-
100-
150-
300
75-
50
100
25
100
mean
standard deviation
mean
standard deviation
(a) Study
(b) Study
Figure 5: Distributions of the means and standard deviations observed for Studies and 2, computed over the three annotations obtained per stimulus.
Information Sheet that explains their data protection rights, the goal of the study and the compensation that they will receive. We set out to pay participants at a rate of fl per hour (above the minimum wage), and estimated both studies to take 12 minutes to fill out.
Participants annotate the data via survey the Qualtrics platform. We first ask participants to familiarise themselves with the sentiment labels, by showing seven subphrases from the SST dataset and requesting the participants match them with the correct sentiment label. Afterwards, the correct answers are shown. These subphrases and their labels are taken from the SST dataset. Afterwards, the participants are shown 93 or 94 subphrases in isolation, and annotate them (the number of stimuli could not evenly be divided over participants while adhering to the desired study length of approximately 12 minutes). At the very end, the participants can provide feedback on the task. This study constitutes step 2 in Figure 2.
Procedure Study 1
Procedure Study 2 The second study has very similar procedure. First, the participants familiarise themselves with matching subphrases with the seven labels. Afterwards, they are asked to use the same labels but now assign them to subphrase combinations. In both cases, the correct answers are shown, afterwards. We also explain that the subphrase combinations can seem odd semantically but that if the participants encounter ungrammatical subphrase combinations, they can flag that using checkbox. Afterwards, the participants are shown 60 or 61 subphrase combinations that they then annotate. We clearly mark the segmentation of the phrase into two parts using colour. Participants receive fewer questions compared to the first study, due to the longer explanation that is included in Study 2, and due the fact that annotating longer phrases may simply take longer. At the very end, the participants can provide feedback on the task. This study constitutes step in Figure
Participants We recruited participants via Prolific, requiring them to be located in the United Kingdom. Further selection criteria were that they have listed English as their first language and that they have perfect Prolific approval rate over a minimum of 20 completed studies. Participants were initially compensated f2.20 for both studies. The median completion time for Study 1 was 10:30 minutes. The median completion time for Study 2 was 13:16 minutes; these partieipants were paid bonus to increase the mean reward per hour fll. In both studies, we excluded the results for participants whose sentiment annotations for the practice questions were, on average, more than point removed from the correct labels (according to SST), or for whom the correlation between their responses and the labels of these practice questions was below 0.8 (according to Spearman's p).
Annotations Figure 5 includes the distributions over the sentiment labels that the participants assign. For 104 phrases, there was one annotator that indicated that the phrase was ungrammatical, and for 5 phrases, annotators indicate that the phrase was ungrammatical. Sentiment label '3' represents the neutral label. In Study 1, this label is much more frequent compared to Study 2. This is in line with findings from Socher et al. (2013), who established that the longer the phrase, the more frequent the phrase considered sentiment-ladee Across the board, the positive phrases are slightly over- ever-represented. In Study 2, there are both natural and control stimuli, but the distributions are not substantially different.

---

C
Visualisation of the non-compositionalit ratings
Figure 6 and Figure we separate the MAXABS and ABS non-conpositionality ratings per sentiment composition type, per length combination, and per syntactic category combination. Only sentiment composition type is clear dominant factor lcading to higher ratings. Even though there are some longer phrase combinations with high average ratings, data from those categories is also rather scarce. Table 3 gives 60 examples of phrases whose MAXABS rating exceeds 1, ranked from the highest to the lowest.
3.
0.8
HHlI
0.9
1.1
sentiment
NP
corottbuurt
mnain
0.7
5+.4
0.8
S+ S+
non-compoditonallly rating
ren-campoiiionllly rating
non-compos borality rating
Figure 6: MAXABS ratings (that assign phrase "A B" the highest rating out of the two obtained for A and B).
shown separately per sentiment composition type, length combination type and syntactic category.
sentimet compostion
0.6
0.7
constituert
SAAR
54.6
Hilt:
5+
non www.compciitooallyyy.y...o...... rating
non-compsseenally rating
non-compestirrstly rating
Figure 7: ABS ratings (that include the ratings for subphrases A and B separately), shown per sentiment composition type, length combination type and syntactic category.

---

Fig.
nearly terminal case the franchise's s best years the excitement Phrase A
Phruse B
Ruting
Sent. II
Sent. R-1.
of the cutes are long past eating vatmeal done in artless style of horrified awe is in the works of explosions and violence of gory mayhem
4.11
5.33
2.00
-4.11
0.33
1.00
2.00
2.33
just simple fable pressure cooker something creepy und vague a uo-suuprisc scries the numerous scenes the momentary joys artsy fantasy sequeaces the overall feel of the film the contrived nature fans of the animated wildlife adventure show pointed little chiller a real human soul the comic heights the belly laughs the intellectual and emotional pedigree the experience of going to a film festival serious exploration a simple tale the real charm human story pale imitation main character the obligatory moments the actresses in the lead roles the dehate joins an audacious tour only document the life experiences the eatire point an inexpressible and drab wannabe everyone involved with movienuking a very original artist the visuals eccenticcitess violent initiation these curious owners this engaging mix sudden lunch rush the true potential u therapeutic zup exotic world the real story the most offensive thing the issue of faith a big box the playful paranoia the overlooked pitfalls modern of the chase sequence the sheer beauty compelling slice great actress much of the writing an especially poignant portrait the insight homesty the emotional the kind of shots new software program a very good time
-2.56
1.00
1.33
-2.56
1.00
3.67
2.33
3.67
2.00
-2.22
0.67
2.00
-189
1.67
3.00
of pretty and weightless intellectual entertainment that simply feel wrong is pretly cheesy of its provocative conclusion will be in warthog heaven
-1.89
2.0
flourishes
-1.78
-1.67
0.67
-1.67
1.35
2.00
1.56
5.67
5.00
about the frightening seductiveness of new technology
-1.56
2.67
4.00
baried beneath a spellbinding serpent's smirk
1.56
4.67
5.00
obviously desired
2.67
3.67
of lowbrow comedy of your date
1.56
4.67
-1.56
2.67
3.33
is rewarding of naclear temorism
1.56
6.00
5.00
1.56
4.33
4.00
of an unlikely friendship of this trifle
1.56
4.00
-1.44
3.67
4.00
pushed of the real deal
-1.44
2.00
-1.44
0.6
who sometimes defies sympathy of sentimental oove are all more than competent is a necessary and timely one the past of the worst possibilities mankind of purticular theatrical family of shaggy dog story looking for that exact niche
-1.44
1.67
2.33
-1.44
2.00
3.00
1.44
5.33
5.00
1.44
4.67
5.00
-1.3
2.67
1.33
1.00
1,33
3.00
1.35
4.00
3.00
1.35
2.00
1.00
is a con artist and liar in his medium
-.33
0.00
1.00
1.35
5.00
many the audience of architectural oddities of love and bloodletting at the diner of the medium
characters
1.33
3.00
-1.22
2,00
2.67
-1.22
2.67
3.00
1.22
5.00
4.67
-1.22
2.67
3.00
-1.22
3.00
4.00
treatment
3.33
belly dancing
3.67
just around the about the movie
1.22
3.67
3.00
-1.22
1.00
1.00
is not explored very deeply of consolation candy of the film's past endeavour
-1.22
1.00
2.00
1.22
3.67
3.00
1.22
3.67
-1.22
3.00
returns with chase to end chases
1.11
of his images of awkward emotions tearing into a landmark role is genuinely witty of her friendship of this disarming indie
1.11
5.67
5.00
LII
4.67
5. 5.00
-1.11
4.67
5.33
L.11
5.00
5.00
-1.11
3.67
5.00
of new blues soundtrack that fill gallery shows spit out the screenplay at the cinema
1.11
4.33
1.11
4.67
4.00
1.11
2.67
3.00
1.11
5.67
5.67
Table 3: 60 phrases with a rating that exceeds an absolute value of 1. We indicate the rating from the MAX variant of resource. We indicate whether we believe the phrase to represent figurative phrase, and also provide the sentiment assigned by the annotators (Sent. H.) and by ROBERTA- ROBERTA-LARGE LARGE (Sent. R-L)

---

D Models & model evaluation
Experimental setup
We fine-tume the following models, all obtained from the HuggingFace model hub:
ROBERTA-BASE
ROBERTA-LARGE ht http:///////////ooe..oo.oorrrrrr
TIMELM (Loureiro et al., 2022):
BERTTWEET (Nguyen et al., http:///wwp.//ggoooo..e////////a
Camacho-collauos et al. (2022):
Pérez et al. (2021):
Hartmann et (2021):
IMDB LOBERTA-BASE. http://///gggggggoo....oo//ottaa
The experimental setup used to fine-tune them on SST is as follows: we use batch size of 32, learning rate 5e-6, 6, the Adam optimiser with a cosine-based warmup scheduler (warmup is 20%). We train for 5 epochs, selecting the best model based on the validation data and evaluate that on the test sets. We do not experiment with these hyperparameters extensively, they are within the range of recommended settings and all models the same base models (ROBERTA- BASS ROBERTA-LARGEL
We train using NVIDIA A100-SXM-8060 GPUs which training model for one seed takes up 30 minutes for the BASE model, and 50 minutes for the LARGE model.
Characterising our experiments Through our experiments, we contribute to generalisation evaluation in NLP: evaluating models' generalisation capabilities beyond standard i.idd. testing, in particular through the evaluation on the 67 most non-compositional phrases. We characterise our experiments using the taxonomy of Hupkes et al. (2023): our experiments are cognitively motivated and focus on (non-)compositionaa generalisation. By separating inputs based on compositionality, shift between train and test data. That shift has a partitioned natural source, since the stimuli are from corpus, but the split we create is curated. In our experiments, we examine the influence of fine-tuning on SST-7 and evaluating on non-compositional examples (fine-tune train-test locus) but also examine the effect of a data shift in the pretrain train locus, by considering models with different pretraining corpora.
create covariate
Motivation
Practical
Cogaltive
Faimess
Generalisation type
Cumpositiona
Stractonul
Criss Tick
Crouz Longuage Ciow Domale
Robustiess
Shift type
Conariote
Label
Full
Arnoned
Shift source
Notumally werwing
Portiniuge natura!
Generttd stif)
Fully gronntes
Shift locus


Reviews:
[{"review_id": "CWijdcSgIC", "reviewer": "Reviewer_zamx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We were glad to hear you value the original approach of our work and the fact that we adequately list the existing limitations of our work; we appreciate the time and effort that went into the reviewing process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The resources generated by the authors could be useful to other researchers to further explore non-compositionality of sentiment, as well as the evolution of systems when sentiment nuances enter into play. The work is original and goes beyond the usual \"turn-of-the-screw, improving-on-benchmarks\" paper.", "reasons_to_reject": "The paper has certain limitations, mostly methodological, which the authors acknowledge and explain in detail, but I don't think these take away any value from the research.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper analyzes the role of non-compositionality of sentiment in phrases, understood as those whose sentiment deviates from what is expected. Given that success in sentiment analysis systems relies on the model's capability to compose sentiment, the paper aims to identify how non-compositional sentiment meaning is handled by systems. The authors extract a sample of 500 phrases from the SST dataset following certain criteria (phrases consisting of two sub-phrases that contain 3-8 tokens) and then replace one of the sub-phrases with other phrases of the same syntactic type and with the same sentiment. Then the phrases are annotated by annotators recruited via Prolific and annotation is carried out via an online survey system (Qualtrics). Inter-annotator agreement is calculated, returning a good score. The final sentiment labels for phrases are averaged and the non-compositionality ratings are calculated. The final dataset consists of 259 rated phrases. \n\nThe results show that most phrases follow the compositionality principle in terms of sentiment, but there are exception, which are usually the result of figures of speech, common sense reasoning, and  discourse relations.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IC6drBMCgB", "reviewer": "Reviewer_SRwS", "summary": "", "questions": "Can you summarize the conclusion of this paper?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughts about our work! It’s good to see that you appreciate the new perspective on sentiment analysis that goes beyond the standard methods of evaluation.\n\n**Regarding your question about the conclusion of this paper**: let us rephrase the conclusion using new words. Our findings are four-fold:\n- Firstly, we have successfully established a methodology to obtain non-compositionality ratings from human annotators. This is not as simple as one might expect when thinking about how to obtain those ratings: meaning composition is a process that occurs implicitly, and we cannot simply ask “is this sentiment compositional”. Therefore, care went into designing our experimental protocols, which led to a successful ranking for the 259 phrases involved.\n- Secondly, we have confirmed what is generally believed, namely that sentiment is a largely compositional process, where the sentiment of a phrase is fairly predictable from the sentiment of the subphrases.\n- Thirdly, nonetheless, we confirm that a subset of the phrases is not actually compositional, and we focus on the 67 with the highest non-compositionality scores (>1), in particular. We observe that some composition types (such as + - and - +) are more likely to be perceived as non-compositional, and that figurative phrases are more prominent among non-compositional examples. Non-compositional examples are also more challenging to the models, which we elaborate on in the response to reviewer btm9, and we will elaborate on this in the paper.\n- Fourthly, we show that the non-compositionality ratings can be used for more fine-grained model evaluation in the task of sentiment analysis.\n\nSpecifically, we would like to point out that you may be incorrect when mentioning that models can already handle compositional and non-compositional examples. In fact, models with reasonable SST F1 scores can have NonComp scores that are quite low, such as BertTweet and the model of Perez et al. in Table 1. \n\n**Regarding the metrics included in section 5**: we will elaborate the discussion of the results in the paper, including results presented below to reviewer btm9. In the table below, we report the MAE for the Pearson's correlations from Table 1. In general, the two metrics show similar patterns. We do believe the correlations to be a better fitting evaluation metric because of the emphasis on ranking examples based on their non-compositionality rather than getting the non-compositionality prediction exactly right. \n\n| model | Max (r / MAE) | MaxAbs (r / MAE) | All (r / MAE) | AllAbs (r / MAE) | AllClean (r / MAE) |\n| --- | --- | --- | --- | --- | --- |\n| roberta-base | 0.36   / 0.79  | 0.31   / 0.49  | 0.41   / 0.61  | 0.22   / 0.48  | 0.43   / 0.61  |\n| roberta-large | 0.42   / 0.74  | 0.38   / 0.47  | 0.44   / 0.59  | 0.30   / 0.46  | 0.46   / 0.58  |\n| timelm | 0.30   / 0.80  | 0.32   / 0.49  | 0.34   / 0.62  | 0.25   / 0.47  | 0.36   / 0.61  |\n| berttweet | 0.22   / 0.86  | 0.20   / 0.50  | 0.25   / 0.66  | 0.15   / 0.49  | 0.27   / 0.66  |\n| camacho-collados et al | 0.33   / 0.79  | 0.36   / 0.45  | 0.36   / 0.62  | 0.22   / 0.47  | 0.38   / 0.62  |\n| perez et al | 0.13   / 0.89  | 0.21   / 0.53  | 0.20   / 0.66  | 0.16   / 0.49  | 0.21   / 0.66 |\n| hartmann et al | 0.38   / 0.77  | 0.34   / 0.47  | 0.41   / 0.60  | 0.28   / 0.46  | 0.44   / 0.59 |\n| imdb roberta | 0.37   / 0.78  | 0.31   / 0.48  | 0.41   / 0.60  | 0.24   / 0.46  | 0.44   / 0.60 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This article presents a new perspective (combinatorial and non-combinatorial) to examine the prevailing sentiment analysis methods.", "reasons_to_reject": "1. Although this article provides a fresh perspective, it does not reach a conclusion. Or rather, the article suggests that existing sentiment analysis models are already capable of handling both combinatorial and uncombinatorial sentences. \n2. Only F1 and Pearson'R are used for evaluation, and MAE and MES will be more detailed when added.", "questions_to_authors": "Can you summarize the conclusion of this paper?", "paper_topic_and_main_contributions": "The author argues that most of the existing sentiment analysis research is combinatorial. So he did research on non-combinatorial sentiment analysis.\n\ncontributions are as follows: a) a methodology for obtaining those non-compositionality ratings, b) a resource of ratings for 259 phrases along with an analysis of that resource, and c) an evaluation of computational models for sentiment analysis.\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sr59pNME7Q", "reviewer": "Reviewer_btm9", "summary": "", "questions": "- Q1: Appendix A, L540:\n\t- \"4) The standard deviation ...\": What does it mean that the std dev lies within a range of 5? What's the std dev computed over? Only the sentiment scores of tokens in 1 phrase? Over all tokens in the dataset?\n\t- Assuming a normal distribution of sentiment values, a value lying outside of 5 std dev is very unlikely to occur in the dataset (probability of about 5.7e-7). How many phrases does that apply to?\n- Q2: Fig 3\n\t- What are the number of examples in each \"sentiment composition\" group?\n\t- The differences in mean values are quite small. Have you run a significance test showing if they are indeed statistically significant?\n\t- How do you explain the examples that are \"- +\" that have a very low non-compositionality rating? Are these examples mentioned in the introduction (e.g., \"yeah right\")? Adding such examples for illustration purposes would be useful.\n- Q4: L182: \"Together, the two ratings...\": How are the 2 numbers combined into one non-compositionality scores?\n- Q5: L184: Same question for ALL, ALLABS and ALLCLEAN. Correlation in Sec. 5/Tab. 1 indicates they are merged into 1 number. How?\n- Q6: Sec. 5\n\t- It would be useful to know the number of training examples and how they were selected. Is it all of SST training data minus the overlapping examples?\n\t- It would be very interesting to see the model performance being broken down by category as in Fig. 3 to further understand strengths and weaknesses of the models.\n- Q7: Tab. 1: It's expected that the *ABS variants perform worse than their non-*ABS counterparts, because the absolute value is correlated with the linear 7-point Likert scale of SST. A negative value in the former (which would mean low value in Likert scale) is mapped to a positive value (which would mean a higher value in Likert scale). As such seeing these 2 variants in a correlation analysis does not provide any useful information.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the elaborate review! These questions are very useful to further improve the writing of the paper. We answer your questions below.\n\n**Question 1**: The Stanford Sentiment Treebank originally collected sentiment labels using a multi-stop slider bar with 25 ticks. Every phrase was annotated by three annotators. The “range of five” refers to the standard deviation among those three annotations. This excludes approximately 5% of the data and acts as a filter that removes phrases that were very ambiguous in the original data annotation.\n\n**Question 2**: \n- Individual data points are indicated on the graph with dots. The group sizes vary from 4 to 68, with a mean of 28. Given the small dataset size and even smaller subgroup sizes, it is hard to draw definite conclusions about the significance of results, but within the results displayed in Figure 3, the `+-’ non-compositionality ratings are significantly (p<0.05) greater than those of six other composition types (~~, +~, ~+, ~-, ++, --) as per Welch’s t-test (because of the unequal variance and varying group sizes). The figurative expressions also have a significantly greater mean compared to the controls (p=0.013).\n- Regarding the examples with low non-compositionality scores and `- +’ sentiment: opposing sentiment is not necessarily a reason for non-compositional sentiment, even if the results show that it is more likely. In “A near terminal case of the cutes” (high non-compositionality and also - +) it is the case that the contrast amplifies in a non-compositional way, but in some other cases, such as “the problems and characters it reveals are universal and involving” (non-compositionality score of 0.22 for MaxABS) it is not. “the problems and characters it reveals <insert positive phrase>”/\"<insert negative phrase> are universal and involving\" thus behave more predictably, compositionally, compared to “a near terminal case of <insert positive phrase>”/\"<insert negative phrase> of the cutes\".\n\n**Question 3**: you may have omitted Q3 from your review. Feel free to post it in the comments during the discussion period if this was by accident.\n\n**Questions 4 and 5** (how are the two ratings combined into one number?): That depends on the ranking: in AllABS, All and AllClean, they are not combined into one number; they are included as two separate ratings. In Max and MaxABS, they are combined by taking the maximum of the two.\nTherefore, the correlations in Table 1 / Section 5 are computed over 259 x 2 numbers for All, AllClean and AllABS.\n\n**Question 6**: the training set includes the phrases normally included in training, excluding sentences that contain any of the phrases involved in obtaining the ranking (note that these include simple phrases such as “of a movie” that might appear in multiple sentences from the dataset). The SST training data normally includes 320k phrases (including sentences and subphrases from sentences), of which 160k unique. We train on 274k phrases.\n\nRegarding the breakdown of performance: when breaking down by composition type, some correlations are not reliable due to small subgroup sizes (Pearson’s r with p-values >> 0.05). As an alternative, two pieces of information are provided below:\n- The first markdown table shows the F1 of the 259 phrases, using the humans’ predictions as labels (different from the F1 for the test set from Table 1), and contrasts that to the F1 computed over just the 67 least compositional phrases.\n- The second markdown table shows the MAE for the MaxABS ranking, comparing the non-compositionality scores of the models to that of humans per composition type.\n\nBoth results illustrate that the composition types with higher non-compositionality ratings are also more challenging in terms of predicting the sentiment labels. Thanks for asking for these additional results; we would like to expand upon the results discussion in the camera-ready version of the paper.\n\n| model | F1 over all 259 phrases | F1 over 67 least compositional |\n| ----- | ----- |  ----- | \n| roberta-base | 40 | 32 |\n| roberta-large | 47 | 37 |\n| timelm | 43 | 38 |\n| berttweet | 43 | 30 |\n| camacho-collados et al | 49 | 43 |\n| perez et al | 42 | 26 |\n| hartmann et al | 45 | 33 |\n| imdb roberta | 47 | 43 |\n\n\n| model | ~ ~ | + ~ | — ~ | ~ + | ~ — | — + | + + | + — | — — |\n| ---- | ---- | ---- | ---- | ---- | ---- | ----| ----| ----| ----\n| roberta-base | 0.40| 0.49| 0.58| 0.50| 0.47| 0.92| 0.40| 0.49| 0.56 |\n| roberta-large | 0.43| 0.43| 0.57| 0.44| 0.44| 0.99| 0.36| 0.50| 0.61 |\n| timelm | 0.43| 0.50| 0.58| 0.47| 0.47| 0.75| 0.40| 0.50| 0.39 |\n| berttweet | 0.44| 0.47| 0.55| 0.54| 0.47| 0.77| 0.34| 0.67| 0.56 |\n| camacho-collados et al | 0.38| 0.45| 0.56| 0.49| 0.39| 0.63| 0.38| 0.49| 0.42 |\n| perez et al | 0.48| 0.55| 0.53| 0.56| 0.50| 0.73| 0.35| 0.65| 0.56 |\n| hartmann et al | 0.43| 0.44| 0.53| 0.49| 0.51| 0.99| 0.35| 0.48| 0.36 |\n| imdb roberta | 0.34| 0.47| 0.55| 0.51| 0.56| 0.90| 0.36| 0.51| 0.56 |\n| *mean* | 0.42 | 0.48 | 0.56 | 0.50 | 0.48 | 0.84 | 0.37 | 0.54 | 0.50 |\n\n\n**Question 7**: You are correct, and we mainly included all performance measures for consistency; the non-ABS variants might be useful during model analysis since they provide a more accurate representation of the targets, but the ABS variants are more intuitive during the data analysis. After all, we set out to compute a non-compositionality metric, which is more easily understood by the absolute value.\n\nThanks for going to great lengths in your review, even pointing out potential representation improvements. We were indeed short on space – 4 pages is a challenging paper length – and will gladly use some space from the 5th page to increase figures and caption sizes. We will revise the wording of the non-compositionality ratings carefully but opted for the somewhat odd phrasing of \"low/high non-compositionality rating\" because the *non-compositionality* rating is the central theme and is depicted in Figure 3 / Table 2, where higher numbers represented higher non-compositionality ratings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The analysis is interesting and gains some insights into compositionality of sentiment labels.\n- The created dataset is a nice resource.\n- The paper is nicely written and most steps seem very well thought through.\n", "reasons_to_reject": "- In some places details are missing (see questions below).\n- It's not clear to me how strong \"sentiment composition\" types relate to non-compositionality (Fig. 3). The absolute differences in the figure are rather small and details are missing in order to judge the strength of the statements (e.g., number of examples per box, test for significance).", "questions_to_authors": "- Q1: Appendix A, L540:\n\t- \"4) The standard deviation ...\": What does it mean that the std dev lies within a range of 5? What's the std dev computed over? Only the sentiment scores of tokens in 1 phrase? Over all tokens in the dataset?\n\t- Assuming a normal distribution of sentiment values, a value lying outside of 5 std dev is very unlikely to occur in the dataset (probability of about 5.7e-7). How many phrases does that apply to?\n- Q2: Fig 3\n\t- What are the number of examples in each \"sentiment composition\" group?\n\t- The differences in mean values are quite small. Have you run a significance test showing if they are indeed statistically significant?\n\t- How do you explain the examples that are \"- +\" that have a very low non-compositionality rating? Are these examples mentioned in the introduction (e.g., \"yeah right\")? Adding such examples for illustration purposes would be useful.\n- Q4: L182: \"Together, the two ratings...\": How are the 2 numbers combined into one non-compositionality scores?\n- Q5: L184: Same question for ALL, ALLABS and ALLCLEAN. Correlation in Sec. 5/Tab. 1 indicates they are merged into 1 number. How?\n- Q6: Sec. 5\n\t- It would be useful to know the number of training examples and how they were selected. Is it all of SST training data minus the overlapping examples?\n\t- It would be very interesting to see the model performance being broken down by category as in Fig. 3 to further understand strengths and weaknesses of the models.\n- Q7: Tab. 1: It's expected that the *ABS variants perform worse than their non-*ABS counterparts, because the absolute value is correlated with the linear 7-point Likert scale of SST. A negative value in the former (which would mean low value in Likert scale) is mapped to a positive value (which would mean a higher value in Likert scale). As such seeing these 2 variants in a correlation analysis does not provide any useful information.\n", "paper_topic_and_main_contributions": "- The paper aims to identify which phrases are compositional in terms of sentiment and which are not. Non-compositionality means that the sentiment of the entire phrase is \"unexpected given the sentiment of the individual parts\".\n- The authors create a new datasource (subset of SST2) with non-compositionality labels. They analyze the datasource and run classification models on it.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Be careful with shrinking vertical spaces, e.g., under table/figure captions. It blends into the text too much.\n- The paper could be easier to follow if it was using \"compositionality\" vs \"non-compositionality\", because the paper often talks about low non-compositionality (which is basically double negation)."}}]

Author rebuttal:

Meta-review: The paper explores the role of non-compositionality in sentiment within phrases, aiming to understand how sentiment analysis systems handle phrases whose sentiment doesn't align with expectations. Utilizing a sample of 500 phrases from the SST dataset, the authors conduct a rigorous experiment involving syntactic replacements and human annotations. The annotation process employs an online survey system, Qualtrics, and yields a good inter-annotator agreement score. The final dataset comprises 259 rated phrases, which are then analyzed for their non-compositionality ratings.

The findings reveal that although most phrases adhere to the principle of compositionality in terms of sentiment, exceptions exist. These are typically the result of figures of speech, common sense reasoning, and discourse relations. The paper not only offers valuable insights into the compositionality of sentiment labels but also provides a useful dataset for further research. Written clearly and thoughtfully, the paper presents a novel perspective for evaluating existing sentiment analysis methods.

Overall, despite minor concerns of reviewers, I would still suggest the paper get accepted.
Decision: Accept-Findings