# Multiword expressions in computational linguistics

Carlos Ramisch

► To cite this version:

Carlos Ramisch. Multiword expressions in computational linguistics: Down the rabbit hole and through the looking glass. Computer Science [cs]. Aix Marseille Université (AMU), 2023. tel-04216223

HAL Id: tel-04216223

https://theses.hal.science/tel-04216223

Submitted on 23 Sep 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Multiword
expressions in
computational
linguistics

Down the rabbit hole and through
the looking glass

Carlos Ramisch

Habilitation à diriger des recherches

<figure><img src="image_2.png" /></figure>

Aix*Marseille

université

Socialement engagée
Carlos Ramisch. 2023. *Multiword expressions in computational linguistics: Down the rabbit hole and through the looking glass* (Habilitation à diriger des recherches). Aix Marseille University.

This manuscript adapts the template from Language Science Press used by the
Phraseology and Multiword Expressions book series::

https://langsci-press.org/catalog/series/pmwe

© 2023, Carlos Ramisch

Published under the Creative Commons Attribution 4.0 Licence (CC BY 4.0):

http://creativecommons.org/licenses/by/4.0/

<figure><img src="image_3.png" /></figure>

Evaluation committee:

• Francis Bond (reviewer)

• Alain Polguère (reviewer)

• Leo Wanner (reviewer)

• Agnès Tutin

• Alexis Nasr (tutor)

Cover and concept of design: Ulrike Harbort

Fonts: Libertinus, Arimo, DejaVu Sans Mono

Typesetting software: X\LATEX
Acknowledgements

Dedicating one's career to the study of multiword expressions might sound odd. But it is not chance nor fate: it is a choice. I enumerate some scientific motivations for this decision in §2.3. But truth be told, *people* are the real stimulus for my profound love for multiword expressions. There is a *je ne sais quoi* in this topic that attracts incredible human beings who are also talented researchers. I feel incredibly at home in this community and, at the same time, being a part of it allows me to connect with diverse viewpoints.

I am deeply grateful to my NLP colleagues and friends. Aline, for her support and loyalty throughout the years, and for releasing me from Manichean thinking thanks to the “ice-cream axiom”. Agata, for her humour, intelligence, organisation, and for being my accomplice in over-optimistic hiking time estimates. Alexis, for his trust, scientific inspiration, his talent for seeing the big picture, and his reading recommendations over lunch. Marco, for his ability to empathetically disagree and deeply listen. Stella, for telling me all that I know about Mediterranean plants and Greek mythology. Benoit, for being a great office mate, and for generously sharing his imaginativeness and serenity. Marie, for her impetus, her attention to the details, and for reminding me that a phone call is worth a thousand emails. Mathieu, for his tranquillity, leadership, and creativity. Silvio, for his hard work, great hikes, Python programming tips, and for keeping calm when we had to break into a car in the middle of the desert. Caroline, for her courage to make a second PhD, and for preparing the best meetings ever. Manon, for her joy, enthusiasm, stamina, and for encouraging me to finish this manuscript. Léo, for sharing his curiosity and strength. The TALEP and PARSEME teams: being part of them is a pride and a daily source of inspiration. Thanks to Skype, Zoom, Overleaf, Gdocs, and all the wonderful technology that makes collaboration possible regardless of geography. Thanks to my co-authors, who enrich my research and life with their diverse backgrounds and skills.

Finally, I would like to thank my inner circle: Damien, Solange, Lorreine, Renata, Margaux, my parents. Thanks for always laughing when I make marginally funny multiword expressions jokes. Thanks for your non-compositional love which is much much greater than the sum of its parts.

Marseille, 2022
*El lenguage, mal que les pese a las Academias de la Lengua, nos pertenece a
la gente que lo usamos, que lo vivimos, que nos nombramos a través de él.
Atrevernos a usar un lenguage que nos represente, sin necesidad de tener el
permiso de la Academia, es una forma de subversión.<fnref n="1" />*

— Brigitte Vasallo, *Piensamiento monógamo, terror poliamoroso*

<footnote n="1">¹Language, notwithstanding the opinion of the Language Academies, belongs to us, the people who use it, who live it, who name ourselves through it. To dare using a language which represents us, without having to ask the authorisation of the Academy, is a form of subversion.</footnote>
Abstract

One of the most intriguing phenomena in human languages is the creation and
use of idiomatic expressions which defy all rules of logical composition. For in-
stance, in Brazilian Portuguese, one can express disagreement with [pt nem
aqui nem na China] (lit. ‘and-not here and-not in-the China’) ‘absolutely not’ or
[pt nem que a vaca tussa] (lit. ‘and-not if the cow coughs’) ‘absolutely never’.
Idioms like these are prototypical multiword expressions (MWEs), that is, odd in-
terpretations associated to particular word combinations.

Much ink has been spilled on *computational processing of MWEs* since the fa-
mous “pain-in-the-neck” paper by Sag et al. (2002). This manuscript overviews
research on this topic, with a particular focus on my own scientific interests. I
start with a descriptive chapter covering both the linguistic phenomenon and
its computational processing, motivating and illustrating abstract notions with
textbook-style examples.

The two following chapters cover the tasks of automatic MWE discovery and
identification. For both chapters, I start by surveying resources (datasets, cor-
pora), including those whose creation I contributed to. Then, I cover the mod-
els used to (a) predict the compositionality of nominal compounds in English,
French, and Portuguese, and (b) identify verbal MWEs in running text in the con-
text of the PARSEME project. Both chapters detail the challenges in evaluating
the tasks, and contain empirical evaluation results.

Last but not least, I summarise my main results and explore paths of future
research which look promising to me. These include the follow-up of MWE pro-
cessing, semantic lexicon induction, and diversity-oriented NLP. More than a
synthesis, this manuscript contains original surveys of related work, contextu-
alises, extends, and articulates my contributions to the field.
<figure><img src="image_4.png" /></figure>
Résumé<fnref n="2" />

Un des phénomènes les plus fascinants des langues humaines est la création et l'utilisation d'expressions idiomatiques qui défient toutes les règles de composition logique. Par exemple, en portugais brésilien, on peut exprimer un désaccord avec *nem aqui nem na China* (lit. 'et-pas ici et-pas en Chine') 'absolument pas' ou *nem que a vaca tussa* (lit. 'et-pas si la vache tousse') 'absolument jamais'. Les expressions idiomatiques de ce type sont des *expressions polylexicales* (*EP*) prototypiques, c'est-à-dire des interprétations idiosyncratiques associées à des combinaisons de mots particulières.

Beaucoup d'encre a coulé sur le *traitement informatique des EP dans le TAL* depuis le célèbre article de Sag et al. (2002). Le présent manuscrit donne un aperçu de la recherche sur ce sujet, en mettant l'accent sur mes propres intérêts scientifiques. Je commence par un chapitre descriptif couvrant à la fois le phénomène linguistique et son traitement informatique, motivant et illustrant les notions abstraites par des exemples pédagogiques.

Les deux chapitres suivants couvrent les tâches d'identification et de découverte automatique d'EP. Pour ces deux chapitres, je commence par passer en revue les ressources (jeux de données et corpus), notamment celles auxquelles j'ai contribué. Ensuite, je présente les modèles utilisés pour (a) prédire la compositionnalité des EP nominales en anglais, français et portugais, et (b) identifier les EP verbales en contexte, dans le cadre du projet PARSEME. Les deux chapitres détaillent les défis posés par l'évaluation de ces tâches et contiennent des résultats d'évaluation empiriques.

Enfin, je résume mes principales contributions et explore les pistes de recherche futures qui me semblent prometteuses. Celles-ci incluent la poursuite du travail sur les EP, l'induction de lexiques sémantiques, et le TAL orienté diversité. Plus qu'une synthèse, ce manuscrit contient des études originales de travaux connexes, contextualise, étend et articule mes contributions au domaine.

<footnote n="2">²Translated with the help of DeepL: https://www.deepl.com/</footnote>
<figure><img src="image_5.png" /></figure>
# Contents

<table><tr><td>1</td><td>Introduction</td><td>1</td></tr><tr><td>1.1</td><td>Chapters' walk-through</td><td>3</td></tr><tr><td>2</td><td>MWEs in a nutshell</td><td>5</td></tr><tr><td>2.1</td><td>First things first</td><td>5</td></tr><tr><td>2.1.1</td><td>The building blocks</td><td>5</td></tr><tr><td>2.1.2</td><td>MWE definitions: a rat's nest</td><td>7</td></tr><tr><td>2.1.3</td><td>Getting notation out of the way</td><td>12</td></tr><tr><td>2.1.4</td><td>All shapes and sizes</td><td>13</td></tr><tr><td>2.1.5</td><td>A hard nut to crack</td><td>19</td></tr><tr><td>2.2</td><td>Getting our hands dirty</td><td>21</td></tr><tr><td>2.2.1</td><td>A task definition taken for granted</td><td>21</td></tr><tr><td>2.2.2</td><td>Resources worth their weight in gold</td><td>24</td></tr><tr><td>2.2.3</td><td>A pain in the neck or a bed of roses?</td><td>26</td></tr><tr><td>2.3</td><td>A big deal</td><td>27</td></tr><tr><td>2.3.1</td><td>A whole lot of them</td><td>27</td></tr><tr><td>2.3.2</td><td>Flowing like a river</td><td>31</td></tr><tr><td>2.3.3</td><td>Getting to the meaning</td><td>32</td></tr><tr><td>2.3.4</td><td>There is beauty in chaos</td><td>33</td></tr><tr><td>2.3.5</td><td>What if Jelinek was right?</td><td>34</td></tr><tr><td>2.4</td><td>In short</td><td>35</td></tr><tr><td>2.5</td><td>For the record</td><td>36</td></tr><tr><td>3</td><td>Fifty shades of compositionality</td><td>37</td></tr><tr><td>3.1</td><td>A word on discovery</td><td>38</td></tr><tr><td>3.2</td><td>Resources</td><td>40</td></tr><tr><td>3.2.1</td><td>Existing datasets</td><td>41</td></tr><tr><td>3.2.2</td><td>Discussion</td><td>45</td></tr><tr><td>3.2.3</td><td>Nominal compounds in English, French and Portuguese</td><td>51</td></tr><tr><td>3.2.4</td><td>Fine-grained annotation of literal occurrences</td><td>63</td></tr><tr><td>3.3</td><td>Methods and evaluation</td><td>68</td></tr><tr><td>3.3.1</td><td>Existing methods</td><td>68</td></tr></table>
Contents

<table><tbody><tr><td>3.3.2</td><td>Compositionality prediction</td><td>72</td></tr><tr><td>3.3.3</td><td>Experiments and results</td><td>75</td></tr><tr><td>3.3.4</td><td>Going further</td><td>79</td></tr><tr><td>3.4</td><td>In short</td><td>80</td></tr><tr><td>3.5</td><td>For the record</td><td>81</td></tr><tr><td>4</td><td>Down-to-earth MWE identification</td><td>83</td></tr><tr><td>4.1</td><td>Setting the scene</td><td>83</td></tr><tr><td>4.1.1</td><td>Sparks of an idea</td><td>84</td></tr><tr><td>4.1.2</td><td>Sequence tagging</td><td>84</td></tr><tr><td>4.1.3</td><td>By-product of parsing</td><td>86</td></tr><tr><td>4.1.4</td><td>DiMSUM and PARSEME: the big bang</td><td>89</td></tr><tr><td>4.2</td><td>The PARSEME galaxy: corpora</td><td>90</td></tr><tr><td>4.2.1</td><td>Guidelines: finding true North</td><td>91</td></tr><tr><td>4.2.2</td><td>Environment and tools</td><td>100</td></tr><tr><td>4.2.3</td><td>Corpus stats</td><td>104</td></tr><tr><td>4.3</td><td>The PARSEME galaxy: shared tasks</td><td>108</td></tr><tr><td>4.3.1</td><td>Countdown: data preparation</td><td>109</td></tr><tr><td>4.3.2</td><td>Evaluation metrics are not rocket science</td><td>113</td></tr><tr><td>4.3.3</td><td>MWE identification systems go into orbit</td><td>117</td></tr><tr><td>4.3.4</td><td>Results: Houston, we got a problem</td><td>124</td></tr><tr><td>4.4</td><td>In short</td><td>126</td></tr><tr><td>4.5</td><td>For the record</td><td>128</td></tr><tr><td>5</td><td>The big picture</td><td>129</td></tr><tr><td>5.1</td><td>Summary of MWE contributions</td><td>129</td></tr><tr><td>5.1.1</td><td>Theoretical framework</td><td>129</td></tr><tr><td>5.1.2</td><td>Methodological framework</td><td>131</td></tr><tr><td>5.1.3</td><td>Empirical results</td><td>131</td></tr><tr><td>5.1.4</td><td>Resources</td><td>132</td></tr><tr><td>5.1.5</td><td>Software</td><td>133</td></tr><tr><td>5.2</td><td>Summary of other NLP contributions</td><td>134</td></tr><tr><td>5.3</td><td>To infinity and beyond!</td><td>136</td></tr><tr><td>5.3.1</td><td>PARSEME 2030: keeping the ball rolling</td><td>136</td></tr><tr><td>5.3.2</td><td>Without lexicons, NLP cannot fly</td><td>139</td></tr><tr><td>5.3.3</td><td>Diversity in NLP: the more the merrier</td><td>143</td></tr><tr><td>References</td><td></td><td>145</td></tr></tbody></table>
# 1 Introduction

O falar não se restringe ao ato de emitir palavras, mas a poder existir.<fnref n="1" />

— Djamila Ribeiro, *Lugar de fala*

You just moved to Budapest. For the last few years, you have been learning the Hungarian language. You feel quite comfortable with its 18 nominal cases and start appreciating its relatively free word order. You meet your new Hungarian boss, who friendly greets you [hu] *pálinkás jó reggelt* (lit. ‘good morning with palinka’)! You get a bit worried about your boss’ breakfast habits.<fnref n="2" /> When the work day is over, she invites you to a happy hour with friends, and adds [hu] *nem erőszak a disznótor* (no violence the pig-killing | lit. ‘the pig killing is no offense’). The awkwardness of this phrase makes you think that Hungarian is indeed “the only tongue in the world that the devil respects” (Buarque 2003).

Fortunately, no one tries to kill a pig that evening. Also, your boss does not drink brandy with her morning meal. Actually, when you learn a new language, there are numerous situations like this, when words are combined in unexpected ways yielding unpredictable meanings, regardless of the fact that you are familiar with the words, their individual meanings, and how to combine them. Such odd situations are frequently due to the presence of **multiword expressions**. These expressions abound not only in Hungarian, but in all human languages. They are a pain in the neck for language learners, leading to all kinds of more or less hilarious or dramatic misunderstandings.

Humans are social animals, and languages are the thread that weaves relationships among them. Speaking and writing a language enables basic socialisation such as talking with your boss, ordering food, or getting basic health care. Mastering the language employed by our community allows us not only to meet our vital needs, but also to acquire trust, credibility, affection and even privilege and power. Conversely, limited language proficiency can be a serious drawback, for example, for a migrant in a new linguistic environment, like in the anecdote above, ultimately leading to marginalisation or exclusion.

<footnote n="1">¹Speaking is not only emitting words, but being able to exist.</footnote>

<footnote n="2">²Pálinka is a Hungarian spirit with about 40% alcohol concentration.</footnote>
# 1 Introduction

Languages are one of the foundations of all contemporary human societies. The ability to speak about abstract concepts, keep written records and tell stories is believed to be one of *homo sapiens'* evolutionary advantages with respect to other species (Harari 2015). Furthermore, the language faculty is often associated with the notion of (artificial) intelligence itself (Turing 1950).

The development of computer systems and devices able to assist us in tasks related to language constitutes a major achievement in recent human history. Such systems open exciting perspectives to enhance communication among people and their interaction with computational devices. The holy grail of language technology is often depicted as a machine able to understand the meaning of any utterance in any language, process its content (e.g. translate it), and then generate a natural utterance in response. Examples of such universal translators in science fiction include devices such as Doctor Who's TARDIS and the babel fish (Adams 1979).

Beyond fiction, limited-capability machines of this kind are not only a reality, but are becoming omnipresent in our daily lives. Decent translations can be obtained for free from Google Translate and DeepL on any smartphone; most big tech companies offer vocal assistant services such as Alexa and Siri; chatbots are the norm in customer service; automatic text completion, spell checking and question answering technologies are massively deployed as components of information retrieval and messaging, etc.

In spite of vertiginous progress, if we ask an online translation system to help us with the pig-killing situation, it will not be of much help.<fnref n="3" /> Ideally, a computer should be able to grasp the *meaning* of the whole expression instead of dully combining each word's meanings. The expression actually has nothing to do with pigs or butchers, but it simply means that your colleague would not be offended if you declined her invitation to the happy hour.<fnref n="4" /> This meaning is far from obvious given only the words of the expression and the way they are combined. Nonetheless, it seems reasonable to expect that the babel fish or the TARDIS should be more useful than Google Translate to help us interpret and reply appropriately.

Fast progress in language technologies is driven by the research community in **natural language processing** (NLP), of which I am part. My personal holy grail

<footnote n="3">³Google translate: en no violence to the pig or fr pas de violence envers le cochon (May 1, 2021, https://translate.google.com/).</footnote>

<footnote n="4">⁴In Hungarian villages the “pig killing” is a big group activity/party: it is positive to be invited. The idiom is used when someone offers something with good intentions, but does not want to be pushy about it: “I am inviting you to the pig killing to have fun, but I will not take offense if you do not come”. Katalin Simkó and Veronika Vincze, personal communication.</footnote>
in NLP would be achieving robust and precise representation and processing of
multiword expressions (such as the idioms above) in computational linguistic
models and applications. Much water has flown under the bridge between the
famous pain-in-the-neck paper (Sag et al. 2002) and the latest edition of the mul-
tilingual PARSEME shared task (Ramisch et al. 2020). For the last 15 years, I have
been eating, sleeping and breathing multiword expressions, witnessing and con-
tributing to significant advances in how we deal with them in text processing. It
comes as no surprise that in this thesis, which summarises my research contribu-
tions, MWEs play the main role. This manuscript is designed as a partial survey,
that is, one in which I do not only summarise and contextualise my work with
respect to related work, but also share my understanding of the field, its achieve-
ments, open issues, and how they intertwine with my own research record and
perspectives.

Because the phenomenon is so rich, I have mostly succeeded in coming up with
new ways to illustrate what multiword expressions are and why it is important to
work on this subject (e.g. the Hungarian examples above). Nonetheless, to make
this manuscript more self-contained, parts of it are copied and/or slightly adapted
from previous publications. As a convention, I will use a *lighter font color* to
indicate these parts. The publications from which these excerpts were taken are
listed at the end of each chapter, as well as the co-authors who contributed to
them. I will try to keep self-plagiarism to a minimum, favouring single-authored
publications and focusing on the parts of these papers to which I contributed.

1.1 Chapters’ walk-through

Chapter 2 motivates the need for MWE research, illustrates and updates the main concepts that forged my view of the field throughout the years. Then, my contributions are structured into two parts corresponding to MWE-related tasks: discovery and identification. Chapter 3 covers MWE discovery. First, I describe resources that I have contributed to, in particular to model type-based compositionality of nominal MWEs. Second, I describe and evaluate methods used to discover MWEs and predict their compositionality using word embeddings. Third, I present a study on in-context annotation of literal, idiomatic and coincidental occurrences of potential MWEs.

The second task, MWE identification, is covered in Chapter 4. Again, I start
with an overview of the existing resources and methods. Then the rest of the
chapter focuses on the PARSEME corpora and shared tasks. Within this frame-
work, I summarise several collaborations that aimed at developing systems and
# 1 Introduction

models for automatic MWE identification, from rule-based methods to neural networks. Finally, I provide a snapshot of the state of the art in MWE identification which can serve as a basis for the definition of the next steps in this task.

Last but not least, Chapter 5 recalls my main contributions and explores paths of future research which look promising to me. Notice that each chapter contains a final section with the highlights, for those who prefer going straight to the point, and a detailed list of publications to acknowledge my colleagues without whom the research described in this manuscript would not have been possible. Without further ado, I invite you for a journey into the fascinating world of NLP research in multiword expressions. And of course, always remember that there is no violence in the pig killing ✡.
# 2 MWEs in a nutshell

> Assim é que esta história será feita de palavras que se agrupam em frases e destas se evola um sentido secreto que ultrapassa palavras e frases.<fnref n="1" />

> — Clarice Lispector, *A hora da estrela*

This chapter sets the background and defines the scope of the research presented in the next chapters. We start by circumscribing the linguistic notion of MWE (§2.1), then we describe the computational tasks associated with the phenomenon – discovery and identification – that structure this manuscript (§2.2). Finally, we motivate the interest of the NLP community in MWEs (§2.3).

## 2.1 First things first

Before introducing the computational concepts involved in MWE processing, we need to detail some linguistic aspects. This section overviews the basic notions, definitions and scope of the MWE phenomenon. Given the rapidly evolving nature of the field, this review helps contextualising the research contributions presented in the next chapters.

### 2.1.1 The building blocks

Multiword expressions are composed of several *words*. Therefore, a proper definition of MWEs requires clarification of the meaning of the word *word* itself. Most linguists and computational linguists will agree that this is a tricky question with no consensual answer (Mel'čuk et al. 1995; Manning & Schütze 1999; Church 2013). For instance, in Universal Dependencies, “words are the basic elements connected by dependency relations; they have morphological properties and enter into syntactic relations” (de Marneffe et al. 2021). Although useful in practice, this definition lacks operational criteria to deal with borderline cases, for which we can only rely on each language’s traditional grammar.

<footnote n="1">¹Hence this story will be made of words which are grouped into sentences from which a secret sense gives off surpassing words and sentences.</footnote>
## 2 MWEs in a nutshell

Alternatively, we can refer to a slightly different notion, assuming that MWEs are formed by multiple lexemes instead of words. Lexemes are lexical items (or units) or elementary units of meaning that represent basic blocks of a language's lexicon. Most of what we call "words" in everyday language are actually lexemes. Affixes like the plural marker -s or the final -ing in gerund verbs are not lexemes. A useful test to define a lexeme is to ask whether it should be listed as a dictionary headword. Although in this manuscript I will often employ the popular and most widely employed terms "word" and "multiword", it would have been more precise to talk about "lexemes" instead. By the way, in French, the standard term for MWE is *expression polylexicale* (polylexical expression), preferring the *lexis* radical over the less precise term *word*.

Lexemes (or words) are a linguistic notion that is often confused with the related computational notion of tokens. Tokens are the result of a computational process of tokenization, that is, splitting the text into units for further processing (e.g. parsing, translation, and so on). Tokens can also be non-word units such as punctuation, dates, URLs, etc. The word-token distinction adopted here stems from the one adopted in PARSEME,<fnref n="2" /> which in turn is based on the Universal Dependencies guidelines.<fnref n="3" />

Ideally, tokens and lexemes should have a 1-to-1 correspondence, that is, they should be equivalent notions. In languages that use spaces to separate words, this is relatively straightforward to achieve. However, perfect lexeme tokenization based on spaces between words is not always possible, due to complex linguistic phenomena such as compounding (*snowman*, *dataset*), contractions (*don't*), clitics (*Laura's*) and orthography conventions (*pre-existing*, *part-of-speech tag*). Moreover, the use of whitespace to separate words is not universal: some languages such as Chinese and Japanese do not visually split words, whereas others may have different conventions for some parts of speech (e.g. nouns in German compounds are not separated by spaces).<fnref n="4" /> Single-token compounds exist also in other Germanic languages such as English (e.g. *snowman*, *wallpaper*) and may be difficult to tell apart from regular prefixation (e.g. is *out* in *outcome* a prefix or a word?).<fnref n="5" />

As a consequence, MWEs may or may not contain spaces, as this depends on orthography conventions and/or on specific tokenisation software used to

<footnote n="2">²See Savary et al. (2018: p. 92) and the online guidelines for details: https://parsemefr.lis-lab.fr/parseme-st-guidelines/1.2/?page=wordsandtokens.</footnote>

<footnote n="3">³See de Marneffe et al. (2021: p. 259) and the online guidelines for details: https://universaldependencies.org/u/overview/tokenization.</footnote>

<footnote n="4">⁴See the Rhababerbarbara story: https://www.youtube.com/watch?v=IFOysFAKnM</footnote>

<footnote n="5">⁵The PARSEME guidelines have a whole section on particles vs. prefixes: https://parsemefr.lis-lab.fr/parseme-st-guidelines/1.2/?page=particles.</footnote>
split the text into tokens. In practical terms, we consider that multiword tokens (e.g. *whitespace*) can be MWEs, since they contain at least two lexemes (*white* and *space*), whereas multi-token words (e.g. [fr] *aujourd’hui* ‘today’ tokenised as *au-jourdُ,’[hui]*) are never MWEs since their components are not standalone lexemes. However, being a multiword token is not a sufficient criterion to be considered as a MWE (§2.1.2).

2.1.2 MWE definitions: a rat’s nest

Multiword expressions are often seen as a fuzzy concept, a sort of vague “umbrella” term under which a large number of heterogeneous linguistic objects can be grouped (see §2.1.4). As a consequence, definitions abound. Probably the shortest definition we could give is the following:

**Definition 2.1** *Multiword expressions are words that belong together.*<sup>6</sup>

On the other hand, the phenomenon can be finely described and much ink has
been spilled to define and characterise MWEs. For example, the PDF version of
edition 1.2 of the PARSEME guidelines – covering only verbal MWEs – has 63
pages when only English examples are shown and 134 pages with examples for
all languages.

Between the laconic four-words definition 2.1 and the dozens of pages of the de-
tailed PARSEME guidelines, intermediate-length attempts to define MWEs usu-
ally focus on different aspects, with slightly different scopes. Smadja (1993) em-
phasizes frequency, defining them as “arbitrary and recurrent word combina-
tions”. Choueka (1988) states that a MWE is “a syntactic and semantic unit whose
exact and unambiguous meaning or connotation cannot be derived directly from
the meaning or connotation of its components”. In the famous “pain-in-the-neck”
paper, Sag et al. (2002) define MWEs as “idiosyncratic interpretations that cross
word boundaries (or spaces)”. An excellent survey of MWE definitions is pro-
vided in appendix B of Seretan (2011: p. 182-184).

One of the reasons for this diversity in definitions is the fact that several
knowledge areas, with multiple goals and points of view, have interest in the
phenomenon. These include linguistics, computational linguistics, cognitive sci-
ences, and subfields such as phraseology, lexicography, terminology, semantics,
corpus annotation, e-lexicography, etc. In computational linguistics, and in par-
ticular in the MWE community, the two definitions below are widely adopted in
most recent work:

<footnote n="6">⁶This minimalist definition was suggested to me during an informal chat when I visited the University of Granada in 2019.</footnote>
**Definition 2.2** "*Multiword expressions are lexical items that: (a) can be decomposed into multiple lexemes; and (b) display lexical, syntactic, semantic, pragmatic and/or statistical idiomaticity*" (Baldwin & Kim 2010).

**Definition 2.3** "*Multiword expressions are understood as (continuous or discontinuous) sequences of words which (a) contain at least two component words which are lexicalised, i.e. always realised by the same lexemes, including a head word and at least one other syntactically related word, and (b) display some degree of lexical, morphological, syntactic and/or semantic idiosyncrasy*" (Savary et al. 2018).

A broad discussion of MWE definitions is out of scope of this work. Instead, we will focus on comparing the standard definition 2.2 by Baldwin & Kim (2010) with the updated and narrower definition 2.3 provided by PARSEME, briefly discussing their three main axes: idiomaticity, structure, and lexicalisation. In this manuscript, we adopt definition 2.3, in line with the PARSEME framework.

**Idiomaticity or idiosyncrasy** Part (b) of both definitions focuses on exceptional or unpredictable behaviour, referring to it as “idiomaticity” (2.2) and “idiosyncrasy” (2.3). Both refer to the fact that MWEs deviate from standard composition rules, resulting in unpredictable combinations. Hence, we could loosely define MWEs as “exceptions that occur when words are combined”. This idiosyncrasy is often semantic, “the meaning of a MWE not being explicitly derivable from its parts” (Baldwin & Kim 2010), for instance, *flower + child* does not add up to the meaning of *flower child* ‘hippie’. While semantic idiomaticity is one of the most prototypical characteristic of MWEs, they can also have other types of idiosyncrasies. For example, *syntactic idiomaticity* occurs when we combine lexical items in ways that seem to breach syntactic rules, like the strange inflection of the verb *to be in truth be told* ‘honestly’. Both definitions agree that idiosyncrasies may be of lexical, syntactic and semantic nature. Definition 2.2 includes pragmatic and statistical levels, the latter being often a consequence of other types of idiosyncrasy. On the other hand, definition 2.3 adds morphology (e.g. the lack of agreement in *fr grand-mère* (lit. ‘great.MASC-mother.FEM’) ‘grandmother’). The specific type of idiosyncrasy often depends on a project’s goals and scope, and can be adjusted using more detailed criteria, tests and examples. I consider that the idiosyncrasy of MWEs is not limited to semantic idiomaticity, but this manuscript does not cover purely statistical idiosyncrasy (collocations).

**Syntactic cohesion** In most definitions, the genus of MWE is either “(lexical) unit” (2.2) or “sequence” (2.3), the former being more appropriate for lexicogra-
phy whereas the later is a better fit for annotation.<fnref n="7" /> Although these terms do not specify the internal structure of the two or more lexemes involved in the MWE, structural assumptions are usually implicit in the (syntactic) framework of the definition. For example, PARSEME’s definition 2.3 relies on dependency syntax, specifically on Universal Dependencies (UD), defining MWEs as subtrees formed by lexemes that are not necessarily contiguous in text. Differently from phrases or chunks, subtrees in UD’s lexicalist model allow excluding variable items from the annotation span, for example, *pt ter como/por objetivo* (lit. ‘have as/for objective’) ‘to have as objective’ excludes the interchangeable prepositions, which do not link the verb and the object as in traditional dependency syntax, but rather depend on the noun *objetivo*. On the other hand, annotating MWEs formed only by function words (e.g. *even though*) requires acrobatic workarounds such as using UD’s *fixed relation*. Even though they are mutually dependent on each other, the French double negation particles *ne* and *pas* cannot be considered as MWEs in PARSEME because both depend on a variable verb, excluded from the annotation span. As we can see, the choice of underlying syntactic formalism, although apparently minor, influences the scope of what counts as a MWE. The syntactic cohesion property, often taken for granted in MWE definitions, matches the intuition that an MWE “acts as a single unit at some level of linguistic analysis” (Calzolari et al. 2002). If MWEs behave as atomic units to some extent, they can be assigned parts of speech (Kahane et al. 2017) and senses (Schneider & Smith 2015), although it is unclear whether their tagsets should be the same as those employed for single words.

### Lexicalised components
Anyone carrying out corpus annotation or resource creation knows that the devil hides in the details. One question that quickly arises when annotating MWEs concerns their span. For instance, should the determiner *the* be annotated as part of the MWE in *take the shower*? Or in *take the cake*? In PARSEME, the guidelines for the annotation span rely on the notion of lexicalised components. A lexicalised component is a component word that is always realised by the same lexeme in all possible occurrences of the MWE. As a corollary, lexicalised components cannot be omitted or replaced by synonyms, otherwise the MWE would become ungrammatical or acquire a new (non-idiomatic) sense.<fnref n="8" /> Thus, only lexicalised components are part of the MWE,

<footnote n="7">⁷The term “combination” sometimes denotes a discontinuous sequence, although sequences do not need to be continuous, as formalised in Savary, Cordeiro, Lichte, et al. (2019: pp. 9–15).</footnote>

<footnote n="8">⁸For instance, in *take the shower*, the determiner *the* is not a lexicalised component, since it can be replaced by another determiner without losing the idiomatic meaning (*take a shower*).</footnote>
whereas variable complements and modifiers are considered open slots.<fnref n="9" /> This notion is tied to a common formal test to identify MWEs, in which inflexibility is seen as a proxy for semantic non-compositionality, leading to an unexpected meaning shift (Candito et al. 2021: p. 467). Notice that the term “lexicalised” has a different meaning when applied to MWEs as a whole (and not to its component lexemes) in a diachronic perspective. MWEs can be seen as the result of a process of **lexicalisation** by which the components gradually lose their independence and the whole combination gets more and more fixed to finally become an autonomous lexical unit which “has to be listed in a lexicon” (Evert 2004).

**Separating the wheat from the chaff** From time to time, I have been contacted by colleagues looking for help to deal with multiword units in their NLP applications. On a closer look, it turned out that these multiword units were actually not MWEs, but regular recurrent phrases. Indeed, being composed of multiple lexical units is only one aspect of our working definition for MWEs. In addition, MWEs must exhibit some level of irregularity (i.e. idiosyncrasy) with respect to structurally similar expressions that are considered regular, compositional and productive in a given language, that is, part (b) of definition 2.3 above. Therefore, now that we have defined what MWEs are, let us briefly survey that they are *not*.

**Collocations** are often seen as synonyms for MWEs. The term “collocation” is used with different meanings according to the linguistic tradition. Here, we assume that they are word combinations presenting empirical/statistical idiosyncrasies, that is, outstanding co-occurrence or word combinations that appear together more often than expected by pure chance.<fnref n="10" /> Although there is significant overlap, collocations – or institutionalised phrases (Sag et al. 2002) – can be completely regular combinations presenting only statistical association preferences, with no other idiosyncrasy. Thus, some MWE definitions include collocations (e.g. 2.2) while others do not (2.3). It may be hard to distinguish collocations from regular recurring **phrases** because they are defined in terms of frequency (e.g. in which corpus?). We consider that statistical salience is not a sufficient criterion to characterise MWEs.

**Compounds** are lexemes resulting from the word formation process of compounding (i.e. juxtaposition of two or more lexemes, sometimes with minor mor-

However, *take the cake* cannot mean 'to win' in *take a cake*, so *the* is lexicalised here. In functional expressions, the replacement of a lexicalised component by a related word can lead to ungrammaticality, e.g. *it appeared at once* → *it appeared at twice*

<footnote n="9">⁹This term stems from the parsing literature; a lexicalised parser uses the words themselves, not only their categories, to make decisions (e.g. in rules).</footnote>

<footnote n="10">¹⁰See Evert (2009) for a detailed discussion.</footnote>
phological and/or orthographic adaptations). Depending on the language, compounding does not necessarily constitute an idiosyncratic behaviour, so that not all compounds are MWEs.<fnref n="11" />

Idioms are a particular category of MWE with idiosyncratic semantics (see §2.1.4). **Metaphors**, on the other hand, can evolve into idioms, but are usually much more flexible (e.g. the *heart* is associated with emotions, so *break one's heart* can be paraphrased as *tear/rip/destroy one's heart/feelings/love*).<fnref n="12" /> In metaphors, it is usually difficult to identify at least two lexicalised components that cannot be paraphrased, so there can be no MWE (Cruse 1986). Moreover, metaphors and figurative language are not necessarily multiword. In phraseology, **phrasemes** are formulaic multiword sequences that can be MWEs, but are not necessarily cohesive syntactic units.

Constructions are defined as form-meaning pairings in which the form is usually a syntactic pattern not always fully lexicalised, that is, containing open slots (Fillmore et al. 1988). The links between MWEs and constructions have not been thoroughly studied yet, but it is probable that the notion of construction subsumes that of MWE.

Finally, MWEs may be defined as combinations that “correspond to some conventional way of saying things” (Manning & Schütze 1999). However, *conventionality* is also at play in **named entities** and domain-specific **terms**. Named entities are expressions referring to particular entities in the world such as the names of persons (e.g. *Lady Gaga*), places (e.g. *Clermont-Ferrand*), and organisations (e.g. *Extinction Rebellion*). Terms denote specific concepts in a technical or scientific domain (e.g. *recurrent neural network*, *polymerase chain reaction*). Both can be composed of single or multiple words. Although it is possible to see multiword terms and multiword named entities as MWEs, this is not extremely convenient. First, the conventional nature of terms and named entities can usually be verified institutionally (e.g. official websites, standardisation organisms, Wikipedia pages), which is not the case for MWEs. Second, this introduces a different treatment for single-word and multiword items (e.g. *the Eternal City* would be annotated as an MWE, but not *Rome*). Third, given the complex nature of MWEs, it seems reasonable to delegate the treatment of terms and named entities to other research communities. A more in-depth discussion of MWEs vs. named entities can be found in Candito et al. (2021). In this manuscript, we assume that conventionalisation is not a sufficient criterion for MWEs.

<footnote n="11">¹¹In Universal Dependencies, *compound* is misleadingly listed under MWE relations.</footnote>

<footnote n="12">¹²Some idioms are “frozen” metaphors, although their etymology is not readily available, e.g. *crocodile tears* (https://en.wikipedia.org/wiki/Crocodile_tears).</footnote>
2.1.3 Getting notation out of the way

Working with MWEs in different languages led to the development of a set of notational conventions for MWE examples. Such conventions became necessary to homogenise examples across languages throughout the PARSEME guidelines, whose version 1.2 includes examples in 28 languages.<fnref n="13" /> Together with my colleagues, I have worked towards adapting these guidelines to scientific articles containing MWE examples (Markantonatou et al. 2018: p. vii–viii). The latest effort to improve this set of notational conventions was carried out within the group of editors of the Phraseology and Multiword Expressions series of Language Science Press (Markantonatou et al. 2021). In the long run, we hope that such efforts will lead to the development and adoption of notation standards in the research community, which could favour the use of multilingually more diverse examples without losing in readability.

To make this manuscript as self-contained as possible, I summarise the main principles of these conventions below. MWE examples are composed of seven parts: (a) a text fragment containing the MWE, (b) the language name or code, (c) a transliteration if the example uses a different script, (d) word-by-word glosses following the Leipzig rules,<fnref n="14" /> (e) a literal translation in single quotes preceded by *lit.*, (f) an idiomatic translation in single quotes, and (g) the source of the example, when available. The lexicalised components of the MWE are shown in boldface. The numbered example below illustrates items (a-g) except (c), where each example part is explicitly annotated:

(1) **nie zagrzać miejsca w pracy** (a)
not warm place at work. (d)
lit. 'not to warm one's place at work.' (e)
'not to stay long at work.' (f)

(pl) (b)

(PARSEME 1.2 guidelines) (g)

Obvious items may be omitted and none of them is mandatory, although I try to include at least items (a), (b), (d), and (f). Examples can also be shown inline for brevity, e.g. pl nie zagrzać miejsca w pracy (lit. 'not to warm one's place at work') 'not to stay long at work'. Many examples in this manuscript are in English, to favour the fluidity of the text, and in this case I omit the language code (b). However, I often include the idiomatic translation (f) out of consideration for non-native speakers. I try to include examples in other languages from time to

<footnote n="13">¹³https://parsemefr.lis-lab.fr/parseme-st-guidelines/1.2/?page=notation</footnote>

<footnote n="14">¹⁴https://www.eva.mpg.de/lingua/resources/glossing-rules.php</footnote>
time as well, to favour linguistic diversity. For more details on the conventions I
refer to the full description provided in Markantonatou et al. (2021).

### 2.1.4 All shapes and sizes

Despite the fact that multiword expressions defy numerous attempts to categorise them, it may be useful to group similar expressions into categories,<fnref n="15" /> both for lexical resource creation and corpus annotation. In my PhD thesis (Ramisch 2012) and its extended version published as a book (Ramisch 2015), I summarised a set of MWE classifications proposed in the literature, covering construction grammar (Fillmore et al. 1988), meaning-text theory (Mel'čuk & Polguère 1987; Mel'čuk 2023), and NLP-oriented ones (Smadja 1993; Sag et al. 2002). I also proposed a typology based on two somehow orthogonal axes: morphosyntactic distribution of the MWE as a whole and level of “difficulty” (Ramisch 2015).

Since then, new categorisation proposals emerged, like the one by Escartín et al. (2018) for Spanish. Their work includes an interesting comparative overview of the proposals mentioned above, including the one in Ramisch (2012). The authors propose not only categories but also criteria to classify MWEs. They conclude that “MWE typologies should be adapted to the language under research, and classic typologies mainly based on the English language do not seem adequate to describe and classify MWEs in other languages”. For instance, verb-particle constructions (e.g. *take off*, in the aircraft *took off*) are often considered a major MWE category, although irrelevant for many language families such as Slavic and Romance. Inherently reflexive verbs (e.g. *fr se suicider* (lit. ‘self suicide’) ‘to suicide’) are much more common in these language families but are rarely included in English-centric classifications.

A unique (or flexible) cross-lingually valid and operational MWE categorisation, covering all phenomena that match definition 2.3, remains an open and ambitious research question. Nonetheless, an important step in this direction is the PARSEME typology for verbal MWEs, which covers a large number of languages from different families (Savary et al. 2018). Its generalisation to other morphosyntactic categories as an extension of the PARSEME guidelines is part of my future work perspectives (§5.3.1).

In this manuscript, I will limit myself to a tentative new taxonomy proposal, comprising a brief overview and a few examples of some common categories that are recurrent in my work. This categorisation is based on the external syntactic

<footnote n="15">¹⁵Given that the term “type” is ambiguous (also used to refer to the token vs. type distinction), I systematically employ the term “category” when referring to a conventionally named group of MWEs sharing some characteristics.</footnote>
distribution of MWEs (i.e. their role/function in the sentence), which is an im-
perfect but intuitive criterion to group similar MWEs. The choice to ignore the
internal syntactic structure is motivated by the existence of syntactically irregu-
lar (or exocentric) MWEs, for instance, *fr n'importe quoi* (lit. 'no matter what')
'anything' is a verb phrase which acts as a pronoun (Kahane et al. 2017). Thus, it
seems tricky to take these constructions into account in a taxonomy that focuses
on the individual POS and dependencies within MWEs.

Nonetheless, a (morpho-)syntactic characterisation of MWEs based on their
external syntactic distribution is not flawless, since these linguistic objects share
properties with both single words and phrases. In particular, should coarse MWE
categories be based on the parts of speech of single words (NOUN, VERB, etc.), or
rather on phrasal structure tags (NP, VP, PP, etc.)? Adopting the same categories
as single words is tempting, since there would be no need to create a new tagset.
It would also correspond to the intuition that MWEs are “words with spaces” and
should be considered as single words. However, there are some downsides to this
approach. We would end up with quite large MWE categories in which there is
no component with the same POS as the whole (e.g. *at stake* is composed by
a preposition and a noun, but acts as an adjective). Moreover, some expressions
can be quite complex, such as fully saturated verbal idioms like *pt quem vê cara
não vê coração* (lit. ‘who sees face doesn’t see heart’) ‘one can lie/omit their true
feelings’. It might sound artificial to call these “verbs” instead of phrases or sen-
tences. Finally, the criteria to distinguish some categories might not be clear cut
(e.g. adverbs vs. adjectives for some preposition+noun expressions). This would
require either having multiple POS for the same MWE, with contextual disam-
biguation rules, or arbitrary categorisations (classify all such phrases as adverbs,
regardless of their distribution).

The solution proposed here relies on phrasal structure rather than on single-
word POS tags, taking advantage of the significant progress made in syntactic
annotation in Universal Dependencies (de Marneffe et al. 2021). One clear ad-
vantage of adopting UD's view is that it has been put to a test for treebank-
ing in many languages, increasing the potential cross-lingual plausability of the
proposed MWE taxonomy. In UD, linguistic units are classified as **nominals** re-
ferring to entities (usually nouns), **clauses** referring to events or states (usually
verbs), and **modifiers** used to specify the attributes of nominals, clauses or other
modifiers (traditionally adjectives and adverbs). In addition, a set of **functional**
items such as determiners and auxiliary verbs are not independent, but act as
specifiers of the meaning or syntactic role of the three main categories. The ty-
pology proposed here and presented in Figure 2.1 extends these four notions to
multiword expressions. Notice that collocations or institutionalised phrases in
the sense of Sag et al. (2002) are considered as out of scope here, so we focus on
lexicalised phrases only.

<figure><img src="image_6.png" /><figcaption>Figure 2.1: Taxonomy proposed to categorise MWEs along with examples. With a few exceptions, examples were extracted or adapted from the current manuscript itself.</figcaption></figure>

**Clausal MWEs** This broad category is equivalent to PARSEME's verbal MWEs and encompasses six finer categories mostly determined by the nature of the complement taken by the head verb: **multi-verb construction** if it is another verb, **inherently reflexive verb** if it is a reflexive clitic, **inherently clitic verb** if it is another (non-reflexive) clitic, **verb-particle construction (VPC)** if it is a particle homonymous to a preposition or adverb.16 All other types of idiosyncratic clausal MWEs should, in theory, belong to the **verbal idiom (VID)** category.17 However, a special category takes precedence over VID: **light-verb constructions (LVCs)** are

<footnote n="16">¹⁶The term *verb-particle* seems quite biased towards Germanic languages. A more generic term like *verb-satellite* may be more inclusive of non-European languages such as VV constructions in Japanese and Chinese.

¹⁷Version 1.0 of the guidelines included a category for *sentential MWEs* corresponding to sayings (e.g. *the early bird catches the worm* 'early action increases the chances of success'), but these are now seen as saturated idioms (VID).</footnote>
formed by predicative nouns (denoting events or states) supported by a light verb that mainly modifies the event/state via its morphological features. Examples of each category, mostly taken or adapted from the current manuscript, are shown in purple boxes in Figure 2.1.<fnref n="18" /> This is the most developed part of the taxonomy, not necessarily because clausal MWEs are more numerous or diverse, but simply because it benefits from the experience of PARSEME. For instance, multi-verb constructions and inherently reflexive verbs were added after confronting the guidelines to languages in which they are frequent (as opposed to English). The main differences between clausal MWEs proposed here and PARSME’s verbal MWEs are:

* We arbitrarily exclude the experimental category *inherently adpositional verb* (e.g. *rely on*). Although it fits part (b) of our MWE definition, by adopting UD as underlying syntactic formalism, it becomes impossible to annotate selected prepositions governing non-lexicalised complements because the result would not form a connected subtree, as required by part (a) of definition 2.3.

* We assume that the language-specific category *inherently clitic verb* can be generalised to other languages.

* VPCs and LVCs further split into sub-categories not detailed here: VPC.full, VPC.semi, LVC.full and LVC.cause. Notice that PARSEME excludes from the scope of annotation regular aspectual LVCs (e.g. *to start a presentation*) and annotates idiosyncratic ones as VIDs (e.g. *to fall in love* 'to start loving'). However, this arbitrary decision could be questioned in the future (Fotopoulou et al. 2021).

In accordance with the PARSEME and PARSEME-FR guidelines, adpositions and complementisers (e.g. *that*) selected by non-lexicalised complements (e.g. *the show makes fun of celebrities*) are not part of the MWE, even if they are always present, because adding them would violate the connected dependency subtree constraint, implicitly required by definition 2.3.

**Nominal MWEs** Nominal idioms correspond to any combination presenting some idiosyncrasy in definition 2.3 and functioning as a nominal in the sentence, in the sense of UD (de Marneffe et al. 2021). In line with previous work (Cordeiro et al. 2019), we propose not to distinguish nominal idioms by the type

<footnote n="18">¹⁸Details in the guidelines: http://parsemefr.lis-lab.fr/parseme-st-guidelines/</footnote>
of complement they take, including bare nouns (e.g. **science fiction**, **dataset**), genitive nouns (e.g. **rat's nest**), prepositional phrases (e.g. **bed of roses**, **pain in the neck**), adjectives (e.g. **big deal**, **hotline**), clausal phrases (e.g. **hard nut to crack**), conjunctive terms (e.g. **bed and breakfast**). We consider that compounding is a word formation process orthogonal to MWEs, so nominal idioms include expressions whose elements are concatenated (**chatbot**), separated by spaces (**science fiction**) or hyphens (**science-fiction movie**). A second (minor) category of nominal MWEs contains nominal pro-forms, that is, **multiword nouns**. Since most of the time multiword pronouns contain no content word, it is hard to apply idiomaticity tests to them, so they are probably better defined as closed lists of multiword items with the distribution of nouns. Nominal MWEs (but also any of the other categories) can be exogenous, that is, their syntactic head does not need to be POS-tagged as a noun (e.g. **merry-go-round**). We initially propose that MWEs functioning as nominals but derived from clausal MWEs (referring to events or states) are annotated as clausal MWEs (e.g. **the progress made**), in line with the PARSEME guidelines for verbal MWEs and differently from UD. This also applies to nominals acting as modifiers, importantly covering prepositional phrases such as **from time to time** and **by the way**. In UD, prepositional phrases are considered nominals independently of their role, even when they act as modifiers. It seems more convenient for MWEs to take a more semantic-oriented position and assume that the linguistic tests characterising modifier MWEs will be more appropriate for nominals behaving so. Finally, we exclude multiword terms and named entities for the sake of simplicity (in accordance with the discussion in §2.1.2), although this could be questioned in the future, once these categories are described more finely in annotation guidelines.

**Modifier MWEs** This coarse category includes the multiword version of the two traditional modifier classes: adjectives, which modify nominals, and adverbs, which modify clauses and other modifiers. Like for single words, though, the distinction between multiword adjectives and adverbials may be tricky, and some constructions may behave as both, depending on the context. Beyond obvious adjectives such as **old school** and **full-fledged**, some MWEs, especially idiosyncratic prepositional phrases, may modify both nominals and clauses (e.g. **get notation out of the way** vs. **with notation out of the way**). One possible solution for this issue is to classify as multiword adjectives only those MWEs that cannot modify anything but nominals, and as multiword adverbials all MWEs which can modify clausals or nominals. However, this is not the whole story, as modifier MWEs also stand somehow in between content MWEs and functional
MWEs. Thus, many prepositional phrases composed by a content word and a single fixed preposition (e.g. *In addition, a set of [...]*) can take complements and be seen as multiword prepositions (e.g. *in addition to pragmatics*, [...]). We propose to treat all these cases as multiword adverbials, and categorise as multiword adpositions only those MWEs that cannot occur without complements (e.g. *proportion of MWEs with respect to single words* but not *proportion of MWES with respect*). Adverbial MWEs are not limited to prepositional phrases, they can be multiword nominals (e.g. *day after day*), coordinated adjectives (e.g. *safe and sound*), coordinated adverbs (e.g. *back and forth*), coordinated heterogeneous items (e.g. *time and again, by and large*), etc. As long as they function as modifiers in the sentence, they will be categorised as such. Designing linguistic tests to capture the idiosyncrasies of modifier MWEs might be tricky because, differently from clausal and nominal MWEs, they tend to contain only one content word (this is also the case for functional MWEs, as described below).

**Functional MWEs** This category covers multiword adpositions (e.g. prepositions in English), determiners and conjunctions. Although apparently simple to circumscribe, this coarse category has its share of challenges. One criterion often used to characterise these MWEs is syntactic irregularity, but regularity is far from being binary: regular sub-systems exist inside irregular classes (Kahane et al. 2017). Functional MWEs are also sometimes considered as completely frozen or flat structures, but non-functional MWEs may also exhibit these properties. We propose using syntactic distribution to classify these closed-class MWEs into multiword determiners, adpositions and conjunctions. Multiword adpositions are usually prepositional phrases that cannot occur without a complement, as discussed above (e.g. *with respect to*). Determiners include idiosyncratic quantifiers (e.g. *a few examples*) but they can also be ambiguous with multiword adverbials (e.g. *a lot of examples* vs. *we eat a lot*) and in this case the latter should be preferred. To date, it is unclear whether numerals composed by several words should be included in the category of multiword determiners, as standard idiosyncrasy tests hardly apply for them. Multiword conjunctions are a particular exception to the connected subtree rule, since they usually contain no content word and would depend on non-lexicalised complements. The trick here is to make them into a connected tree using UD's flat relation. Moreover, some complex conjunctions may introduce their complements using prepositions (*as well as*) or complementisers (*now that*), which are also exceptional in that they are considered as part of the MWE. One of the reasons why complex conjunctions are so hard to delimit is that traditional MWE tests are usually designed for MWEs
containing at least one content words, and most multiword conjunctions con-
tain none or, at most, an adverb. Finally, the category of multiword interjections
might become necessary to model MWEs in speech and dialogue, but is omitted
for now.

2.1.5 A hard nut to crack

After this convoluted attempt to characterise MWEs according to their syntactic
distribution, trying to persuade the reader that MWEs are hard to model and
process would be to preach to the converted. Therefore, I will focus here on three
quite ubiquitous properties of most MWE categories discussed above, which are
at the same difficult to cope with and interesting to exploit (Constant et al. 2017).

**Ambiguity** Some properties of MWEs make their identification particularly hard for NLP. The first challenge is ambiguity, whereby a given combination of lexemes can be an MWE or a regular combination depending on the context. For example, *a piece of cake* is something very easy in *the exam was a piece of cake*, but is not an MWE in *I ate a piece of cake and left*. This ambiguity, due to literal interpretations of an expression as above, is analogous to polysemy for single words. Furthermore, MWEs are also ambiguous because of coincidental co-occurrence, like in *I recognize him by the way he walks*, where *by the way* is not a synonym of *incidentally*.<fnref n="19" /> MWE ambiguity has been widely studied, in particular in cognitive studies interested in how idiomatic meaning is stored and accessed in the human brain (Popiel & McRae 1988; Geeraert et al. 2018). However, in practice, it seems that the importance of the problem has been over-estimated. We have shown in Savary, Cordeiro, Lichte, et al. (2019) that, at least for verbal MWEs in five diverse languages, the ratio of literal readings with respect to idiomatic + literal occurrences is neglectable (2-4%). This supposes being able to discard coincidental co-occurrence, which is not always straightforward e.g. when text is parsed automatically. Nonetheless, a set of well designed rules should be able to eliminate most coincidental instances (Pasquer, Savary, Ramisch, et al. 2020b). For English, numerous datasets exist to support the task of distinguishing idiomatic and literal readings: the MAGPIE corpus (Haagsma et al. 2020), the VNC-tokens dataset (Cook et al. 2008), and dedicated LVC (Tu & Roth 2011) and VPC datasets (Tu & Roth 2012). Still, most of them focus on clausal MWEs and present skewed distributions, with most types strongly preferring idiomatic

<footnote n="19">¹⁹Savary, Cordeiro, Lichte, et al. (2019) formally define idiomatic, literal and coincidental occurrences.</footnote>
or literal readings across most occurrences. This is all the more surprising given that these datasets contain cherry-picked sentences for frequent idioms, and that frequency and polysemy are often correlated, at least for single words.

**Variability** One of the issues with MWEs is that, although prototypical examples are completely fixed, in practice there is significant variation, especially for some categories, namely clausal and, to a lesser extent, nominal MWEs. This is a direct consequence of semantic idiosyncrasy or, in other words, limited semantic compositionality. Limited variability constitutes an observable property which is often used as in linguistic tests for MWEness. Many variability tests have been designed to capture the morphological, lexical, syntactic, semantic and pragmatic idiomaticity of MWEs (Schneider & Smith 2015; Savary et al. 2018). On the lexical-semantic level, limited variability has also been referred to as non-substitutability (Manning & Schütze 1999). Replacement with a synonym or related lexeme is a useful test to verify (a) if a component is lexicalised and (b) if the combination presents some degree of semantic idiosyncrasy, since the result is often not acceptable/grammatical or yields an unexpected meaning shift. For example, while it is possible to replace the colour *red* by *pink* for a flower, it is not possible to say *?pink herring* without loosing the idiomatic reading of *red herring* 'misleading clue'. On the morphological and syntactic levels, limited variability often manifests through irregular syntactic behaviour (e.g. *truth be told*) with respect to syntactically similar constructions (e.g. *?truth was told*). Limited syntactic variability has also been referred to as extra-grammaticality (Fillmore et al. 1988). Variability is a double-edged sword: at the same time as it helps identify MWEs, it also makes it difficult to identify them (and distinguish them from literal and coincidental counterparts) when a given form is known (e.g. in a lexicon).

**Arbitrariness** Inside MWEs, words combine and interact in unusual ways, taking unexpected meanings or even completely losing their original meanings. Given that the lexicalised components of MWEs are arbitrary, they are hard to predict and, in particular, hard to generate using compositional mechanisms. One prototypical example is the generation of MWEs in machine-translated text. Word-for-word translation of MWEs can generate unnatural, wrong or even funny translations. For example, the French expression *coûter les yeux de la tête* would become *to cost the eyes from the head* if literally translated into English, whereas the correct and idiomatic translation would be *to cost an arm and a leg*. It sounds unreasonable to expect that an MT system would be able to generate the English translation given the French phrase without external
knowledge about MWEs in French *and* in English. Like for variability, this property is both a curse and a blessing: inability or awkwardness when generating (e.g. translating) an MWE can be used as an MWE test. While useful, this property cannot be taken as a deterministic criterion, as some regular combinations cannot be translated word-for-word simply because language structures are different. Conversely, some MWEs happen to have literal translations (e.g. *yellow fever*, *fr fièvre jaune*).

To these three challenging characteristics, we can add non-compositionality, discontinuity, and overlaps, which are not only difficult to model but challenge frequent assumptions of NLP models in which local context is compositionally combined to disambiguate items and generalise across phrases, sentences, etc. More details and examples on why MWEs are a hard nut to crack for NLP models can be found in Constant et al. (2017) and Ramisch & Villavicencio (2018).

## 2.2 Getting our hands dirty

Up to now, the current chapter covered a set of linguistic notions: words, MWE definitions, categories, and main characteristics. This section focuses on computational notions, introducing the tasks (§2.2.1) and resources (§2.2.2) usually manipulated when dealing with MWEs in NLP. It concludes with an overview of the research landscape in the field of automatic MWE processing (§2.2.3).

### 2.2.1 A task definition taken for granted

MWE definitions, categories and properties are not the only source of vagueness and disagreement in the field. One of the major terminological messes concerns the naming and definition of the task at hand. What we call “MWE processing” has been referred to as MWE *identification* (Tsvetkov & Wintner 2011), *extraction* (Tsvetkov & Wintner 2012), *acquisition* (Ramisch 2015), *dictionary induction* (Schone & Jurafsky 2001), *learning* (Korkontzelos 2011) and so on. More mature fields often have more standardised task nomenclatures and definitions. For instance, no one would refer to named entity *recognition* as named entity detection or learning, or to word sense *disambiguation* as tagging or identification.

To address this problematic situation, the 2017 survey by Constant et al. (2017) proposed a conceptual framework within which both the challenges and the different research contributions can be positioned. This framework is the main outcome of a working group of the PARSEME COST Action and has since been adopted quite widely by the community. It proposes to divide MWE processing
into two main subtasks: **MWE discovery** and **MWE identification**. MWE discovery is concerned with finding new MWEs (types) in text, and storing them for future use in a repository of some kind such as a lexicon. In contrast, MWE identification is the process of automatically annotating MWEs (tokens) in running text by associating them with known MWEs (types). These two tasks interact not only with each other, but also with other fundamental and applied tasks in NLP. The way and in particular the *order* in which MWE processing intervenes with respect to other tasks is referred to as **orchestration**.

<figure><img src="image_7.png" /><figcaption>Figure 2.2: MWE discovery process: rules or models are applied to raw text, generating a list of MWE candidates. The generated list is often filtered manually by human experts before being added into a lexicon (adapted from Constant et al. (2017)).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 2.3: MWE identification process: rules, models, or MWE lexicons are applied to raw text, generating a new version of the text in which the identified MWEs are tagged (adapted from Constant et al. (2017)).</figcaption></figure>

The delineation of the two tasks is fundamental because, although both processes take (raw) text as input, their results are distinct (Figure 2.2 vs. Figure 2.3).
The output of discovery is a list of MWE lexical entries, while, for identification, it is an annotated text. The list of MWE candidates usually requires some filtering by experts before being added into a lexicon which may or may not already contain MWE entries. Identification, on the other hand, generates annotations that can help getting to the meaning of the text in further processing. Both tasks also often employ different approaches and evaluation strategies. Authors of discovery methods tend to apply unsupervised techniques which are evaluated in terms of the quality of MWEs discovered. On the other hand, identification approaches are often based on supervised learning models whose results are evaluated by comparing automatically tagged text to reference annotations. As illustrated in Figure 2.3, an MWE lexicon, potentially created with the help of MWE discovery, can be helpful for MWE identification (Savary, Cordeiro & Ramisch 2019).

MWE discovery is a very popular task and has been a very active research area since the end of the 80's. Numerous methods, tools and systems have been proposed throughout the decades since the publication of influential seminal papers (Choueka 1988; Church & Hanks 1990; Dunning 1993; Smadja 1993; Justeson & Katz 1995). At the beginning of the 2000's and under the influence of the “pain-in-the-neck” paper (Sag et al. 2002), the MWE workshop series were launched, putting discovery in focus, seen as one of the main bottlenecks of NLP technology. The Achilles heel of discovery is its evaluation, as it is difficult to assess the quality of automatically extracted multiword units out of context and independently of downstream applications.

Nowadays, research on new discovery methods has lost some of its impetus, giving way to identification. The current focus on the latter is leveraged by the creation and release of annotated corpora and shared tasks focusing on this task (Schneider et al. 2016; Savary et al. 2017; Ramisch, Cordeiro, et al. 2018; Ramisch et al. 2020), as well as the development of adaptable tagging systems and pretrained language models able to annotate text with relatively limited effort (Scholivet & Ramisch 2017; Zampieri et al. 2018; Waszczuk et al. 2019; Taslimipoor et al. 2020). This active research landscape gives rise to the proposal of standard formats, datasets, annotation schemes, evaluation procedures, evaluation metrics, and benchmarks for MWE identification.

The fundamental distinction between discovery and identification structures this manuscript and enables me to organise my scientific contributions along this backbone. In particular, while my PhD thesis focused on traditional MWE discovery methods (n-grams, POS patterns, association measures), I have since turned towards a more specific tasks: compositionality prediction (Cordeiro et al. 2019). This task corresponds to assigning a compositionality label or score to a given word combination. When performed out of context (i.e. in terms of types,
not tokens), this can be seen as discovering, among a list of candidates, which ones are less compositional, thus more likely to be MWEs. This task and my contributions to it will be discussed in Chapter 3.

The division of labour between discovery and identification, although very useful, is somehow simplistic, and fails to cover the richness of research contributions in the field. For instance, MWE identification has been used to provide additional features for other NLP tasks. In machine translation, the translation quality of MWEs has been evaluated (Barreiro et al. 2013; Ramisch, Besacier, et al. 2013) and several results demonstrated that explicit modelling can help generate higher-quality MWE translations (Carpuat & Diab 2010; Bouamor et al. 2012; Stymne et al. 2013; Tan & Pal 2014; Cap et al. 2014; Zaninello & Birch 2020). Explicit MWE identification can also benefit syntactic parsing (Nivre & Nilsson 2004) or be performed jointly with it (Constant & Nivre 2016). Other contexts in which MWE identification has been assessed include information retrieval (Acosta et al. 2011), word sense disambiguation (Finlayson & Kulkarni 2011), supersense tagging (Schneider & Smith 2015; Liu et al. 2021), sentiment analysis (Hwang & Hidey 2019), complexity estimation (Gooding et al. 2020), metaphor identification (Rohanian et al. 2020), hate speech detection (Zampieri et al. 2021).

Finally, the notion of **orchestration** is central when integrating MWE processing with other tasks (Constant et al. 2017). Put simply, orchestration concerns the decisions as to *when* in a pipeline should MWE identification take place: before, during or after other task(s)? This is an open issue, although some light has been shed on it for parsing (Constant et al. 2019). Recent advances in natural language generation based on pre-trained neural encoder-decoder models enable progress in idiomatic language generation (Navigli 2020; Zhou et al. 2021), in which MWEs play a major role, probably requiring an update of the task definitions to cover generation tasks as well.

### 2.2.2 Resources worth their weight in gold

In Figure 2.2 and in Figure 2.3, dynamic processes are shown as rounded boxes, whereas rectangles are to static resources. Resources are often what connects not only MWE discovery and identification, but also both processes to other NLP tasks. Two main types of resources are involved in MWE processing: lexicons and corpora. Several axes can be used to describe them, including their structure, granularity or level of detail, number of languages, MWE categories covered, level of cross-lingual alignment, quality, size and purpose. Below, we define and exemplify some of these resources, whose development and publication is crucial
for improving the state of the art in the field. §3.2 and §4.2 provide details on resources for MWE discovery and identification to which I contributed.

**Lexicons** The simplest form of MWE lexicon is a list containing multiword entries in a given language. These are quite common too for single- and multiword terms and named entities (where they are called “gazetteers”). More sophisticated forms of MWE lexicons can include information about an entry’s category, POS, syntax, sense, definition, translations, etc. Especially relevant for MWE lexicons are the variability constraints applied to some elements of the expression (e.g. mandatory plural for *ends in make ends meet* ‘earn enough money to live on’). The representation of such constraints has been studied in several frameworks, e.g. Gross (1986) in lexicon-grammar, Mel’čuk (2023) in meaning-text theory, Grégoire (2010) using equivalence classes, Przepiórkowski et al. (2014) using a valence dictionary, Savary et al. (2020) using XMG, and so on. Section 2 of Savary et al. (2020) provides a more detailed overview of lexical encoding of MWE variability constraints. A minimal lexicon structure for MWE identification is proposed in Savary, Cordeiro & Ramisch (2019), where we also analyse the central role of lexicons for generalisation in this task.

Most of these lexicons are built manually, although automatic MWE discovery tools (whose product are MWE “pre-lexicons”) might have been used to guide the process. MWE lexicons with varying granularities and sizes exist for several languages such as Greek (Markantonatou et al. 2019), French (Gross 1986; Ramisch, Nasr, et al. 2016), Dutch (Grégoire 2010), Polish (Graliński et al. 2010; Przepiórkowski et al. 2014), Spanish dialects (Bogantes et al. 2016), etc. Bilingual MWE lexicons are rare gems (Fisas et al. 2020), but their utility for MWE-aware language generation is unquestionable. Datasets containing typewise numerical or categorical compositionality judgments for MWEs out of context can be considered as a special kind of lexicon, as detailed in Chapter 3.2. The survey by Losnegaard et al. (2016) provides a broad overview of MWE resources, with a special focus on lexicons. Finally, a large number of paper and electronic dictionaries designed for humans (e.g. non-native language learners) exist and often contain MWEs either as regular entries, as entries related to a simple headword, or in specialised MWE dictionaries (Walter 2006; Press 1997; Sinclair 1989).<fnref n="20" />

**Corpora** Another important resource, especially for MWE identification, is text annotated for MWEs. Sometimes MWEs are obtained as a by-product of syntactic

<footnote n="20">²⁰They were useful when writing this manuscript, for instance, to find the right MWE to convey a given meaning or check the exact wording of a vaguely remembered expression.</footnote>
annotation in treebanks (Rosén et al. 2016), while other projects annotate MWEs independently of other types of annotation (Schneider & Smith 2015; Savary et al. 2018; Candito et al. 2021). In turn, MWE annotation may serve as a basis for semantic annotation such as supersenses (Schneider & Smith 2015; Barque et al. 2020). The scope of MWE categories varies considerably across corpora, which influences the overall MWE distribution. In addition, while some corpora contain full-text annotation (Candito et al. 2021), others annotate selected sentences containing the target constructions only (Garcia, Salido, Sotelo, et al. 2019). Corpora are used as training and test sets for MWE identification, as detailed in §4.3.3. Parallel corpora annotated for MWEs may be useful to evaluate MT quality (Ramisch, Besacier, et al. 2013; Monti et al. 2015). In addition, datasets annotated for compositionality on a sentence/token basis can also be considered as focused corpora that allow the development of in-context compositionality prediction methods (Cook et al. 2008; Tu & Roth 2011; 2012; Haagsma et al. 2020; Garcia et al. 2021a). Details about guidelines, annotation methodology, quality checks, formats, and release of MWE-annotated corpora will be illustrated in §4.2.

### 2.2.3 A pain in the neck or a bed of roses?

I have been deeply involved in the organisation of the PARSEME shared tasks, MWE workshops, and SIGLEX-MWE Section. Let us conclude this section with a brief snapshot of the MWE research landscape at the time of writing. The main forum for publishing and discussing advances in the computational treatment of MWEs is the annual MWE workshop held in conjunction with major conferences in computational linguistics.<fnref n="21" /> The workshop, organised by the SIGLEX MWE Section,<fnref n="22" /> has its proceedings available on the ACL Anthology.<fnref n="23" /> Other workshops focus on particular aspects of MWE processing, such as the MUMTTT workshop on the translation of multiword units (Monti et al. 2017).<fnref n="24" />

The book collection *Phraseology and Multiword Expressions* publishes books on recent topics in the field.<fnref n="25" /> This book collection is one of the outcomes of the PARSEME project, a network of researchers in Europe which made significant progress in the field (Savary et al. 2015).<fnref n="26" /> It has built many useful resources such

<footnote n="21">²¹https://multiword.org</footnote>

<footnote n="22">²²The MWE Section is coordinated by a standing committee of which I was part in 2016-2018 (nominated officer) and 2020-2022 (elected Section representative).</footnote>

<footnote n="23">²³https://aclanthology.org/</footnote>

<footnote n="24">²⁴Latest edition in 2019: http://www.lexytrad.es/europhras2019/mumttt-2019-2/</footnote>

<footnote n="25">²⁵https://langsci-press.org/catalog/series/pmwe</footnote>

<footnote n="26">²⁶http://parseme.eu</footnote>
as a list of MWE-aware treebanks (Rosén et al. 2016),<fnref n="27" /> and a list of MWE lexical resources (Losnegaard et al. 2016). Additionally, the PARSEME shared task on verbal MWE identification released MWE-annotated corpora for 20+ languages.<fnref n="28" />

In addition to the PARSEME shared task, SEMEVAL features tasks related to MWEs, like noun compound classification (Hendrickx et al. 2010), noun compound interpretation (Butnariu et al. 2010) and keyphrase extraction (Kim et al. 2010). In 2016, the SEMEVAL DIMSUM shared task focused on token-based MWE identification in running text, releasing corpora with comprehensive MWE annotation for English (Schneider et al. 2016).<fnref n="29" /> Task 2 of SEMEVAL 2022 is on token-based idiomaticity prediction and representation.<fnref n="30" />

## 2.3 A big deal

Before we dig into computational resources and experiments of the next chapters, let me try and persuade you that MWE processing is an interesting and important topic for NLP. It is not only their frequency, but rather their pervasiveness (§2.3.1), which confers them a special status in NLP applications involving language generation (§2.3.2) and analysis (§2.3.3). Moreover, the inherent irregularities of MWEs (see definition 2.1) constitute a fascinating excuse to revise structural/inductive assumptions of NLP models (§2.3.4) and to question the usefulness and role of linguistic theory in neural/end-to-end models which have taken the field by storm (§2.3.5).

### 2.3.1 A whole lot of them

The frequency of MWEs in human languages is often taken for granted within the MWE community. Seen from other communities, though, this assumption meets some (justified) skepticism. Indeed, one might argue that, beyond prototypical “kick-the-bucket” examples, MWEs actually do not come in buckets in linguistic resources and models.<fnref n="31" />

<footnote n="27">²⁷https://clarino.uib.no/iness/page?page-id=MWEs_in_Parseme</footnote>

<footnote n="28">²⁸https://gitlab.com/parseme/corpora/-/wikis/home</footnote>

<footnote n="29">²⁹https://dimsum16.github.io</footnote>

<footnote n="30">³⁰https://sites.google.com/view/semeval2022task2-idiomaticity</footnote>

<footnote n="31">³¹In the 38-billion-words English Web 2020 (enTenTen20) corpus, Sketch Engine returns 4,234 occurrences of the lemma *kick* followed by *the* and *bucket* at most 3 words to its right (e.g. *kick the ADJ bucket*). Among the first 100 concordance lines, 13 are meta-linguistic: definitions, English language forums, linguistics texts, e.g. [...] *meanings of the words that make it up, i.e. cannot be translated literally.* Examples: “*under the weather*”, “*kick the bucket*”. (June 8, 2021, https://app.sketchengine.eu/).</footnote>
This suspicion might stem from some highly cited (but rarely questioned) estimates. Jackendoff (1997) speculates that the number of multiword items in a speaker's mental lexicon is roughly equivalent to the number of single words. Moreover, Sag et al. (2002) suggest that this might be an underestimate, as technical language would add more multiword than single-word terms to the vocabulary. However, these claims are based on (armchair) linguistic introspection about abstract lexicons, the size and form of which are not clearly delineated.

<table><caption>Table 2.1: Number and ratio of MWEs with respect to total entries for languages with ≥10k entries and ≥100 MWEs, data from Princeton Wordnet (Fellbaum 1998) and Open Multilingual Wordnet (Bond & Foster 2013), NLTK versions (Bird et al. 2009).</caption><thead><tr><td>Language (POS)</td><td>Wordnet</td><td>MWE #/total = Ratio</td></tr></thead><tbody><tr><td>Farsi</td><td>Open</td><td>13,313/ 30,462 = 43.70%</td></tr><tr><td>English</td><td>Princeton</td><td>64,331/155,287 = 41.42%</td></tr><tr><td>Galician</td><td>Open</td><td>9,107/ 27,139 = 33.55%</td></tr><tr><td>Greek</td><td>Open</td><td>6,816/ 24,107 = 28.27%</td></tr><tr><td>Spanish</td><td>Open</td><td>14,479/ 57,765 = 25.06%</td></tr><tr><td>Portuguese</td><td>Open</td><td>17,587/ 74,011 = 23.76%</td></tr><tr><td>Arabic</td><td>Open</td><td>8,234/ 37,336 = 22.05%</td></tr><tr><td>Catalan</td><td>Open</td><td>15,937/ 70,625 = 22.56%</td></tr><tr><td>Finnish</td><td>Open</td><td>41,076/189,228 = 21.70%</td></tr><tr><td>Polish</td><td>Open</td><td>11,328/ 52,379 = 21.62%</td></tr><tr><td>French</td><td>Open</td><td>18,481/102,671 = 18.00%</td></tr><tr><td>Slovene</td><td>Open</td><td>12,194/ 71,830 = 16.97%</td></tr><tr><td>Croatian</td><td>Open</td><td>7,650/ 47,922 = 15.96%</td></tr><tr><td>Italian</td><td>Open</td><td>8,805/ 63,134 = 13.94%</td></tr><tr><td>Basque</td><td>Open</td><td>5,840/ 48,935 = 11.93%</td></tr><tr><td>Indonesian</td><td>Open</td><td>11,728/106,689 = 10.99%</td></tr><tr><td>Malay</td><td>Open</td><td>11,019/105,029 = 10.49%</td></tr><tr><td>Thai</td><td>Open</td><td>4,404/ 93,046 = 04.73%</td></tr><tr><td>Dutch</td><td>Open</td><td>2,304/ 60,260 = 03.82%</td></tr></tbody></table>

MWE frequency figures are harder to interpret when it comes to concrete lexical resources. For instance, Ramisch, Villavicencio & Kordoni (2013) report that the English Princeton wordnet 3.0 contains 51.2% of multiword nouns (60,292 out of 117,798 nouns) and 25.5% of multiword verbs (2,829 out of 11,529 verbs). However, many nouns are named entities, for which most MWE definitions do
not fit like a glove (see §2.1). On the other hand, the Princeton wordnet does not
cover light-verb constructions, which represent a large amount of verbal MWEs
in corpora (Ramisch et al. 2020). Table 2.1 presents the overall number and ratio
of MWEs in wordnets in 19 languages, ranging from 3.8% of the 60k Dutch entries
to 43.7% of the 30k Farsi entries. This huge variability questions the representativ-
ity of MWEs in wordnets. Lexicons and lists containing only multiword entries
tend to cover heterogeneous MWE categories and vary immensely in terms of
context of creation, budget, purpose and formalism. The survey by Losnegaard
et al. (2016) describes MWE lexicons for 19 languages, the largest of which con-
tained 140,000 base forms. Although these resources do not allow us to estimate
the relative proportion of MWEs with respect to single words, their number does
not seem negligible either.

Instead of looking at lexicons and type-based proportions, it may be more in-
sightful to look at token-based statistics in annotated corpora. Probably one of
the most complete resource of this type is the French PARSEME-FR corpus.<fnref n="32" />
This corpus contains 6,579 annotated MWEs and named entities for 3,099 sen-
tences, that is, about 2 annotations per sentence and one annotation every 10
tokens in average. If we disregard named entities, there are still 3,451 annotated
MWEs, that is, about one per sentence or one every 20 tokens in average. These
proportions are quite stable across the four domains/ genres represented in the
corpus (Candito et al. 2021).

The English STREUSLE corpus presents similar proportions, with 3,718 strong and weak MWE annotations in 3,813 sentences from a web treebank, that is, about one MWE per sentence (Schneider & Smith 2015).<fnref n="33" /> It is undoubtedly surprising that the frequency of MWEs is so similar in these two corpora (STREUSLE and PARSEME-FR), given the different languages, text genres, annotation scope and guidelines adopted by the independent teams creating each resource.

Table 2.2 summarises the corpus sizes in edition 1.2 of the PARSEME shared task.<fnref n="34" /> These statistics are probably more representative of the diversity of the phenomenon (Ramisch et al. 2020). The PARSEME shared task 1.2 corpora were annotated by 14 independent but coordinated language teams for *verbal* MWEs only, following common multilingual guidelines (details in §4.2.1). The last column of Table 2.2 shows the average number of sentences for each annotated verbal MWE; table rows are sorted by ascending sentences/MWE ratios. This ratio can be seen as the average number of sentences we have to go through before we

<footnote n="32">³²http://hdl.handle.net/11234/1-3429</footnote>

<footnote n="33">³³Statistics from v4.4 (2020-11-04) at https://github.com/nert-nlp/streusle</footnote>

<footnote n="34">³⁴I am deeply involved in PARSEME as shared task co-organiser, data provider and system co-author, so it will recurrently pop up throughout this manuscript.</footnote>
## 2 MWEs in a nutshell

<table><caption>Table 2.2: Number of sentences (# sent.), tokens (# tok.) and annotated verbal MWEs in the PARSEME shared task 1.2 corpora, along with the MWE density expressed as the ratio of sentences per MWE – one MWE every X sentences on average. The table is sorted by descending MWE density (ascending sentence/MWE ratio).</caption><thead><tr><th>Language</th><th># sent.</th><th># tok.</th><th># MWEs</th><th>Sent/MWE</th></tr></thead><tbody><tr><td>Hindi</td><td>1,684</td><td>35,430</td><td>1,034</td><td>1.63</td></tr><tr><td>Swedish</td><td>4,304</td><td>65,482</td><td>1,991</td><td>2.16</td></tr><tr><td>German</td><td>8,996</td><td>173,562</td><td>4,041</td><td>2.23</td></tr><tr><td>Irish</td><td>1,700</td><td>39,216</td><td>662</td><td>2.57</td></tr><tr><td>Basque</td><td>11,158</td><td>157,807</td><td>4,246</td><td>2.63</td></tr><tr><td>Greek</td><td>21,447</td><td>579,032</td><td>7,444</td><td>2.88</td></tr><tr><td>Turkish</td><td>22,311</td><td>332,229</td><td>7,730</td><td>2.89</td></tr><tr><td>Polish</td><td>23,547</td><td>396,140</td><td>7,186</td><td>3.28</td></tr><tr><td>French</td><td>20,961</td><td>525,992</td><td>5,654</td><td>3.71</td></tr><tr><td>Italian</td><td>15,728</td><td>430,789</td><td>4,210</td><td>3.74</td></tr><tr><td>Chinese</td><td>39,929</td><td>649,576</td><td>9,164</td><td>4.36</td></tr><tr><td>Portuguese</td><td>32,117</td><td>728,550</td><td>6,437</td><td>4.99</td></tr><tr><td>Hebrew</td><td>19,200</td><td>388,481</td><td>2,533</td><td>7.58</td></tr><tr><td>Romanian</td><td>56,703</td><td>1,015,624</td><td>6,171</td><td>9.19</td></tr></tbody><tfoot><tr><td>Total</td><td>279,785</td><td>5,517,910</td><td>68,503</td><td>4.08</td></tr></tfoot></table>

come across the first MWE. For instance, there is one MWE every 4.36 sentences in Chinese, and one MWE every 7.58 sentences in Hebrew, on average.

The highest MWE density is observed for Hindi, with one MWE every 1.63 sentences, whereas Romanian has the lowest ratio, with one MWE every 9.19 sentences. There seems to be a correlation between these ratios and the size of the corpora. The four languages with highest MWE density (Hindi, Swedish, German, Irish) are those with the smallest number of annotated MWEs. At the other end, only Hebrew has less than 6,000 annotated MWEs among the four languages with the lowest MWE density (Chinese, Portuguese, Hebrew, Romanian). The Spearman correlations between the sentences/MWE ratio and the size of the corpora in terms of number of sentences and tokens are $\rho$ = 0.80 and $\rho$ = 0.83, respectively. In other words, based on these samples, we can assume that the “true” ratio across languages and domains, as annotated corpora get larger (thus more representative of the language), would be close to one verbal MWE every 3
to 5 sentences.<fnref n="35" /> These comparisons should be taken with a pinch of salt, though, since MWE ratios also depend on other factors such as the corpus register and domain, the background and training of annotators, and their interpretation of the guidelines with respect to existing resources and linguistic tradition.

Corpora annotated for MWEs, especially those to which I contributed, are discussed in more detail in §4.2. For now, these examples should be illustrative of the pervasiveness of MWEs in human languages. In short, MWE frequency varies considerably according to the annotation scope, text genre/domain, language, and the way we count (lexicon vs. corpus, that is, types vs. tokens). Though, the phenomenon seems to be frequent enough to deserve the attention of the NLP community. As a final (meta-text) argument for MWE frequency, I would like to mention this chapter itself. Throughout the text, I try to employ as many MWEs as possible (e.g. in section and subsection headings).

### 2.3.2 Flowing like a river

Learning a language as an adult is a radical experience of language's complexity and arbitrariness. I remember French classes, when I tried to express something slightly elaborate, and was often rewarded with a disappointing "ça ne se dit pas".<fnref n="36" /> Differently from artificial (e.g. programming) languages, mastering the words (lexicon), their meanings (semantics) and the rules used to modify (morphology) and combine them (syntax) does not suffice. In addition to pragmatics, cultural and common-sense knowledge, one also has to learn how to use arbitrary word combinations that confer naturalness to speech. In other words, MWEs constitute an important part of languages, and fluently speaking involves learning how to employ MWEs in a way that mimics that of other speakers. Whenever accent does not come into play (e.g. written communication), infrequent or inadequate use of MWEs can (unconsciously) help recognise non-native speakers.

As technology evolves, we are led to interact not only with fellow humans, but also with computer devices which produce natural sounding utterances. Examples of such systems include personal assistants, chatbots, automated translation and summarisation, only to name a few. All these NLP applications share the fact that their result is provided in natural language. Just like non-native speakers, they also have to produce text and speech that contains MWEs whenever appropriate. MWEs can convey a message more efficiently and succinctly than compositional paraphrases.

<footnote n="35">³⁵For other, non-verbal MWE categories, we cannot make such cross-lingual estimations because we currently lack corpora annotated for non-verbal MWEs in most languages.</footnote>

<footnote n="36">³⁶You cannot say that.</footnote>
Much of current MWE research focuses on text *analysis*, such as syntactic and semantic parsing, where the goal is to interpret MWEs in textual input (§2.3.3). Automatic *generation* of MWE-aware language has received considerably less attention, even if being able to generate MWEs is equally important to achieve human-like language skills. Selecting MWEs instead of compositional paraphrases from time to time could greatly increase the fluency of system outputs. This can help, for instance, increase the trust and credibility that users grant to the responses given by their personal assistant or to automatically translated text.

### 2.3.3 Getting to the meaning

Utterances and sentences convey meaning, and most NLP models are expected to somehow access and represent this meaning when processing documents. Automatic analysis and representation of natural languages' meaning is the topic of the field of **computational semantics**. It is traditionally assumed that semantic processing can be decomposed into lexical semantics and compositional semantics. While the former is concerned with assigning abstract sense representations to atomic linguistic elements, the latter deals with combining individual sense representations into larger units (phrases, sentences, paragraphs, etc.).

Although not always structurally visible, these assumptions (implicitly) rely on the notion of compositionality (Frege 1892). This principle, according to which the meaning of the whole can be built from the meanings of its components, is useful to conceive both human and computational language understanding as a tractable process, able to deal with an unbounded number of utterances using limited resources (e.g. memory). MWEs are at the core of computational semantics, as interpreting them is one of the requirements of full-fledged robust semantic processing. However, the distinction between lexical and compositional semantics is challenged by MWEs: their components lose their original meaning(s) when combined idiomatically, and the whole assumes a new meaning not necessarily related to the its components (Ramisch & Villavicencio 2018).

Progress in computational semantics and MWE processing in the last 20 years has been significant. Nonetheless, the place and role of MWEs in NLP models is still unclear. An illustrative example of the interaction between MWEs and (lexical) semantics is the English STREUSLE corpus, annotated for both MWEs and supersenses in a consistent way (Schneider & Smith 2015).<fnref n="37" /> This strategy was also employed for the French SemCor corpus, building on PARSEME-FR

<footnote n="37">³⁷Supersenses approximate word meaning through a small set of coarse semantic categories (e.g., PERSON, FOOD, SUBSTANCE) based on WordNet's lexical files (Ciaramita & Johnson 2003).</footnote>
MWE annotations for supersense assignment (Barque et al. 2020). Performing lexical segmentation and sense tagging simultaneously looks like a promising approach given the results obtained by computational models predicting them jointly (Schneider et al. 2016; Liu et al. 2021). Given the complexity of the phenomenon, deeper understanding and satisfactory technological solutions for automatic MWE processing can potentially contribute to tackle the long-standing mystery of meaning representation itself.

### 2.3.4 There is beauty in chaos

We are living a time of great enthusiasm, with NLP systems reaching near-human performance in many complex tasks such as natural language inference and understanding (Rajpurkar et al. 2016; Devlin et al. 2019). As a consequence, language technology is becoming more and more present in our every day lives, in our telephones, web services, personal assistants, etc. At the same time, awareness about diversity is rising in almost every domain of human societies, from global ecosystems to individual households. Languages are an important aspect of interaction, playing a role in power systems based on discrimination by gender, race, age, sexual orientation, disabilities, etc. Given the role of language technology in our daily lives, computational linguistics is directly concerned by these questions.

In accordance with major social movements of our time (e.g. #MeToo, Black Lives Matter, LGBTQIA+ rights, Fridays for Future), time has come for NLP to aim beyond solutions that work for most frequent items and dominant languages.<fnref n="38" /> Addressing a phenomenon such as MWEs, individually diverse but collectively representative of a language's culture and real use, can be seen as a sign of the field's social commitment.

MWEs are also often closely related to a linguistic community's history. Thus, they are a fascinating door into a the richness and diversity of the culture in which a language evolves. MWEs are often used creatively in irony, plays on words, jokes, taglines, ads, songs, poetry, literature, thus offsetting a language's complexity with beauty and fun. Literal translations of multiword idioms can be used in games to raise awareness about language diversity (Krstev & Savary 2018).<fnref n="39" /> Moreover, MWEs being related to a culture, taking them into account (for less resourced languages) is also a sign of acknowledgement of the history of language and its context, could potentially contribute to decolonising NLP (Bird 2020). In short, MWE research can favour diversity and inclusion in NLP

<footnote n="38">³⁸For instance, ACL 2021’s special theme was “NLP for social good” while in 2022 it focuses on “language diversity: from low-resource to endangered languages”.</footnote>

<footnote n="39">³⁹See also https://gitlab.com/ceramisch/eacl21diversity/-/wikis/</footnote>
research, both in terms of (i) diverse phenomena covered in a language, and (ii) accounting for the cultural heritage and richness of diverse languages.<fnref n="40" />

Multiword expressions are rebels: they can be seen as exceptions that occur when words get together. Their idosyncratic behaviour is hard to account for and challenges the traditional lexicon-grammar distinction (Sag et al. 2002). For instance, to date, there is no widely adopted technical solution to model their variability constraints in lexical resources (Lichte et al. 2019). It is impossible to predict when, how and why new expressions will appear and get adopted. They are the arbitrary results of uncontrollable linguistic trajectories tied to a language's transformation over time.

### 2.3.5 What if Jelinek was right?

Carrying out research on MWEs might seem old school in a time in which linguistics has become out of fashion in NLP. When I joined the field, it was permeated by a dichotomy between linguistic theory vs. corpus-based approaches. Other views of this binary distinction over time include rule-based vs. statistical models, symbolic vs. continuous representations, and expert vs. machine learning methods. It seems clear now that corpus-based, statistical, continuous and machine learning methods have became the mainstream approach for developing new NLP systems. In other words, we have been observing evidence of Fred Jelinek's famous quote "whenever I fire a linguist, our system performance improves" (Jelinek 2005). In modern NLP approaches, there seems to be no use for complex linguistic theories and models developed over the years.

However, I believe that MWEs are beyond this Manichean duality, somehow escaping the artificial dichotomy between "pure" linguistics and "hardcore" engineering. Given their fuzzy and untamed nature, they represent the perfect excuse for a researcher to work in several tasks and sub-fields of NLP, from syntactic and semantic parsing to machine translation and information extraction. On all these fronts, MWE processing is far from being a solved problem, providing numerous opportunities to propose original research contributions that blur the lines between linguistics and machine learning.

Moreover, I believe that the role of linguistics in NLP has shifted from tasks and models to data and evaluation. Before creating a system, one needs to carefully prepare (annotated) data for which linguistic expertise is often necessary. Once a system produces predictions, linguistic-based evaluation methods can help in error analysis, system inspection (e.g. probing) and thus provide some feedback

<footnote n="40">⁴⁰See the related UniDive COST Action: https://www.cost.eu/actions/CA21167/</footnote>
on the data and architecture of the system for the next iteration. Many of my con-
tributions involve data creation and curation (e.g. §4.2) and linguistic-oriented
evaluation protocols (e.g. §4.3.2).

Finally, when working with language, and in particular in NLP, one will nec-
essarily come across MWEs at some point. However, these linguistic odd birds
are seen as marginal or too difficult to deal with. Over the years, I have observed
that MWEs are often relegated to the “future work” section of research papers.
I hope that the work carried out in the MWE community, including my own,
contributes to breaking this curse and making computational linguistics a more
diverse and interesting place.

## 2.4 In short

Multiword expressions are made up of *lexemes*, that is, minimal semantic units
that compose a language’s lexicon, not to be confused with *tokens* resulting from
a computational text segmentation process. Although a 1-to-1 correspondence
would be ideal, borderline cases are hard and numerous.

*Multiword expressions* are combinations that (a) include two or more lexicalised lexemes, (b) form a connected dependency subtree and, (c) present some degree of lexical, morphological, syntactic or semantic idiosyncrasy. Lexemes are considered as *lexicalised components* of MWEs if their absence prevents idiomatic reading. It is not always easy to distinguish MWEs from related phenomena such as metaphors, collocations, phraseology, compounds, terms, etc. MWE typologies and categorisations abound. Although imperfect by definition, they are often useful to design linguistic tests in guidelines. I propose a new experimental taxonomy based on Universal Dependencies’ main concepts: clauses, nominals, modifiers, and function words. Among the properties that make MWEs difficult to represent and process in NLP, we emphasise their *ambiguity* (including literal and coincidental occurrences), their irregular *variability*, and their *arbitrariness*.

Computational processes dealing with MWEs can be roughly divided into
MWE *discovery*, which extracts MWEs from text and generates candidate lists for
inclusion in a lexicon, and MWE *identification*, which annotates text for MWEs in
context. Other tasks such as paraphrasing and translating MWEs automatically
are also explored in the literature. Resources involved in this task are *lexicons* or
various shapes and sizes and *corpora* annotated for MWEs.

I sum up the motivations for doing research in MWEs as follows:

1. MWEs are frequent, both token- and type-wise, in all languages, registers, and domains;
## 2 MWEs in a nutshell

2. Language generation systems must be able to produce them for natural and fluent output;

3. Language analysis systems involving sort of semantic processing must be able to interpret their (non-compositional) meanings;

4. Their arbitrariness and unpredictability places them among the most challenging (and beautiful) aspects of human languages;

5. Their intersectional nature is a perfect excuse to survey (and bring together) diverse approaches, fields and, points of view in NLP.

Although much progress has been made in the recent years, MWE processing is still not part of main NLP suites and pipelines. Although they are probably not a pain in the neck anymore, at least for “major” languages, much remains to be discovered and developed before dealing with MWEs becomes a bed of roses.

## 2.5 For the record

Most of the material reused in this chapter comes from the survey on MWEs published in Computational Linguistics (Constant et al. 2017) and from the book chapter on MWEs of the handbook on computational linguistics (Ramisch & Villavicencio 2018). Some material was adapted from the slides prepared for the ESSLLI 2018 course “Multiword Expressions in a Nutshell” with Agata Savary and Aline Villavicencio.<fnref n="41" /> Working within the PARSEME community shaped my view of MWE research, and this is indirectly reflected in this chapter. The “pig slaughter” example in Chapter 1 was explained to me by Veronika Vincze and Katalin Simkó at the 2nd PARSEME training school.<fnref n="42" /> Another experience that widened my view on NLP and helped me contextualise parts of this chapter was the EACL 2021 panel and games on language diversity.<fnref n="43" />

<footnote n="41">⁴¹https://gitlab.com/parseme/mwesinanutshell</footnote>

<footnote n="42">⁴²https://typo.uni-konstanz.de/parseme/index.php/2-general/148-2nd-training-school-la-rochelle</footnote>

<footnote n="43">⁴³https://gitlab.com/ceramisch/eacl21diversity/wikis/</footnote>
## 3 Fifty shades of compositionality

> *Quem sabe se o mundo não seria um pouco mais decente se soubéssemos
> como reunir umas quantas palavras que andam por aí soltas.<fnref n="1" />*

> — José Saramago, *Ensaio sobre a lucidez*

The creation of lexicographic resources such as electronic dictionaries often relies on corpora, necessary to study the use and distribution of the target lexical units. This also holds for multiword units, and the development of computational techniques and tools to support lexicographic work motivated the first proposals of automatic MWE discovery methods (Choueka 1988; Church & Hanks 1990; Smadja 1993). These seminal works influenced other areas of computational linguistics. For instance, pointwise mutual information, initially proposed by Church & Hanks (1990) as a collocation discovery metric, plays a major role in count-based word embedding models (Levy et al. 2015). Moreover, MWE discovery methods have been used as an aid to terminology for extending specialised lexical resources, for various languages and domains (Justeson & Katz 1995; Cárdenas & Ramisch 2019).

The **mwetoolkit** is a software implementing many of such MWE discovery techniques. It was initially developed during my PhD thesis (Ramisch et al. 2010; Araujo et al. 2011; Ramisch 2012) and further improved throughout the years (Cordeiro et al. 2015; Cordeiro, Ramisch & Villavicencio 2016a; Ramisch 2020). Here, we will overview some of the techniques implemented in the **mwetoolkit**, as well as other techniques described in the literature. More detailed surveys on MWE discovery can be found in Section 2 of Constant et al. (2017) and in Section 3 of Ramisch & Villavicencio (2018).

However, most of the current chapter will be dedicated to a specific task related to MWE discovery: compositionality prediction. Put simply, given a combination of two or more words, a **compositionality prediction** model must decide whether or not (or to what extent) the meaning of the whole phrase can be transparently inferred from its components and structure. This can be seen as a reframing of the

<footnote n="1">¹Perhaps the world would be a little more decent if we only knew how to gather some words that are out there somewhere.</footnote>
discovery task: given a certain method that generates MWE candidates, we must predict whether their compositionality is sufficiently low to be worth including them in a lexicon or giving them some special treatment. Thus, combinations are semantically more opaque will be “discovered” among a list of candidates with various degrees of compositionality.

After a brief overview of techniques used for MWE discovery in general (§3.1), the reminder of this chapter will focus on compositionality. I will present a new survey of datasets containing compositionality annotations, and focus on the ones developed in my own work (§3.2). Then I will present some models and evaluation results on compositionality prediction, showing that high-quality word embeddings play an important role in this task (§3.3).

## 3.1 A word on discovery

As introduced in §2.2, the task of MWE discovery consists in finding new multiword lexical units. Existing methods for this task exploit several information sources and clues from corpora. MWE candidates can be extracted from corpora by applying simple morphosyntactic patterns to automatically parsed text. For instance, Justeson & Katz (1995) defined variations of noun sequences that include other nouns (N), adjectives (A) and prepositions (P), such as *linear regression* (AN), *Gaussian random variable* (AAN) and *degrees of freedom* (NPN). Syntactic patterns enable the discovery of more flexible categories, for example, to capture verbal MWEs whose components appear in non-canonical order when passivised (Seretan 2011). Syntactically flexible MWE categories allow non-contiguous components, so their patterns may specify maximum gap size, the type of constituent allowed inside the gap, and delimiters for their boundaries. The **mwetoolkit** implements morphosyntactic patterns based on lemmas, POS tags and dependency relations using a multi-level extension of regular expressions (Ramisch 2015).

Prominent co-occurrence counts are often used as a basis for MWE discovery, often combined with morphosyntactic patterns to avoid retrieving uninteresting combinations of frequent function words, and to target specific MWE categories. While frequency is useful to find recurrent patterns, it may not be able to distinguish MWEs from n-grams that have high frequencies because they contain frequent words co-occurring by chance. An alternative is the use of statistical association scores that estimate the strength of the relation between observed and expected co-occurrence counts. Association scores take into account the possibility of words co-occurring by chance: if components are very frequent, their
frequent co-occurrence is expected, and they will be less strongly associated. On
the other hand, if they are rare, their co-occurrence is more significant.

A popular association score is pointwise mutual information (Church & Hanks 1990), which expresses the log-ratio between observed counts $c(w_1 ... w_n)$ and expected counts $E(w_1 ... w_n)$.<fnref n="2" /> Values close to zero indicate independence and the candidate words are discarded, whereas large values indicate probable MWEs. Scores based on hypothesis testing assume as null hypothesis that, if words are independent, their observed and expected counts should be identical, that is $H_0 : c(w_1 ... w_n) = E(w_1 ... w_n)$. Using a test statistic like Student's t, large values are strong evidence to reject the null hypothesis, confirming that the candidate is indeed a MWE. Several tools to calculate association scores exist (Pedersen et al. 2011), including the **mwetoolkit** (Ramisch 2015).

The adaptation of 2-word association scores to arbitrary *n*-word candidates is
not straightforward. The LocalMaxs method finds optimal MWE boundaries by
recursively including left and right context words, stopping when the association
decreases (da Silva et al. 1999). A similar approach, using a lexical tightness mea-
sure, was proposed to segment Chinese MWEs (Xu et al. 2010). Evert (2004) dis-
cusses more than 30 association scores, while the work of Pecina (2008) includes
87 association scores in total. Several studies performed comparative evaluations
of different association scores. Most of them confirm that there is no silver bul-
let, that is, the best score depends on the target language, corpus register, target
MWE category, among other factors (Evert 2004; Pecina 2008; Hoang et al. 2009;
Ramisch et al. 2012; Garcia, Salido & Alonso-Ramos 2019)

Another family of discovery methods is based on **substitutability**, mimicking
the lexical replacement test used in MWE annotation (§4.2.1.1). Substitutability
methods rely on substitution, paraphrasing and insertion (including permutation,
syntactic alternations, etc.) of one or more components of the MWE. If variants
generated automatically from the candidate are attested in a large corpus (or in
the web), then the candidate is unlikely to be an MWE, and vice-versa. Limited
semantic substitutability was exploited by Pearce (2001), who used synonyms
from WordNet to generate possible combinations from a seed candidate. Villavi-
cencio et al. (2007) and Ramisch et al. (2008) use a similar technique, but focus on
limited syntactic variability, generating variants by reordering the components
of the candidate. Riedl & Biemann (2015) designed a measure that takes into ac-
count the substitutability of an MWE by single words, assuming that MWEs tend

<footnote n="2">²While observed counts $c(w_1 ... w_n)$ are obtained directly from the corpus, expected counts $E(w_1 ... w_n)$ are usually estimated by assuming statistical independence among the components: $E(w_1 ... w_n) = \frac{c(w_1) ... c(w_n)}{N!}$, where *N* is the total number of tokens in the corpus.</footnote>
### 3 Fifty shades of compositionality

to represent more succinct concepts. While quite precise, these methods are hard to generalise, as they model specific limitations that depend on the language and MWE category, as well as on external paraphrase or synonym resources.

Translation and multilingual resources can also be useful for the task. On the one hand, some MWE categories (e.g. light-verb constructions) can share similar structures across (similar) languages, favouring cross-lingual transfer (Zarrieß & Kuhn 2009). On the other hand, translation asymmetries are also common when idiomaticity is involved. For instance, if a given sequence of two or more words in a source language is aligned to a single word in the target language, this is a good indication of a possible MWE (Caseli et al. 2010). Such asymmetries can be mined from parallel corpora, but also from resources like the Wikipedia page titles (Attia et al. 2010) or translation links in the Wiktionary (Salehi et al. 2014). Bilingual alignments can also be used to filter *out* candidates that are unlikely to be MWEs, considering the remaining ones as true MWEs (Tsvetkov & Wintner 2010). Features from parallel corpora can be combined with those extracted from monolingual corpora in supervised settings (Cap et al. 2013).

Finally, discovery methods may benefit from distributional information (Fazly et al. 2009) and in particular from word embeddings (Katz & Giesbrecht 2006; Reddy et al. 2011; Cordeiro et al. 2019). These methods are usually tailored for modelling compositionality and will thus be discussed in 3.3.

## 3.2 Resources

The principle of **compositionality** assumes that the meaning of phrases, expressions or sentences can be determined by the meanings of their parts and by the rules used to combine them.<fnref n="3" /> In other words, the “meaning of a typical sentence in a natural language is complex in that it results from the combination of meanings which are in some sense simpler” (Cruse 1986: p. 24). As a consequence, we are able to assign interpretations even to new sentences, involving unseen combinations of familiar parts (Goldberg 2015).

Compositionality has gained increasing attention in NLP research the last decade, since vector representations became mainstream in computational linguistics (Mikolov et al. 2013). Indeed, current language models often rely on composing vector representations via (learned) mathematical functions. Thus, a significant part of their performance depends on the extent to which these functions approximate the principle of compositionality inherent to human language processing (Yu & Ettinger 2020).

<footnote n="3">³The principle of compositionality is often attributed to Frege (1892).</footnote>
While compositionality in general is a wide subject, it plays a fundamental role in processing semantically idiomatic MWEs. Deviations from regular semantic composition are not only part of our MWE definition 2.3, but are also one of their most prototypical properties. Modern techniques such as large pretrained language models seem to have a hard time modelling non-compositional or partly compositional expressions (Shwartz 2019; Madabushi et al. 2021). Thus, it is crucial to better understand compositionality, especially at its limits, to develop better language models capable of taking (idiomatic) MWEs into account. In this section, I will survey existing compositionality datasets (§3.2.1), discuss their main characteristics (§3.2.2), and then focus on two particular datasets to which I contributed: one annotated on the level of types (§3.2.3) and another on the level of tokens (§3.2.4).

### 3.2.1 Existing datasets

Compositionality prediction models estimate to what extent a given word combination is compositional or non-compositional (that is, idiomatic).<fnref n="4" /> One prerequisite to develop such models is to be able to assess (and tune) their predictions using known (gold) compositionality values. Thus, compositionality datasets have been created in several languages, covering different MWE categories, at different granularities.

An example of such dataset is presented in Table 3.1. For each entry, three scores were provided by annotators on a 0–5 scale: the contribution of the head, the contribution of the modifier, and the compositionality of the head-modifier combination. Values close to 0 indicate idiomaticity whereas values close to 5 indicate compositional interpretation.

In this section, we present the results of a survey of existing datasets containing gold compositionality judgments, provided by human annotators. Table 3.2 summarises 33 compositionality datasets, covering almost 20 years of research in the field. Here, we briefly overview these resources in chronological order. §3.2.2 discusses and compares some of their characteristics and design choices.

**Discovery evaluation: a hard nut to crack** Evaluating the discovery of new MWEs has always been seen as a difficult problem. This is because the discovered MWE candidates are either absent from lexical resources, requiring expert assessment, or they are already known, in which case they their discovery is not very interesting in a realistic scenario.

<footnote n="4">⁴We make the simplifying assumption that non-compositional = idiomatic. In practice, a non-compositional MWE is not always idiomatic: it might be a proper noun, a metaphor, etc.</footnote>
## 3 Fifty shades of compositionality

<table><caption>Table 3.1: Dataset excerpt from Ramisch, Cordeiro, Zilio, et al. (2016). Compositionality score of the head, modifier and combination for the most polemic and consensual compounds (average ± std. deviation).</caption><thead><tr><th>compound</th><th>head</th><th>mod.</th><th>both</th></tr></thead><tbody><tr><td>match nul</td><td>4.4 ±1.3</td><td>2.2 ±2.3</td><td>2.5 ±2.1</td></tr><tr><td>mort né</td><td>4.6 ±1.1</td><td>3.5 ±1.8</td><td>3.2 ±2.0</td></tr><tr><td>carte grise</td><td>4.5 ±0.9</td><td>3.2 ±2.0</td><td>3.1 ±1.9</td></tr><tr><td>second degré</td><td>1.7 ±1.9</td><td>2.4 ±2.1</td><td>1.4 ±1.9</td></tr><tr><td>grippe aviaire</td><td>4.6 ±1.4</td><td>3.8 ±1.9</td><td>3.6 ±1.9</td></tr><tr><td>eau chaude</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td></tr><tr><td>eau potable</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td></tr><tr><td>matière grasse</td><td>4.8 ±0.4</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td></tr><tr><td>poule mouillée</td><td>0.0 ±0.0</td><td>0.0 ±0.0</td><td>0.0 ±0.0</td></tr><tr><td>téléphone portable</td><td>4.9 ±0.5</td><td>4.9 ±0.3</td><td>5.0 ±0.0</td></tr><tr><td>pavio curto</td><td>1.6 ±1.8</td><td>1.1 ±1.9</td><td>1.9 ±2.3</td></tr><tr><td>sexto sentido</td><td>4.0 ±1.4</td><td>2.5 ±2.1</td><td>2.8 ±2.2</td></tr><tr><td>gelo-seco</td><td>3.2 ±1.6</td><td>3.2 ±1.8</td><td>3.0 ±2.1</td></tr><tr><td>mau-olhado</td><td>1.8 ±1.2</td><td>4.2 ±1.5</td><td>2.3 ±2.1</td></tr><tr><td>câmara fria</td><td>3.6 ±2.2</td><td>5.0 ±0.0</td><td>3.4 ±2.1</td></tr><tr><td>núcleo atômico</td><td>5.0 ±0.0</td><td>4.4 ±1.8</td><td>5.0 ±0.0</td></tr><tr><td>pão-duro</td><td>0.0 ±0.0</td><td>1.0 ±1.7</td><td>0.0 ±0.0</td></tr><tr><td>sentença judicial</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td></tr><tr><td>tartaruga-marinha</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td></tr><tr><td>vôo internacional</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td><td>5.0 ±0.0</td></tr></tbody></table>

Two main evaluation strategies to assess MWE discovery have been employed in the past. Lin (1999) and Schone & Jurafsky (2001), for example, compare the discovered MWEs to the list of multiword entries in the English WordNet. They assume that, if the lexicon did not exist, the discovered candidates would have helped create it, providing a certain proportion of useful/correct MWEs.<fnref n="5" />

An alternative evaluation strategy consists in annotating (a sample of) the MWE candidates manually. For instance, Evert & Krenn (2001) estimate the precision of several association measures by annotating *n*-best lists as to whether their elements are true MWEs or free phrases. The latter strategy is probably at the origin of compositionality annotation in NLP. Compositionality can be seen as a binary characteristic of word combinations, or as a continuum value ranging

<footnote n="5">⁵This evaluates the *precision* of the method, assuming that the lexicon has a decent coverage. However, recall is harder to assess, since it would require estimating the number of MWEs present in the corpus used for discovery.</footnote>
from more transparent and more opaque expressions (Cruse 1986: p. 39). For example, Blaheta & Johnson (2001) annotate samples of automatically discovered verbal MWEs for phrasality, transitivity, opacity and relatedness. While the first three aspects are modelled as yes/no flags, the latter is a “purely subjective judgment on a scale from 1–5, on whether a collocation really is strongly related or not” (Blaheta & Johnson 2001).

**The graded compositionality paradigm** The idea of judging compositionality using a graded scale of numerical scores was first introduced by McCarthy et al. (2003). In their work, 3 experts first provide numerical scores ranging from 0 (idiomatic) to 10 (compositional) for 116 verb-particle combinations in English. The average scores across the 3 judges are then used to rank the verb-particle constructions according to their compositionality.<fnref n="6" /> Automatic predictions are obtained for these combinations in various ways from a distributional thesaurus and from resources such as WordNet. The idea of the proposed evaluation strategy is that a good compositionality prediction method should rank the candidates in a similar order as the ranking provided by human experts. Thus, the Spearman correlation rank between the gold standard and the predictions was introduced as an evaluation metric for this task.

This work was extremely influential and inspired a large part of subsequent MWE compositionality research. It was later extended to verb-noun and verb-adjective constructions by McCarthy et al. (2007), using a subset of the dataset built by Venkatapathy & Joshi (2005), containing 1–6 ratings provided by 2 experts. Around the same period, Piao et al. (2006) propose a dataset containing 89 English MWEs of heterogeneous categories, not limited to verb-particle or verb-noun patterns. Groups of 4 to 6 experts annotated these expressions on a 0–10 scale for compositionality, and inter-rater agreement was reported.

**From types to tokens** The 2008 edition of the MWE workshop included a special track for papers describing MWE datasets, also released on the workshop website.<fnref n="7" /> Most of these are MWE lexicons, that is, they contain only entries considered as true MWEs.<fnref n="8" /> Nonetheless, the dataset created by Krenn (2008) contains compositionality ratings for around 21k German preposition-noun-verb constructions. Instead of using a numerical scale, combinations are annotated as

<footnote n="6">⁶Actually, 111 combinations are used, because 5 of them are considered problematic by at least one expert.</footnote>

<footnote n="7">⁷http://multiword.sf.net/PHITE.php?sitesig=FILES&page=FILES_20_Data_Sets</footnote>

<footnote n="8">⁸MWE lexicons are out of scope because they do not contain literal/free phrases, so they are not suitable to assess methods distinguishing idiomatic phrases (MWEs) from literal phrases.</footnote>
to whether they are (a) idiomatic, (b) light-verb constructions, or (c) free phrases. Another important dataset in the MWE 2008 collection is VNC-Tokens (Cook et al. 2008). This was the first collection of binary compositionality judgments in context, that is, annotated at the level of token occurrences, not at the level of types as previous datasets. It contains 2,984 annotated occurrences (token instances) of 53 different English verb-noun constructions (types).

Similarly to VNC-Tokens, the OpenMWE corpus is an impressive collection of almost 103k MWE occurrences in Japanese annotated as compositional or literal (Hashimoto & Kawahara 2008). Tu & Roth (2011) and Tu & Roth (2012) also created datasets similar to VNC-Tokens, but containing English light-verb and verb-particle constructions. Korkontzelos et al. (2013) describe a dataset of English sentences containing potentially idiomatic expressions, used in subtask B of SemEval task 5 in two versions: one in which the test items were observed in the training data, and one in which all test sentences were instances of MWE types not observed in the training data. Also for English, Sporleder & Li (2009) propose a method to detect non-literal language based on discourse cohesion chains. They evaluate their method on a dataset containing about 4k occurrences of 17 verbal MWEs annotated as literal or idiomatic. This was later extended in the IDIX corpus, which uses a slight more complex annotation scheme and contains almost 6k occurrences of 78 English verbal MWEs (Sporleder et al. 2010). Similarly, more than 9,7k sentences containing German preposition-noun-verb constructions are annotated as literal, idiomatic, both (the context does not allow to disambiguate), or extraction errors by Fritzinger et al. (2010). With a focus oriented more towards morphology, Bergsma et al. (2010) propose a dataset of ~1.7k prefix verb occurrences in English with binary compositionality annotations. Birke & Sarkar (2006) perform token annotations of non literal language. Although MWE dictionaries were used to select their sentences, their annotations concern single verbs, not MWEs.

**The wisdom of the crowds** The work of Reddy et al. (2011) revived the interest in type-level compositionality prediction using word embeddings. This was also the first dataset where numerical ratings were obtained via crowdsourcing, averaged over about 30 crowdworkers per MWE. Annotators judge not only the whole nominal compound, but also the contribution of the head and of the modifier towards the meaning of a whole on a 0–5 scale. The type-level German datasets GhoSt-PV (Bott et al. 2016) and GhoSt-NN (im Walde et al. 2016) contain phrasal verbs and noun-noun compounds rated on a 1–6 compositionality scale. These works inspired the English, French and Brazilian Portuguese datasets proposed by Ramisch, Cordeiro, Zilio, et al. (2016) and later extended by Cordeiro
et al. (2019), which will be detailed in §3.2.3. More recently, similar datasets have been proposed for Swedish (Kurfalı et al. 2020) and Chinese (Qi et al. 2019), the latter also including sememe annotation. The Norwegian Blue Parrot dataset contains (mostly nominal) English head-modifier pairs with numerical typicality ratings that can be seen as a proxy for compositionality (Kruszewski & Baroni 2014).

Type-level datasets with discrete 2-way or 3-way ratings include English nominal compounds (Farahmand et al. 2015), Basque verb-noun combinations (Gurutxaga & Alegria 2012), German verb-verb constructions (Horbach et al. 2016), Russian nominal compounds (Puzyrev et al. 2019). The datasets of the DisCo shared task, in English and in German, are quite particular in that entries were annotated on a token basis (sentences), but shared task participants were provided type-level scores averaged over token instances (Biemann & Giesbrecht 2011). Moreover, two versions of the dataset were made available: one with 0–10 numerical scores, and another with a 3-way coarse classification.

### The return of token-level ratings

The first datasets containing token-level numerical compositionality scores were proposed recently, containing entries in English and Portuguese (Garcia et al. 2021b,a). The MAGPIE corpus is a large collection of discrete compositionality ratings for more than 56k English sentences (Haagsma et al. 2020). Similarly, Madabushi et al. (2021) create a dataset containing idiomatic and literal instances of English and Portuguese nominal compounds. This dataset was later enhanced and extended to Galician for the SemEval 2022 task 2 challenge (Madabushi et al. 2022). With the goal of quantifying verbal MWE ambiguity, Savary, Cordeiro, Lichte, et al. (2019) performed a fine-grained annotation of literal occurrences of MWEs from the PARSEME corpora in German, Basque, Greek, Polish and Portuguese, as we will detail in §3.2.4.

### 3.2.2 Discussion

Table 3.2 summarises the datasets described above. Along with each citation of the article describing the dataset, we indicate whether annotation is done at the level of types or tokens, the language(s) of the dataset, as well as the size of the dataset in terms of its number of annotated entries (with the number of types in parentheses for token-level datasets). Then, we specify the target syntactic structure or MWE category of the items, and whether annotations were performed by experts or by crowdworkers (CW). Whenever available, we also indicate the approximate number of annotations per entry. Finally, we indicate whether the judgments are numerical (real) or discrete (e.g. 2-way), along with
3 Fifty shades of compositionality

<table><caption>Table 3.2: Summary of compositionality datasets. CWs: crowd workers.</caption><thead><tr><th>Reference</th><th>Level</th><th>Lang.</th><th>Size (types)</th><th>Category</th><th>Raters/MWE</th><th>Range</th></tr></thead><tbody><tr><td>McCarthy et al. (2003)</td><td>type</td><td>English</td><td>116</td><td>verb-particle</td><td>3 experts</td><td>real: 0-10</td></tr><tr><td>Venkatapathy & Joshi (2005)</td><td>type</td><td>English</td><td>765</td><td>verb-(noun|adj)</td><td>2 experts</td><td>real: 1-6</td></tr><tr><td>Birke & Sarkar (2006)</td><td>token</td><td>English</td><td>3,977 (50)</td><td>verb</td><td>2 experts</td><td>2-way: idiom, lit</td></tr><tr><td>Piao et al. (2006)</td><td>type</td><td>English</td><td>89</td><td>unrestricted</td><td>4 to 6 experts</td><td>real: 0-10</td></tr><tr><td>McCarthy et al. (2007)</td><td>type</td><td>English</td><td>638</td><td>verb-(noun|adj)</td><td>2 experts</td><td>real: 1-6</td></tr><tr><td>Krenn (2008)</td><td>type</td><td>German</td><td>21,796</td><td>prep-noun-verb</td><td>1 expert</td><td>3-way: idiom, LVC, lit</td></tr><tr><td>Cook et al. (2008)</td><td>token</td><td>English</td><td>2,984 (53)</td><td>verb-noun</td><td>2 experts</td><td>3-way: idiom, lit, ?</td></tr><tr><td>Hashimoto & Kawahara (2008)</td><td>token</td><td>Japanese</td><td>102,846 (146)</td><td>unrestricted</td><td>2 experts</td><td>2-way: idiom, lit</td></tr><tr><td>Sporleder & Li (2009)</td><td>token</td><td>English</td><td>3,964 (17)</td><td>verbal</td><td>1 expert</td><td>2-way: idiom, lit</td></tr><tr><td>Sporleder et al. (2010)</td><td>token</td><td>English</td><td>5,836 (78)</td><td>verbal</td><td>experts</td><td>6-way: idiom, lit, both, ?, ...</td></tr><tr><td>Fritzinger et al. (2010)</td><td>token</td><td>German</td><td>9,740 (77)</td><td>prep-noun-verb</td><td>2 experts</td><td>4-way: idiom, lit, both, err</td></tr><tr><td>Bergsma et al. (2010)</td><td>token</td><td>English</td><td>1,718 (1,248)</td><td>prefix-verb</td><td>1 expert</td><td>2-way: idiom, lit</td></tr><tr><td>Reddy et al. (2011)</td><td>type</td><td>English</td><td>90</td><td>(noun|adj)-noun</td><td>~30 CWs</td><td>real: 0-5</td></tr><tr><td rowspan="2">Biemann & Giesbrecht (2011)</td><td rowspan="2">type</td><td>English</td><td>349</td><td rowspan="2">(adj|verb)-noun & verb-object</td><td rowspan="2">~12-20 CWs</td><td>real: 0-10</td></tr><tr><td>German</td><td>297</td><td>3-way: low, mid, high</td></tr><tr><td>Tu & Roth (2011)</td><td>token</td><td>English</td><td>2,162 (1,643)</td><td>verb-noun</td><td>2 experts</td><td>2-way: LVC, non LVC</td></tr><tr><td>Tu & Roth (2012)</td><td>token</td><td>English</td><td>1,348 (23)</td><td>verb-particle</td><td>CWs</td><td>2-way: VPC, non VPC</td></tr><tr><td>Gurrutxaga & Alegria (2012)</td><td>type</td><td>Basque</td><td>590</td><td>verb-noun</td><td>3 experts</td><td>3-way: idiom, colloc, lit</td></tr><tr><td rowspan="2">Korkontzelos et al. (2013)</td><td rowspan="2">token</td><td>English</td><td>2,376 (30)</td><td rowspan="2">unrestricted</td><td rowspan="2">CWs</td><td rowspan="2">3-way: idiom, lit, both</td></tr><tr><td>(unseen)</td><td>1,974 (45)</td></tr><tr><td>Kruszewski & Baroni (2014)</td><td>type</td><td>English</td><td>5,840</td><td>head-modifier</td><td>~10 CWs</td><td>real: 0-1 & 0-7</td></tr><tr><td>Farahmand et al. (2015)</td><td>type</td><td>English</td><td>1,042</td><td>(noun|adj)-noun</td><td>4 experts</td><td>2-way: idiom, lit</td></tr><tr><td>im Walde et al. (2016)</td><td>type</td><td>German</td><td>868</td><td>noun-noun</td><td>experts & CWs</td><td>real: 1-6</td></tr><tr><td>Bott et al. (2016)</td><td>type</td><td>German</td><td>400</td><td>verb-particle</td><td>~16 CWs</td><td>real: 1-6</td></tr><tr><td>Horbach et al. (2016)</td><td>type</td><td>German</td><td>5,950 (6)</td><td>verb-verb</td><td>2 experts</td><td>3-way: idiom, lit, ?</td></tr><tr><td rowspan="3">Cordeiro et al. (2019)</td><td rowspan="3">type</td><td>English</td><td>190</td><td rowspan="3">(noun|adj)-noun</td><td rowspan="3">~15-30 CWs</td><td rowspan="3">real: 0-5</td></tr><tr><td>French</td><td>180</td></tr><tr><td>Portug.</td><td>180</td></tr><tr><td rowspan="4">Savary, Cordeiro, Lichte, et al. (2019)</td><td rowspan="4"></td><td>German</td><td>4,749</td><td rowspan="4">verbal</td><td rowspan="4">1-2 experts</td><td rowspan="4">10-way: idiom, lit, err, ...</td></tr><tr><td>Basque</td><td>2,856</td></tr><tr><td>Greek</td><td>6,441</td></tr><tr><td>Polish</td><td>5,175</td></tr><tr><td>Qi et al. (2019)</td><td>type</td><td>Chinese</td><td>500</td><td>unrestricted</td><td>3 experts</td><td>real: 0-3</td></tr><tr><td>Puzyrev et al. (2019)</td><td>type</td><td>Russian</td><td>220</td><td>(noun|adj)-noun</td><td>2 experts</td><td>3-way: idiom, lit, both</td></tr><tr><td>Kurfali et al. (2020)</td><td>type</td><td>Swedish</td><td>96</td><td>unrestricted</td><td>~17 CWs</td><td>real: 0-5</td></tr><tr><td>Haagsma et al. (2020)</td><td>token</td><td>English</td><td>56,622 (1,756)</td><td>unrestricted</td><td>3-9 CWs</td><td>5-way: idiom, lit, err, ?, other</td></tr><tr><td rowspan="2">Garcia et al. (2021a)</td><td rowspan="2">token</td><td>English</td><td>840 (290)</td><td rowspan="2">(noun|adj)-noun</td><td rowspan="2">~9-21 CWs</td><td rowspan="2">real: 0-5</td></tr><tr><td>Portug.</td><td>540 (180)</td></tr><tr><td rowspan="2">Garcia et al. (2021b)</td><td rowspan="2">token</td><td>English</td><td>5,620 (280)</td><td rowspan="2">(noun|adj)-noun</td><td rowspan="2">not specified</td><td rowspan="2">real: 0-5</td></tr><tr><td>Portug.</td><td>3,600 (180)</td></tr><tr><td rowspan="2">Madabushi et al. (2021)</td><td rowspan="2">token</td><td>English</td><td>4,558 (223)</td><td rowspan="2">(noun|adj)-noun</td><td rowspan="2">experts</td><td rowspan="2">2-way: idiom, lit</td></tr><tr><td>Portug.</td><td>1,872 (113)</td></tr><tr><td rowspan="3">Madabushi et al. (2022)</td><td rowspan="3">token</td><td>English</td><td>5,352 (50)</td><td rowspan="3">(noun|adj)-noun</td><td rowspan="3">experts</td><td rowspan="3">2-way: idiom, lit</td></tr><tr><td>Portug.</td><td>2,555 (50)</td></tr><tr><td>Galician</td><td>776 (50)</td></tr></tbody></table>
the range of values. For numerical ratings, the lower bound (usually 0 or 1) is
used for idiomatic combinations whereas the upper bound of the range indicates
fully compositional entries. The only exception is the dataset by Farahmand et
al. (2015), where 0 means compositional and 1 means idiomatic. A full version of
this table, containing also a download link to the dataset when available, can be
visualised here: https://docs.google.com/spreadsheets/d/1wmlhJcPkqfadRp-
rH_5lCEkwLRgDCrCF20YE3iP_qUg/edit?usp=sharing.

**Context** One of the main difference across compositionality datasets is the amount of context seen by raters and included in the dataset. *Type-level* scores are provided for each candidate expression in isolation, completely out of context, whereas expressions to assess are given within sentences or paragraphs in *token-level* datasets. The amount of context available for token-level annotation varies: most dataset contain a single sentence of occurrence (Cook et al. 2008; Fritzinger et al. 2010; Tu & Roth 2012), but some of them provide larger contexts such as a single previous/next sentence (Madabushi et al. 2021; 2022), a few surrounding sentences (Haagsma et al. 2020), or even a couple of surrounding paragraphs (Sporleder et al. 2010).

One of the advantages of type-level annotation is that it is faster, as it does not require reading whole sentences. On the other hand, many MWEs can have both idiomatic and literal senses depending on their context. As a consequence, the obtained type-level scores are often more subjective since they depend on the most salient contexts for each annotator. One way to attenuate this subjectivity is to average across many (non expert) annotators, assuming that this collective rating should be representative of the most frequent sense(s) of the expression (Reddy et al. 2011; Bott et al. 2016; Kurfalı et al. 2020). Authors may also explicitly ask annotators to think about the most common sense of an MWE, even if it is provided out of context. In addition, it is possible to guide annotators by first priming them with a few sentences containing the expression, and then asking for type-level judgments (Cordeiro et al. 2019), or to provide dictionary definitions of the target MWEs under consideration (Haagsma et al. 2020). Some authors, however, argue that they prefer not biasing annotators, so no help is provided (Kurfalı et al. 2020).

**Granularity or range** Another design choice in these datasets is the set of labels used to represent compositionality. The simplest tagsets make a binary distinction between literal and idiomatic expressions or occurrences (Farahmand
### 3 Fifty shades of compositionality

et al. 2015; Tu & Roth 2012; Madabushi et al. 2021).<fnref n="9" /> When several annotators are involved, usually the ratings are adjudicated so that a single consensual label is provided in the released dataset (Cook et al. 2008). Alternatively, all ratings can be provided separately, so that users of the dataset decide on the best way to evaluate their prediction models (Farahmand et al. 2015). For example, both Yazdani et al. (2015) and Cordeiro et al. (2019) use the sum of the ratings of the 4 experts as their gold reference for the Farahmand et al. (2015) dataset.

Some extra labels may be used in token-level annotation, such as considering some instances as truly ambiguous between literal and idiomatic reading in the absence of extra context (Korkontzelos et al. 2013) or are simply not occurrences of the target combination, that is, false extractions (Fritzinger et al. 2010)<fnref n="10" /> Id- iomatic instances can have multiple senses, e.g. a **black box** can be an opaque model or a recording device in an aircraft. Some datasets include fine-grained sense distinctions for idiomatic readings (Sporleder et al. 2010; Madabushi et al. 2021). On the other hand, literal readings may also be classified as to the reason why they are not instances of the target MWE (Savary, Cordeiro, Lichte, et al. 2019), as detailed in §3.2.4.

Finally, most type-level datasets adopt a numerical range to model the compo- sitionality of the items, averaging the (integer) ratings across all annotators to obtain a single real-valued compositionality score for each combination. How- ever, the range of values varies: while a 6-points likert scale ranging from 0 to 5 (or from 1 to 6) is quite common (Reddy et al. 2011; im Walde et al. 2016; Cordeiro et al. 2019; Kurfahli et al. 2020), other choices include 0–3 (Qi et al. 2019), 0–7 (Kruszewski & Baroni 2014), and 0–10 (McCarthy et al. 2003; Piao et al. 2006; Biemann & Giesbrecht 2011).<fnref n="11" /> Hence, it seems to us that there is some degree of arbitrariness and/or trial and error in the choice of the number of composition- ality levels in the range.

### Selection of entries

Token-based datasets select their context sentences from numerous sources. For English, the BNC corpus is often used to collect occur- rences to annotate (Cook et al. 2008; Sporleder et al. 2010; Haagsma et al. 2020), although larger web-based corpora such as the UkWaC (English), DeWaC (Ger- man) and BrWaC (Brazilian Portuguese) are also sometimes employed (Biemann

<footnote n="9">⁹Some papers will label idiomatic expressions as “true MWEs” or “non-literal”, whereas what we call compositional can sometimes be referred to as “free phrases”.</footnote>

<footnote n="10">¹⁰In Savary, Cordeiro, Lichte, et al. (2019), we formalise this notion and name it “coincidental co-occurrence”.</footnote>

<footnote n="11">¹¹An even number of labels prevents annotators from being indecisive by often choosing the middle score.</footnote>
& Giesbrecht 2011; Garcia et al. 2021a). It is possible to use parsed corpora com-
bined with morpho-syntactic patterns to locate instances of the target MWE
category (Fritzinger et al. 2010). Alternatively, the target idiom types are often
pre-selected as well, favouring entries already present in MWE dictionaries (Ko-
rkontzelos et al. 2013), assessed as potentially ambiguous (Cook et al. 2008), or
annotated as idiomatic elsewhere in the corpus (Savary, Cordeiro, Lichte, et al.
2019). Finally, a different perspective to create a token-level compositionality
dataset consists in providing annotators with the target MWE types (and their
possible senses), and then ask them to provide a certain number of corpus or web
occurrences of the MWEs in each of the given senses (Madabushi et al. 2022).

**Annotator training** Compositionality is a complex linguistic notion, therefore many authors prefer expert annotators over native speaker workers of crowdsourcing platforms (Farahmand et al. 2015). Nevertheless, crowdworkers are often readily available and allow collecting many annotations that can be averaged to compensate for the lack of annotator training (Reddy et al. 2011). When crowdsourcing is employed, some effort goes into designing minimal tasks using accessible terms and simplified questions. For instance, instead of asking to assign a compositionality degree, one can ask to what extent a **black box** is literally a *box* that is *black*. Moreover, crowdsourced datasets constrain annotators to be from a given country (Biemann & Giesbrecht 2011), or ask them to pass language tests to ensure their proficiency in the target language (Reddy et al. 2011). Finally, in addition to requiring a certain level of reliability for the recruited crowdworkers, it is also possible to filter out outlier judgments or data from annotators who do not follow the overall trend of all other annotators (Roller et al. 2013; Ramisch, Cordeiro & Villavicencio 2016; Kurfalı et al. 2020).

**Languages and MWE categories** A total of 12 languages are covered in these datasets, with about half of them (22 out of 45) for English. Except for German (7), Brazilian Portuguese (6) and Basque (2), all other languages only have one dataset. Only 4 datasets represent non Indo-European languages (Japanese, Chinese, and Basque). In terms of the syntactic structure of the annotated entries, 6 datasets (18%) contain unrestricted entries, whereas 10 contain nominal expressions (30%) and the remaining 16 datasets focus on verbal expressions (49%), with one dataset containing both nominal and verbal items (Biemann & Giesbrecht 2011). Among nominal expressions, the most common pattern is composed of modifier nouns or adjectives combined with a head noun, many of them inspired by Reddy et al. (2011). The most common pattern of verbal expression is verb+noun (6 datasets).
Verb-particle constructions are represented in 2 English and 1 German dataset, whereas 2 datasets contain German constructions composed of verbs with prepositional phrases. Less frequent are verb-verb pairs (1) and prefix verbs (1). In sum, the datasets are skewed towards English and quite heterogeneous in terms of syntactic patterns and sizes, ranging from as little as 89 types (Piao et al. 2006) to as many as 102k tokens (Hashimoto & Kawahara 2008).

**Extra annotations** Most recent datasets provide some extra information in addition to compositionality scores. Some type-level datasets include not only the compositionality of the whole expression, but also the semantic contribution of each of its components (Reddy et al. 2011; Cordeiro et al. 2019; Kurfali et al. 2020; Garcia et al. 2021a). The dataset of Farahmand et al. (2015) contains also conventionality annotations, while membership scores are provided by Kruszewski & Baroni (2014). Ghost-PV also includes, for each PV, their frequency, ambiguity degree, and proportion of split occurrences (Bott et al. 2016). GhoSt-NN is enriched with frequency, productivity and ambiguity, and a subset of 180 compounds was selected for balancing these aspects (im Walde et al. 2016).

It is common for non-English datasets to include the lemmas of the components (Hashimoto & Kawahara 2008; Bott et al. 2016; Cordeiro et al. 2019). Sememes of the whole MWE and of each of its component words are provided in the Chinese dataset (Qi et al. 2019). Context sentences are also provided in some type-level datasets (Cordeiro et al. 2019; Puzyrev et al. 2019), as well as extra context in token-level datasets (Haagsma et al. 2020; Madabushi et al. 2022). Idiomatic senses can be listed (Sporleder et al. 2010; Madabushi et al. 2021), and paraphrases of the idiomatic senses can be provided for the expression out of context (Cordeiro et al. 2019), in context (Bergsma et al. 2010; Garcia et al. 2021a), or per MWE sense (Madabushi et al. 2021).

**Related datasets** Some datasets not included in this survey contain annotations related to compositionality. Some datasets model the semantics of nominal compounds using a closed label set of semantic relations (Hendrickx et al. 2010) or an open set of paraphrases involving verbs and prepositions, e.g. *air filter → filter for air* or *filter that cleans the air* (Nakov 2008; Butnariu et al. 2010; Hendrickx et al. 2013). However, these datasets do not explicitly contain idiomatic expressions, where the semantic relation label or a paraphrase involving the component nouns cannot directly express the meaning of the whole expression.

On the other hand, *metaphor* datasets may be quite similar to token-level compositionality datasets (Tong et al. 2021: p. 4675–4676). However, they tend to focus on metaphorical uses of single words, without identifying the collocates that
trigger the non-literal interpretation. Moreover metaphor datasets also usually
do not explicitly address idiomatic or fixed expressions. In practice, there is a
gray zone in the distinction between metaphors and idioms, and it is difficult to
draw a clear cut line (Savary et al. 2017) Empirical results indicate that verbal
MWE identification, for instance, can help in automatic metaphor identification
(Rohanian et al. 2020).

**Availability** To conclude, we underline that the long-term availability of the discussed resources is an issue for reproducible research. The articles describing datasets often contain broken links to personal websites of authors who changed affiliations since. Getting the actual data requires some web archaeology and often involves contacting the authors (after finding out their most recent email address). After some effort, we were able to retrieve 27 out of the 33 surveyed datasets. The retrieved datasets are now available on a centralised hub: https://gitlab.com/ceramisch/comp-datasets. However, the licence files for each dataset were not systematically checked, and still require some future work to ensure that they are actually shareable in this way.

### 3.2.3 Nominal compounds in English, French and Portuguese

This section details the creation of original compositionality datasets for nominal
compounds in three languages. At that time of this work, no dataset was avail-
able for French and Portuguese, and the English dataset of Reddy et al. (2011) con-
tained only 90 nominal compounds. Thus, to study embedding-based composi-
tionality prediction models cross-lingually and consistently, we decided to create
new datasets. This section is structured as follows: §3.2.3.1 introduces the dataset
creation, initially presented in Ramisch, Cordeiro, Zilio, et al. (2016); then §3.2.3.2
presents analyses and data filtering techniques studied in Ramisch, Cordeiro &
Villavicencio (2016); and §3.2.3.3 overviews the extension of the dataset, includ-
ing the addition of lexical replacement candidates (Wilkens et al. 2017). These
datasets were employed to evaluate the compositionality prediction models de-
scribed in §3.3 (Cordeiro et al. 2019).

#### 3.2.3.1 Dataset creation

Instead of covering general (unrestricted) MWEs, our dataset focuses on a par-
ticular category: nominal compounds. The terms *noun compound* and *compound noun* are usually reserved for nominal compounds formed by sequences of nouns
only, typical of Germanic languages like English, but not frequent in Romance
languages. Thus, we define **nominal compounds** more generally as syntactically
well formed and conventionalized noun phrases containing two or more content
words, whose head is a noun.

In the three languages, we selected only 2-word nominal compounds: in En-
glish, the modifier is always preposed to the head noun, and is frequently another
noun (e.g. **en** *fish story* 'exaggerated, unbelievable story'), but can also be an ad-
jective (e.g. **en** *eager beaver* 'enthusiastic person') or a gerund nominalisation
(e.g. **en** *swimming pool*). In Portuguese and in French, we selected nominal com-
pounds whose modifier is always a single adjective, either preposed (e.g., **fr** *carte
bleue* (lit. 'card blue') 'credit card') or postposed (e.g. **pt** *pé-quente* (lit. 'foot-hot')
'lucky person').

As illustrated in these examples, compounding does not imply concatenation,
and all our compounds are written either with a space or a hyphen between
modifiers and head nouns. Although so-called "closed" compounds do exist in
English (e.g. *snowman*, *database*), we do not include them in our datasets, favour-
ing surface form homogeneity. For the same reason, we also did not consider
compounds involving prepositional complements which could be seen as French
and Portuguese equivalents of English noun-noun compounds (e.g. **en** *lung cancer* → **fr** *cancer du poumon*, **pt** *câncer de pulmão*).

One advantage of working with nominals instead of verbal expressions is that
this simplifies their identification in corpora, as their variability is limited to mor-
phological inflection in the target languages, but adjacency and order of compo-
nents are fixed. This will turn out to be convenient later, when building embed-
ding representations for whole compounds from their corpus occurrences (§3.3).

Nominal compounds are conventionalized in the sense that their particular re-
alization is statistically idiosyncratic, and their constituents cannot be replaced
by synonyms. Their semantic interpretation may be straightforwardly composi-
tional, with contributions from both elements (e.g. *climate change*), partly com-
positional, with contribution mainly from one of the elements (e.g. *grandfather
clock* is a clock, but does not need to belong to a grandfather), or idiomatic (e.g. a
*suggar daddy* is neither sweet nor a parent, but a rich older male partner). Our
work follows the protocol proposed by Reddy et al. (2011), where composition-
ality is explained in terms of the literality of the individual parts. This type of
indirect annotation does not require expert linguistic knowledge, and still pro-
vides reliable data, as we show later. For each language, data collection involved
the following steps: (a) compound selection, (b) sentence selection, (c) question-
naire design, and (d) data collection and aggregation.
**Compound selection** The initial set of idiomatic and partially compositional candidates was constructed by introspection, independently for each language, since these may be harder to find in corpora because of their lower frequency. These initial set of compounds was complemented by selecting entries from lists of frequent adjective + noun and noun + noun pairs. The lists were automatically extracted through POS-sequence queries using the mwetoolkit (Ramisch 2015) from the ukWaC (Baroni et al. 2009), frWaC (Ferraresi et al. 2010) and brWaC (Filho et al. 2018) corpora, all containing between 1 and 2 billion tokens. We disregarded all compounds that we considered controversial. Examples include those in which the complement is not an adjective in Portuguese/French (e.g. pt abelha rainha (lit. ‘bee queen’) ‘queen bee’), those in which the head is not necessarily a noun (e.g. fr aller simple (lit. ‘to-go simple’) ‘one-way trip’, as aller is a noun which also occurs frequently as a verb) and those in which the literal sense seems to be very common in the corpus (e.g. en low blow). We did not attempt to select translation equivalents for all three languages. A compound in a given language may have no equivalent idiomatic compound or correspond to a single word in the other languages, and even when it does translate as a compound, its POS pattern and level of compositionality may be very different.

We attempted to pre-select a balanced set of 1/3 idiomatic, 1/3 partially compositional and 1/3 fully compositional compounds. Thus, coarse compositionality labels were assigned by the authors during a preliminary pre-annotation of the candidate entries into one of these three classes. These pre-annotations were used only to select the compounds to be annotated, but were not used in further steps nor shown to annotators. In Portuguese and French, 60 compounds of each coarse compositionality range were selected, for a total of 180 compounds. In English, we initially selected 30 compounds of each coarse range, since we intended to complete the 90 compounds of the Reddy et al. (2011) dataset with 90 compounds, reaching an identical number of 180 compounds per language. Most of the analyses presented here for English consider the union of our 90 English compounds with the Reddy dataset. However, this dataset was later extended to 100 additional English compounds using the same methodology (used as held-out data in our experiments).

**Sentence selection** For each compound, we selected 3 sentences from a WaC corpus where the compound is used. These sentences are used during the data collection process as disambiguating context shown to the annotators to guide their interpretation. We sort them by sentence length, in order to favor shorter sentences, and manually select 3 examples that satisfy the following criteria:
### 3 Fifty shades of compositionality

1. The occurrence of the nominal compound must have the same meaning in all sentences.

2. A sentence must contain enough context to enable mental disambiguation of the compound.

3. Inter-sentence variability can inform the annotators about the different uses of the compound.

<figure><img src="image_9.png" /><figcaption>Figure 3.1: Sample question for the compound en *pocket book*.</figcaption></figure>

**Questionnaire design** We collect data for each compound through a separate task web page containing a short list of instructions followed by the questionnaire associated with that compound. In the instructions, we briefly describe the task and require that the annotators fill in an external identification and training form, following Reddy et al. (2011). This form provides us with demographics about the annotators, ensuring that they are native speakers of the target language. At the end of the form, they are also given extra example questions with annotated answers for training, and must confirm that they have read and understood the instructions. After filling in the identification form, annotators can start working on the task (this is only required the first time).

The task page is structured into 6 subtasks, as illustrated in Figure 3.1:
1. Read the compound itself.

2. Read 3 sentences containing the compound.

3. Provide 2 to 3 synonym expressions for the target compound seen in the sentences.

4. Using a Likert scale from 0 to 5, judge how much of the meaning of the compound can be inferred from the literal meaning of the modifier.

5. Using a Likert scale from 0 to 5, judge how much of the meaning of the compound can be inferred from the literal meaning of the head noun.

6. Using a Likert scale from 0 to 5, judge how much of the meaning of the compound comes from the meanings of both head and modifier.

We have been consciously careful about asking for answers falling within an even-numbered scale (0–5 makes for 6 reply categories), as otherwise, undecided annotators would be biased towards the middle score. We avoid incomplete answers by making subtasks 3–5 mandatory. The order of subtasks has also been taken into account. During a pilot test, we found that presenting the multiple-choice questions (subtasks 4–6) before asking for synonyms (subtask 3) yielded lower agreement, as users were often less self-consistent in the multiple-choice questions (e.g. replying “non-compositional” for subtasks 4 or 5 but “compositional” for subtask 6), even if they carefully selected their synonyms in response to subtask 3. Asking for synonyms prior to the multiple-choice questions helps the user focus on the target meaning for the compound and also have more examples (the synonyms) when considering the semantic contribution of each element of the compound.

The initial instructions are written as concisely as possible, serving more as an introduction to the task and redirecting to the identification and training form for details. For the last three questions, on mouse hover, annotators see a tooltip with an interpretation for each numerical score label, explicating the (potential) relation between the head and the modifier, as shown in Figure 3.1. This guarantees that the annotator knows exactly what reply is being submitted, without relying on their ability to remember all the instructions.

**Data collection and aggregation** Annotators were recruited and paid via the Amazon Mechanical Turk crowdsourcing platform for English and French. The quality of the submitted responses was ensured manually by checking whether the paraphrase suggestions were reasonably related to the compound, rejecting answers that do not pass this assessment. During a pilot test, we noticed the
### 3 Fifty shades of compositionality

lack of qualified Portuguese native speakers on the Amazon platform. Therefore, judgments for this language were provided by volunteers through a standalone web interface that simulated the task page from the crowdsourcing platform. The questionnaire was shared with colleagues of the authors and advertised on Portuguese-speaking NLP mailing lists.

We collected answers from around 15 participants per compound in each language. Then for each compound and for each question in subtasks 4–6, we calculate aggregated scores as the arithmetic averages of all answers across participants, thus summarising a set of integer scores into a single real number ranging from 0 (idiomatic) to 5 (compositional). In the remainder of this section, we refer to these averaged scores as the *human compositionality scores*. We average the answers to the three questions independently, generating three scores: $hc_H$ for the head noun, $hc_M$ for the modifier, and $hc_{HM}$ for the whole compound. In our experiments, we predict $hc_{HM}$ automatically (§3.3).

In the remainder of this Section, we will refer to the French and Portuguese datasets as *FR-comp* and *PT-comp*, whereas the English datasets will be referred to as *EN-comp*$_{90}$ for the initial 90 compounds and *EN-comp*$_{Ext}$ for the held-out part containing 100 compounds. In addition, we will refer to *EN-comp* as the union between *EN-comp*$_{90}$ and the dataset of Reddy et al. (2011), abbreviated as *Reddy*. The final dataset is freely available at: https://doi.org/10.5281/zenodo.8296689.

#### 3.2.3.2 Filtering and analyses

After having collected the datasets, we have performed three analyses. First, we experimented with two techniques to filter out outliers and reduce the variability inherent to the highly subjective nature of the task. Secondly, we studied the distribution of the average scores and of their standard deviations as an indicator of variability. Finally, we estimated inter-annotator agreement via re-annotation of the same compounds by experts.

**Data filtering** We employ two filtering strategies, for individual ratings and for all the ratings from the same annotator (Roller et al. 2013). *Z-score filtering* aims at removing outlier *annotations*. The standard deviation $\sigma$ of a human compositionality score $hc$ estimates its average distance from the mean. Therefore, if annotators agree, $\sigma$ should be low. We remove individual compound *annotations* whose score falls more than $z$ standard deviations away from the average ($\Omega - hc$) of other scores for the same compound.<fnref n="12" /> In other words, we remove

<footnote n="12">¹²The notation $\Omega - x$ denotes all elements that are in the same set as $x$, except $x$ itself.</footnote>
a compound if $\frac{|hc - (\overline{\Omega - hc})|}{\sigma(\Omega - hc)} > z$ for one of the three ratings (hc<sub>HM</sub>, hc<sub>H</sub> or hc<sub>M</sub>). Spearman filtering aims at removing outlier annotators. If two annotators agree, the ranking of the compounds annotated by both must be similar. We compare the ranking of the compounds rated by an annotator *a* with the ranking of the same compounds according to the average of all other annotators $(\overline{\Omega - a})$. In order to consider only order differences rather than value differences, we use Spearman's rank correlation, noted $\rho_{oth}$. We define a threshold *R* on the Spearman rank correlation $\rho_{oth}$ below which we discard all scores provided by the annotator. To assess the effectiveness of filtering, we look at four indicators:

1. $\overline{\sigma}$: average standard deviation of a score hc over all compounds;

2. $P_{\sigma>1.5}$: proportion of compounds with $\sigma$ higher than 1.5 (Reddy et al. 2011);

3. $\overline{n}$: average number of annotations across all compounds; and

4. DRR: data retention rate DRR, that is, the proportion of compounds in the filtered dataset with respect to the initial dataset.

<table><caption>Table 3.3: Intrinsic quality measures for the raw and filtered datasets.</caption><thead><tr><th>Dataset</th><th>n</th><th>σ<sub>HM</sub></th><th>σ<sub>H</sub></th><th>σ<sub>M</sub></th><th>P<sub>σ<sub>HM</sub>>1.5</sub></th><th>P<sub>σ<sub>H</sub>>1.5</sub></th><th>P<sub>σ<sub>M</sub>>1.5</sub></th><th>DRR</th></tr></thead><tbody><tr><td>Reddy</td><td>15</td><td>0.99</td><td>0.94</td><td>0.89</td><td>5.56%</td><td>11.11%</td><td>8.89%</td><td>-</td></tr><tr><td>EN-comp raw</td><td>18.8</td><td>1.17</td><td>1.05</td><td>1.18</td><td>18.89%</td><td>16.67%</td><td>27.78%</td><td>-</td></tr><tr><td>EN-comp filter</td><td>15.7</td><td>0.87</td><td>0.66</td><td>0.88</td><td>3.33%</td><td>10.00%</td><td>14.44%</td><td>83.6%</td></tr><tr><td>FR-comp raw</td><td>14.9</td><td>1.15</td><td>1.08</td><td>1.21</td><td>22.78%</td><td>24.44%</td><td>30.56%</td><td>-</td></tr><tr><td>FR-comp filter</td><td>13</td><td>0.94</td><td>0.83</td><td>0.96</td><td>13.89%</td><td>15.00%</td><td>18.89%</td><td>87.3%</td></tr><tr><td>PT-comp raw</td><td>31.8</td><td>1.22</td><td>1.09</td><td>1.20</td><td>14.44%</td><td>17.22%</td><td>19.44%</td><td>-</td></tr><tr><td>PT-comp filter</td><td>27.9</td><td>1.0</td><td>0.83</td><td>0.97</td><td>6.11%</td><td>8.89%</td><td>12.22%</td><td>87.8%</td></tr></tbody></table>

Table 3.3 presents the quality results for all datasets, in their original form as well as filtered. The filter threshold configurations adopted in these analyses were, for English and Portuguese: $z = 2.2$, $\rho = 0.5$, and for French: $z = 2.5$, $\rho = 0.5$. Filtering does improve the quality of the annotations. The more restrictive the filtering, the lower the number of annotations available, but also the higher is the agreement among annotators, for all languages. When no filtering is performed, there is an average of 14.92 annotations per compound, but average standard deviation values range from 1.08 to 1.21. The proportion of high standard deviation compounds is between 22.78% and 30.56%. With filtering, the number of
### 3 Fifty shades of compositionality

annotations per compound drops to 13.03, but so does the average standard deviation, which becomes smaller than 1. The proportion of high standard deviation compounds is between 14% and 19%.

There is a chance that our filters removes annotations for compounds that are harder to judge, thus artificially inflating the scores for automatic compositionality prediction (§3.3). However, when working with non expert annotators, filtering is required, to mitigate the quality issues due to the wide variability of annotator proficiency, training, and engagement levels. We hope that by keeping an average of 15-30 annotations after filtering, the dataset still reflects somehow the ability of humans to assess compositionality on a graded numerical scale.

**Scores distribution** Figure 3.2(a) shows standard deviation ($\sigma_{HM}$, $\sigma_{H}$ and $\sigma_{M}$) for each compound of FR-comp as a function of its average compound score $hc_{HM}$.<fnref n="13" /> For all three languages, greater agreement was found for compounds at the extremes of the compositionality scale (fully compositional or fully idiomatic) for all scores. These findings can be partly explained by end-of-scale effects, that result in greater variability for the intermediate scores in the Likert scale (from 1 to 4) that correspond to the partly compositional cases. Hence, we expect that it will be easier to predict the compositionality of idiomatic/compositional compounds than of partly compositional ones. Further analyses of the correlation between compositionality, frequency and conventionality can be found in §3.2.2 of Cordeiro et al. (2019) and §3.2 of Cordeiro (2017).

<figure><img src="image_10.png" /><figcaption>Figure 3.2: Left: Standard deviations ($\sigma_{H}$, $\sigma_{M}$ and $\sigma_{HM}$) as a function of $hc_{HM}$ in FR-comp. Right: Average compositionality ($hc_{H}$, $hc_{M}$ and $hc_{HM}$) as a function of $hc_{HM}$ in FR-comp.</figcaption></figure>

<footnote n="13">¹³Only FR-comp is shown as the other datasets display similar patterns.</footnote>
**Inter-annotator agreement** Traditional coefficients such as Cohen's and Fleiss' kappa (Cohen 1960; Fleiss 1971) compare the set of annotations from two or more individuals and yield the proportion of agreeing pairs, taking into account the probability of random agreement. However, they are designed for categorical annotation, whereas our annotators rank items on an ordinal scale. Thus, to measure inter-annotator agreement of multiple participants taking into account the distance between the ordinal ratings of the likert scale, we adopt the α score (Artstein & Poesio 2008). The α score is more appropriate for ordinal data than traditional agreement scores for categorical data. Moreover, due to the use of crowdsourcing, most participants rated only a small number of compounds with very limited chance of overlap among them: the average number of answers per participant is 13.6 for *EN-comp*<sub>90</sub>, 10.2 for *EN-comp*<sub>Ext</sub>, 33.7 for *FR-comp*, and 53.5 for *PT-comp*. Because the α score assumes that each participant rates all the items, we focus on the answers provided by three of the participants, all of them computational linguists, who rated the whole set of 180 compounds in *PT-comp*.

Using a linear distance schema between the answers,<fnref n="14" /> we obtain an agreement of $\alpha = .58$ for head-only, $\alpha = .44$ for modifier-only and $\alpha = .44$ for the whole compound. To further assess the difficulty of this task, we also calculate $\alpha$ for a single expert annotator judging the same set of compounds after an interval of one month. The scores were $\alpha = .69$ for the head and $\alpha = .59$ for both the compound and for the modifier. The Spearman correlation between these two annotations performed by the same expert is $\rho = 0.77$ for $hc_{HM}$. It is hard to determine whether higher agreement scores could be obtained by improving the guidelines and annotation interface, or if this corresponds to a qualitative upper bound for compositionality prediction on *PT-comp*, given the difficulty of the task.

### 3.2.3.3 Extensions

**Lexical substitutes** Numerical compositionality judgments are an interesting model of MWE semantics, but not the only one. An alternative consists in paraphrasing the meaning of the expression using lexical substitutes, that is, words or phrases that express equivalent or similar meaning. Datasets containing lexical substitutes exist for single words (McCarthy & Navigli 2007) and are particularly useful to train and evaluate automatic text simplification systems (Specia et al. 2012; Cholakov et al. 2014). As for multiword expressions, the datasets of Nakov (2008); Hendrickx et al. (2013) contain more or less constrained paraphrases for English nominal compounds.

<footnote n="14">¹⁴A disagreement between answers *a* and *b* is weighted $|a - b|$.</footnote>
## 3 Fifty shades of compositionality

In Wilkens et al. (2017), we propose an extension to the *PT-comp* dataset described above with lexical substitutes. During the data collection process of *PT-comp*, subtask 3 consisted in providing 2 or 3 equivalents (synonyms or paraphrases). The goal of this subtask was to guide annotators to think about the meaning of the compound rather than collecting lexical substitutes. Thus, the quality of these annotations was deemed insufficient, and we proceeded to a new data collection with more explicit guidelines for substitutes.

To create the LexSub-NC dataset, 86 volunteer native speakers of Brazilian Portuguese took part in the annotation campaign. All participants were undergraduate and graduate students in computer science and linguistics. Before the annotation, they were required to take a training session in which examples of compounds in sentences were presented along with the expected responses. Participants were asked to first read the same 3 sentences already used in the creation of *PT-comp*, inducing them to think about the meaning of the compound. Moreover, variability due to polysemy is avoided, since the sentences were manually selected so that a single sense of the compound is represented.<fnref n="15" />

Annotators were asked to provide 3 to 5 substitutes per compound, preferably single words. A minimum of three substitutes was required to allow for a greater diversity of answers.<fnref n="16" /> The annotation interface showed each compound on a separate screen. We estimate that each compound took 1-3 minutes to annotate.

A total of 5,546 responses were collected for the 180 target compounds, with 3,715 unique responses, which were manually verified by a linguist. From these, any response that could not be considered a substitute for the compound was considered invalid, including:

* opinions or judgments about the compound, e.g. [pt país conivente com falcatruas](https://www.google.com/search?q=país+conivente+com+falcatruas) 'country that indulges scams' for [pt paraiso fiscal](https://www.google.com/search?q=paraiso+fiscal) (lit. 'paradise fiscal') 'tax haven';

* semantically related but distinct concepts, e.g. [pt binóculo](https://www.google.com/search?q=binóculo) 'binoculars' for [pt olho mágico](https://www.google.com/search?q=olho+mágico) (lit. 'eye magic') 'peephole');

* tentative explanations, e.g. [pt recipiente de presente secreto](https://www.google.com/search?q=recipiente+de+presente+secreto) 'recipient of secret gift' for **amigo secreto** (lit. 'friend secret') 'secret Santa'.

<footnote n="15">¹⁵Polysemous compounds are rare but do exist, for example, [pt braço direito](https://www.google.com/search?q=braço+direito) (lit. 'arm right') 'reliable assistant' can also be used literally as a body part.</footnote>

<footnote n="16">¹⁶Annotators complained that sometimes it is hard to find more than one substitute. This may explain the large proportion of invalid responses that had to be filtered out.</footnote>
The resulting 3,298 valid responses were aggregated per compound, keeping track of the histogram of substitutes per compound. That is, for each compound, we record the list of unique substitutes and the number of annotators who proposed each substitute. This led to a set of 1,602 substitutes for our set of 180 compounds, with about 8.9 substitutes per compound on average. Each substitute was then manually classified by the expert into the following categories:

*   **Synonyms:** interchangeable equivalents, distinguishing:

- Single-word synonyms (*Syn*<sub>word</sub>) like [pt microchip] for [pt circuito integrado] (lit. 'circuit integrated') 'integrated circuit';

- Multiword synonyms (*Syn*<sub>MWE</sub>) such as [pt pronto-atendimento] 'urgent care' for [pt pronto-socorro] (lit. 'ready help') 'emergency services';

*   **Near synonyms:** semantically related phrases, such as hypernyms, meronyms, and hyponyms, distinguishing:

- Single-word near synonyms (*NearSyn*<sub>word</sub>), like [pt comida] 'food' for [pt batata-doce] (lit. 'potato sweet') 'sweet potato';

- Multiword near synonyms (*NearSyn*<sub>MWE</sub>), like [pt carne de peixe] 'fish meat' for [pt carne branca] (lit. 'meat white') 'white meat';

- **Head** of the compound, as in [pt vinho] 'wine' for [pt vinho branco] 'white wine';

- **Modifier** of the compound, as in [pt doce] 'sweet' for [pt algodão-doce] (lit. 'cotton sweet') 'cotton candy';

*   **Paraphrases:** rewrites as descriptive phrases, such as [pt arma que não é de fogo] 'weapon that is not a firearm' for [pt arma branca] (lit. 'weapon white') 'white weapon';

*   **Definitions:** dictionary-like explanations, such as [pt passagem de ano] 'passage from one year to another' for [pt ano-novo] (lit. 'year new') 'new year'.

Table 3.4 displays the number of total and unique responses per category, along with the number of target compounds that received responses in each category.<fnref n="17" /> In Wilkens et al. (2017), we examine the impact of frequency, con-

<footnote n="17">¹⁷The numbers are slightly different here with respect to the original publication, as they were updated based on the final released dataset.</footnote>
3 Fifty shades of compositionality

ventionality and compositionality on the number and diversity of responses collected for the construction of the dataset. The full resource is publicly available at https://doi.org/10.5281/zenodo.8296689.

<table><caption>Table 3.4: Total and unique (per compound) invalid and valid responses. Valid classified according to their semantic relation to the target compounds. Unique responses are aggregated by compound.</caption><thead><tr><th></th><th># Total Responses</th><th># Unique Responses</th><th># Target Compounds</th></tr></thead><tbody><tr><td>Invalid</td><td>2,248</td><td>2,113</td><td>180</td></tr><tr><td>Valid</td><td>3,298</td><td>1,602</td><td>180</td></tr><tr><td>→ Syn<sub>word</sub></td><td>966</td><td>318</td><td>110</td></tr><tr><td>→ Syn<sub>MWE</sub></td><td>1,257</td><td>684</td><td>164</td></tr><tr><td>→ Head</td><td>232</td><td>56</td><td>56</td></tr><tr><td>→ Modifier</td><td>5</td><td>2</td><td>2</td></tr><tr><td>→ NearSyn<sub>word</sub></td><td>315</td><td>150</td><td>73</td></tr><tr><td>→ NearSyn<sub>MWE</sub></td><td>303</td><td>183</td><td>78</td></tr><tr><td>→ Paraphrases</td><td>54</td><td>47</td><td>32</td></tr><tr><td>→ Definitions</td><td>166</td><td>162</td><td>87</td></tr></tbody></table>

**Token compositionality** As mentioned in §3.2.1, the compositionality datasets described were also extended and used in other works<fnref n="18" /> Garcia et al. (2021b) extend the *EN-comp* and *PT-comp* datasets with neutral context sentences, asking to what extent the context was helpful to predict type-level compositionality scores using pre-trained language models like BERT. Garcia et al. (2021a) collected new compositionality judgments at the token level for sentences containing the same compounds in *EN-comp* and *PT-comp*, extending our previous work on type-based compositionality prediction to token-based prediction. Finally, Madabushi et al. (2021) distinguish idiomatic senses and collect a larger number of sentences per sense for *EN-comp* and *PT-comp*. This dataset was then later cleaned and resplit, and Galician compounds and sentences were added for the SemEval 2022 task 2 challenge on multilingual idiomaticity detection and sentence embedding (Madabushi et al. 2022).<fnref n="19" />

<footnote n="18">¹⁸These works are not co-authored by me, but by some of my co-authors.</footnote>

<footnote n="19">¹⁹https://sites.google.com/view/semeval2022task2-idiomaticity</footnote>
3.2.4 Fine-grained annotation of literal occurrences

The studies summarised in §3.2.3 cover type-level compositionality modelled as
numerical scores. The present section will present a token-level compositionality
resource created in the framework of the PARSEME corpus annotation initiative
(detailed in Chapter 4). In contrast with the type-level resource presented above,
we will now turn to verbal expressions, and model their compositionality using
a fine-grained set of labels. I will summarise here the work presented in Savary,
Cordeiro, Lichte, et al. (2019) for five languages. The goal of this work was to
quantify literal occurrences of potentially idiomatic expressions. Our hypothesis
was that, although the literal/idiomatic ambiguity is widely studied and often
mentioned as an important MWE challenge, truly ambiguous MWEs are not ex-
tremely frequent in corpora.

**Definitions** The article's formalisation relies on the notion of syntactic dependency graph of Universal Dependencies (de Marneffe et al. 2021). In short, an MWE occurrence is defined as a subsequence of tokens in a sentence, with a corresponding dependency subgraph called the **coarse syntactic strucure** (CSS). The lexemes (lemmas and POS tags) of the MWE's lexicalised components are the nodes in the CSS graph, and the dependency relations are the edges. A set of MWE occurrences whose CSS subgraphs share the same nodes (multiset of lexemes) are defined as a **MWE type**.<fnref n="20" /> An MWE's canonical form is an occurrence which is syntactically least marked (e.g. active voice, finite-form verb, singular noun). The CSS of an MWE's canonical forms is its **canonical strucure**. Hence, an **idiomatic occurrence** (IO) of an MWE is one which has an idiomatic meaning (in the sense of definition 2.3), as illustrated in example 1.

(1) Lina não **deu** **no** **pé** ontem. (pt)
Lina not gave in+the foot yesterday.
Lina did not give in the foot yesterday. ‘Lina didn’t escape yesterday.’

The `css(p) = ⟨Vcss(p), Ecss(p)⟩` of this example `p` is composed of the nodes
`Vcss(p) = {⟨dar, VERB⟩, ⟨em, ADP⟩, ⟨O, DET⟩, ⟨pé, NOUN⟩}`. The edges of the CSS are
the set of dependencies `Ecss(p)` represented in Figure 3.3 and this is also the canon-
ical structure of this MWE type, that is, the CSS of one of its least marked forms.

A **literal occurrence** (LO) shares the same multi-set of lexemes of an IO’s MWE
type, but it cannot have this idiomatic meaning. Importantly, an LO can always
be rephrased to have the same canonical structure as an IO. Example 2 below

<footnote n="20">²⁰Notice that an MWE type is a set of MWE occurrences grouped according to shared CSS nodes.</footnote>
### 3 Fifty shades of compositionality

<figure><img src="image_11.png" /><figcaption>Figure 3.3: Coarse syntactic structure of the IO in example 1.</figcaption></figure>

illustrates this when the CSS of the LO is identical to that of the IO. This would not be the case, for instance, if the verb was in passive voice, but there would be a rephrasing (active voice) which keeps the occurrence's meaning and has the same canonical structure as the IO.

(2) *Deu bolha no meu pé.* (pt)
Gave blister in+the my foot.

A blister gave on my foot. 'I got a blister on my foot.'

Finally, a **coincidental occurrence** (CO) is used for co-occurrences of the same multi-set of lexemes, but which cannot be rephrased into an equivalent formulation which matches the canonical structure of the MWE. In other words, in a CO, the lexemes co-occur by chance, but are neither IOs nor LOs because there is no way to rephrase the sentence and find a CSS equivalent to the MWE's canonical structure, as illustrated in example 3, where the preposition and the determiner cannot be attached to the noun *pé* 'foot'.

(3) *Lina dá pé no lago.* (pt)
Lina gives foot in+the lake.

Lina gives foot in the lake. 'Lina can touch the bottom of the lake.'

**Candidate extraction** In practice, the annotation of IOs was performed separately from the annotation of LOs and COs. Our starting point for IOs were the PARSEME corpora of Basque, German, Greek, Polish and Brazilian Portuguese, manually annotated for verbal MWEs in the PARSEME shared task 1.1 (Ramisch, Cordeiro, et al. 2018). The annotation guidelines for IOs will be presented in Chapter 4.

For each language, we automatically extract a list MWE types corresponding to the IOs. Given our definitions above, it should be straightforward to locate and distinguish LOs from COs using syntactic constraints that match the CSS of the gold MWE types against corpus occurrences of the same lexemes. However, most
corpora were automatically parsed, so that the quality of the predicted lemmas, POS tags and dependency relations cannot be trusted 100% correct. Thus, we implemented several heuristics to locate potential LOs and COs to be annotated given an MWE type:

* **WindowGap** returns all occurrences of the same lexemes co-occurring in a fixed-length window in any order. At most *g* = 2 “gap” elements can appear between the occurrence of the first and last matched lexemes.

* **BagOfDeps** returns all occurrence of the same lexemes when they appear in a weakly connected unlabeled subgraph, regardless of the edge directions or labels, and ignoring the MWE type's CSS.

* **UnlabeledDeps** adds a constraint to *BagOfDeps*: the dependency labels are still ignored, but the directions of the dependencies (parent nodes) must be the same as in the CSS of one of the corresponding IOs.

* **LabeledDeps** is the most restrictive heuristic, requiring that the CSS of the candidate is identical to the CSS of one of the IOs, including both the labels and directions of dependencies.

**Annotation** Each LO candidate retrieved by one of the heuristics is assigned a single label among the 9 labels below. The label set covers not only the target phenomena (LOs and COs of MWEs) but also errors due to the original annotation or to the automatic candidate extraction methodology:<fnref n="21" />

1. **IDIOMATIC**: This label is trivially assigned to all annotated MWEs.

2. **ERR-FALSE-IDIOMATIC**: LO candidates that should not have been retrieved, but were found due to a spurious MWE annotation in the original corpus.

3. **ERR-SKIPPED-IDIOMATIC**: candidates that should have been initially annotated as IOs in the corpus, but were not.

4. **NONVERBAL-IDIOMATIC**: candidates that are MWEs, but not verbal, and are thus out of scope.

5. **MISSING-CONTEXT**: more context (e.g. previous/next sentences) would be required to annotate the candidate.

<footnote n="21">²¹Although English is not part of this study, examples were taken from the PARSEME 1.1 English corpus.</footnote>
### 3 Fifty shades of compositionality

6. **WRONG-LEXEMES**: The candidate should not have been extracted, because the lemmas or POS are not the same as in an IO (errors in the corpus' morphosyntactic annotation, or in the candidate extraction method).

* *Coincidental* and *literal* occurrences are our focus. In the latter case, we also wish to check if an LO might be automatically distinguished from an IO, given additional information provided e.g. in MWE lexicons.

7. **COINCIDENTAL**: the candidate contains the correct lexemes (i.e., lemmas and POS), but the dependencies are not the same as in the IO.

- The lexemes *do the job* 'to achieve the required result' co-occur in *why you like the job and do a little bit [...]*, but they do not form (and are not rephrasable to) a connected dependency tree.

8. **LITERAL-MORPH**: the candidate is indeed an LO that could be distinguished from an IO by checking morphological constraints.

- The MWE *get going* 'continue' requires a gerund *going*, which does not occur in *At least you get to go to Florida*

9. **LITERAL-SYNT**: the candidate is indeed an LO that could be distinguished from an IO by checking syntactic constraints.

- The MWE *to have something to do with* selects the preposition *with*, which does not occur in [...] *we have better things to do*.

10. **LITERAL-OTHER**: the candidate is indeed an LO that could be distinguished from an IO only by checking more elaborate constraints (semantic, contextual, extra-linguistic constraints).

- *we've come out of it good friends* is an LO of the MWE *to come of it* 'to result', but it is unclear what kind constraint could distinguish it from an IO.

**Results** The annotation categories above were detailed in the annotation guidelines, which were in turn used to annotate the LO and CO candidates returned by the heuristics. Table 3.5 summarises the results of this annotation for the 5 languages, including the total number of IOs and of annotated LO candidates, and the distribution of the labels.

The main take-home message of this work can be summarised in the last row of Table 3.5. The *idiomaticity rate* is the proportion of IOs with respect to all IOs and LOs $IR = \frac{\#IOs}{\#IOs+\#LOs}$, where #IOs includes the skipped idiomatic category,
<table><caption>Table 3.5: General statistics of the annotation results.</caption><thead><tr><th></th><th>German</th><th>Greek</th><th>Basque</th><th>Polish</th><th>Portug.</th></tr></thead><tbody><tr><td>Annotated IOs</td><td>3,823</td><td>2,405</td><td>3,823</td><td>4,843</td><td>5,536</td></tr><tr><td>LO candidates</td><td>926</td><td>451</td><td>2,618</td><td>332</td><td>1,997</td></tr><tr><td>ERR-FALSE-IDIOMATIC</td><td>21.5%(199)</td><td>12.0% (54)</td><td>9.4% (246)</td><td>0.0% (0)</td><td>3.8% (76)</td></tr><tr><td>ERR-SKIPPED-IDIOMATIC</td><td>27.0%(250)</td><td>47.5% (214)</td><td>17.3% (453)</td><td>5.4% (18)</td><td>10.7% (213)</td></tr><tr><td>NONVERBAL-IDIOMATIC</td><td>0.0% (0)</td><td>0.0% (0)</td><td>0.2% (6)</td><td>0.0% (0)</td><td>0.5% (9)</td></tr><tr><td>MISSING-CONTEXT</td><td>0.3% (3)</td><td>0.2% (1)</td><td>0.5% (12)</td><td>2.1% (7)</td><td>0.7% (13)</td></tr><tr><td>WRONG-LEXEMES</td><td>40.1%(371)</td><td>0.9% (4)</td><td>26.7% (700)</td><td>1.8% (6)</td><td>38.1% (760)</td></tr><tr><td>COINCIDENTAL (COs)</td><td>2.6% (24)</td><td>27.9% (126)</td><td>42.4% (1110)</td><td>61.1% (203)</td><td>33.5% (668)</td></tr><tr><td>LITERAL (LOs)</td><td>8.5% (79)</td><td>11.5% (52)</td><td>3.5% (91)</td><td>29.5% (98)</td><td>12.9% (258)</td></tr><tr><td>→ LITERAL-MORPH</td><td>0.8% (7)</td><td>5.5% (25)</td><td>1.9% (51)</td><td>1.2% (4)</td><td>3.7% (73)</td></tr><tr><td>→ LITERAL-SYNT</td><td>1.5% (14)</td><td>2.0% (9)</td><td>0.7% (19)</td><td>8.1% (27)</td><td>2.2% (44)</td></tr><tr><td>→ LITERAL-OTHER</td><td>6.3% (58)</td><td>4.0% (18)</td><td>0.8% (21)</td><td>20.2% (67)</td><td>7.1% (141)</td></tr><tr><td>Idiomaticity rate</td><td>98%</td><td>98%</td><td>98%</td><td>98%</td><td>96%</td></tr></tbody></table>

e.g. $\frac{3823+250}{3823+250+79}$ for German. The proportion of LOs ranges from 2% to 4%, confirming our initial hypothesis that true literal readings of MWEs are rather rare, at least for verbal MWEs in these 5 languages. The vast majority of all other LO candidates are actually COs and errors due to automatic corpus processing. Similar conclusions have been reached in other studies, e.g. truly ambiguous idioms are rare in the MAGPIE corpus (Haagsma et al. 2020) and the “skewed” part of the VNC-Tokens corpus is the largest one, containing MWEs that exhibit a strong tendency towards one of the interpretations (idiomatic or literal).

It would be straightforward to apply simple, interpretable syntactic and morphological rules to identify IOs, if high-quality parse trees and MWE lexicons were available. We assume that such MWE identification method could reach high precision, given the low rate of LOs in corpora. This finding motivated the design of the Seen2Seen system (Pasquer, Savary, Ramisch, et al. 2020a), which obtained competitive results in edition 1.2 of the PARSEME shared task.

Additional analyses in the article have shown that it is often a very limited number of types that are truly ambiguous. These MWEs in this selected group of “troublemakers” often contains a highly ambiguous lexeme, or has some specific morphological inflection or order constraint (not captured by a CSS). The paper details the linguistic phenomena often at the root of LOs depending on the verbal MWE category and on the language characteristics. In Basque, for example, the lack of morphological information in the CSS resulted in a very large num-
ber of candidates to annotate, but many of them are COs or correspond to the WRONG-LEXEMES due to lemmatisation errors. In Portuguese it was the reflexive clitic *pt se* 'self' which was responsible for many WRONG-LEXEMES because it is homonymous with the conjunction *pt se* 'if'. For more detailed analyses, we suggest reading Sections 6 to 9 of Savary, Cordeiro, Lichte, et al. (2019).

This token-level corpus annotated for compositionality is freely available at http://hdl.handle.net/11372/LRT-2966. While the corpus was studied for the linguistic characteristics of LOs, it was not yet explored for automatic compositionality prediction. Thus, it constitutes a very rich resource for future research in token-based compositionality prediction.

## 3.3 Methods and evaluation

Now that we have discussed compositionality related resources, it is time to turn to computational models that, given a candidate MWE, can predict whether (or to what extent) it is more compositional or more idiomatic. In other words, we would like to develop computational models able to predict a score that quantifies what proportion of the meaning of the whole phrase can be transparently inferred from its components and structure. There is a rich and varied literature on compositionality prediction, so we start with a broad and up-to-date review of existing methods (§3.3.1). These methods are usually trained and/or evaluated on resources such as those described in §3.2. Our proposed methods are unsupervised, so the gold compositionality annotations present in datasets are used only to assess the system, and not to train nor tune it (§3.3.2). We focus on type-level prediction, that is, the MWE is presented to the model without taking into consideration the context in which the MWEs occur. Our algorithms rely on word embedding representations and operations, so we evaluate and compare their characteristics in this task (§3.3.3). We conclude with a list of open issues and some suggestions on how to address them (§3.3.4).

### 3.3.1 Existing methods

In spite of the dominance of supervised methods in other fields of computational linguistics, most methods proposed in the literature do not rely on supervision (§3.3.1.1). This is probably due to the little amount of supervision available, and to the fact that compositionality can also be used to assess the quality and generalisation of distributional language models. Nonetheless, there have been some attempts to learn compositionality functions from data (§3.3.1.2).
### 3.3.1.1 Unsupervised methods

Unsupervised methods rely on several sources of information: lexical resources, fixedness, translation, and distributional similarity.

**Lexical resources** Compositional expressions can usually be paraphrases in terms of their component words. Thus, if we have a dictionary containing definitions of more compositional and more idiomatic phrases, it is likely that the component words of the phrase will appear more often as part of the definition of compositional expressions rather than in the definitions of idiomatic MWEs. This idea was evaluated by Salehi et al. (2014) using the English Wiktionary. The authors also enriched their method by exploiting other relations present in the lexicon, such as synonymy and translations. Similarly, one of the methods proposed by Nandakumar et al. (2019) uses paraphrases instead of definitions, with a similar metric: the overlap between MWE components and paraphrases provided in the *EN-comp* dataset (§3.2.3).

**Fixedness** The extent to which MWE candidates allow for lexical replacement of its components (Pearce 2001), of the whole expression (Riedl & Biemann 2015), or syntactic alternations (Villavicencio et al. 2007; Bannard 2007) has been used to automatically discover new MWEs. However, one of the first unsupervised methods to perform compositionality prediction was the method proposed by Fazly et al. (2009). Their hypothesis is that idiomatic instance of verb-noun constructions in English will be more fixed than their literal counterparts in terms of the number of the noun (singular vs. plural), the presence of a determiner, etc. The variability of the instances across these dimensions allows the authors to perform type-level compositionality prediction. Then, they adapt these measures to identify the canonical form of the MWE and classify token instances as idiomatic if they are in the canonical form, and compositional otherwise. These methods are still frequently employed as a competitive baselines for this task.

**Translation** Idiomatic expressions tend to have translations that are more distant of the translations of their component words. This has been explored by several MWE discovery methods using parallel corpora (Caseli et al. 2010; Tsvetkov & Wintner 2012) and other resources (Attia et al. 2010). However, Salehi & Cook (2013) are the first to propose an approach for compositionality prediction using bilingual lexicons. They assume that the string similarity between a translation of a compositional combination and the translations of its components should be higher than for idiomatic MWEs. This is similar to distributional methods, but