uses translations instead of embeddings and string similarity instead of cosine. The method combines cues from translations into several languages using Pan-Lex. The set of optimal languages is tuned in a supervised manner. Alternatively, Salehi et al. (2016) propose an original method based on cross-lingual transfer. They first train a delexicalised parser on a corpus containing MWE-specific dependencies. Then, this parser is used to parse a new language, and the predicted MWE dependency relations are evaluated as new MWEs in the target language.

**Distributional methods** Schone & Jurafsky (2001) investigate several MWE discovery techniques. Among their proposals, they suggest using latent semantic analysis (LSA) to encode the meaning of *n*-grams and their components. Then, they calculate the cosine similarity between the LSA vectors of the *n*-gram and the sum of the LSA vectors of the components. The results that they obtain for English are disappointing, but the method was evaluated again by Katz & Giesbrecht (2006), now showing interesting results for type-level compositionality detection in German. This simple idea is still used in many distributional compositionality prediction methods, including the $pc_β$ score that we propose in §3.3.2.

The measures proposed by McCarthy et al. (2003) rely on the intersection between the sets of neighbours in a distributional thesaurus. They compare the set of neighbours of the whole verb-particle construction with the set of synonyms of the verb, assuming that more compositional constructions would have more neighbours in common with the verb. They compare several ways to assess this intersection and compare this with frequency, association measures, and WordNet. Baldwin et al. (2003) also analyse to what extent the distributional neighbours of MWEs and their components overlap, but they use LSA as their underlying distributional model. Their hypothesis is the same: if MWEs are more idiomatic, their neighbours should not be similar to those of their components, as their meanings are less related than for compositional expressions.

im Walde et al. (2013) presents a slightly different method to predict the compositionality of German noun-noun compounds. Instead of adding the combining embeddings before calculating the cosine similarity, they first calculate the similarity between the embedding of the compound and the embeddings of its components individually. Then, they add or multiply these scores to obtain a comspositionality score for the whole phrase. This method inspired our $pc_{arth}$ and $pc_{geom}$ scores below.

Salehi et al. (2015) are the first to replace LSA by word embeddings in type-level compositionality prediction. They show that word2vec and the multi-prototype model MSSG obtain better results than count-based distributional models, but
there seems to be no clear advantage in using multi-prototype embeddings rather
than simple static word2vec embeddings. Nandakumar et al. (2019) evaluate a
much wider range of both static and contextual embeddings on 3 English type-
level datasets. Surprisingly, the best model is word2vec rather than more recent
contextual models like ELMO and BERT. Moreover, they show that the combina-
tion weights of the words in the MWE have different optimal values depending
on the underlying embedding model.

A promising alternative to word embedding models are character-level mod-
els, which can create on-the-fly representations for phrases, not requiring token-
level preprocessing to learn MWE embeddings (Parizi & Cook 2018). As for token-
level prediction, methods relying on word embeddings have also been proposed,
both supervised and unsupervised (Gharbieh et al. 2016). Multi-lingual contextu-
alised embeddings are also potentially interesting for token-level compositional-
ity prediction (Fakharian & Cook 2021).

3.3.1.2 Supervised methods

Hashimoto & Kawahara (2008) use an SVM to learn to distinguish literal from idiomatic instances in the Japanese OpenMWE corpus, obtaining impressive accuracies close to 0.9. Fothergill & Baldwin (2011) extend this work by proposing new features and evaluating the model when trained for cross-type classification (the MWE types in the test set do not appear in the training set), also obtaining impressive results. The method proposed by Diab & Bhutada (2009) relies on a sequence SVM classifier that uses a BIO encoding specialised to detect literal and idiomatic verb-noun constructions in English. In addition to traditional features like left and right words, lemmas and character n-grams, they also use named entity placeholders which help reducing sparsity while keeping relevant information, for example, that a co-occurring context is a person (human). Some of the participant systems of the DisCo shared task employ classifiers to predict the compositionality of English and German nominal expressions Biemann & Giesbrecht (2011). The features given to these classifiers vary, from association measures to distributional similarity. However, the shared task results do not show a clear advantage of supervised methods over unsupervised ones.

Muzny & Zettlemoyer (2013) use a supervised classifier to detect whether a
given phrase in Wiktionary refers to an idiom or not. They explore a set of fea-
tures such as the overlap between synonyms or hyponyms of the phrase compo-
nents and words in the phrase’s definition, both in Wiktionary and in Wordnet.
Then, they combine their method with a simple WSD method and show that they
### 3 Fifty shades of compositionality

can reliably identify instances of the target phrases and disambiguate between idiomatic and literal occurrences.

Salton et al. (2016) use skip-thought embeddings to encode the literal and idiomatic instances of the VNC-Tokens dataset. Then, they learn K-NN and SVM classifiers that learn to predict whether a given sentence encoded with skip-thought corresponds to a literal or idiomatic use of an MWE. King & Cook (2018) obtain better results using averaged word embeddings instead of skip-thought, and combine them with the unsupervised features proposed by Fazly et al. (2009), obtaining significant improvements on the VNC-Tokens dataset. Shwartz (2019), on the other hand, evaluate word2vec, fastText, GloVe, ELMO, GPT, and BERT on 6 MWE tasks including 3 compositionality datasets. They find out that not only contextualised models outperform static ones, but they also encode the difference between literal and idiomatic occurrences.

Yazdani et al. (2015) explore several functions to project and combine word embeddings, optimising to on a task that consists in approximating the embeddings learned for a large set of (compositional) held-out phrases. They show that the learned functions can effectively model compositional combinations. A recent review of supervised methods for compositionality prediction can be found in the SemEval 2022 task 2 paper (Madabushi et al. 2022) and in the system description papers corresponding to this task.

#### 3.3.2 Compositionality prediction

The proposed method for compositionality prediction is a generalisation of the methods initially proposed by Schone & Jurafsky (2001) and Baldwin et al. (2003). While their methods relied on latent semantic analysis to model distributional similarity, we assume that any embedding model able to create a (dense or sparse) vector representation for words and MWE candidates can be used to predict compositionality. Given an MWE candidate $w_1 ... w_n$ with n component words, our method relies on four elements, as shown in Figure 3.4:

1. the embeddings of the MWE candidate $v(w_1 ... w_n)$ and of its components,

2. the parametrised function that builds a combined embedding $v_β(w_1, ..., w_n)$ from the embeddings of the MWE components $v(w_1)$ to $v(w_n)$,

3. the comparison function that predicts the compositionality $pc_β(w_1 ... w_n)$ as the similarity between the embedding of the whole MWE candidate $v(w_1 ... w_n)$ and the combined embedding $v_β(w_1, ..., w_n)$, and

4. the evaluation metrics that compare the predicted scores $pc_β(w_1 ... w_n)$ and the gold scores provided by human annotators in the datasets.
<figure><img src="image_12.png" /><figcaption>Figure 3.4: Compositionality prediction: compare MWE embedding with combined component embeddings. Source: Cordeiro, Ramisch & Villavicencio (2016a)</figcaption></figure>

**Embeddings** First, we need an embedding of the whole MWE that captures the distributional co-occurrence patterns of the expression’s use in text. In our framework, this is obtained by corpus pre-processing prior to embedding creation. The list of MWEs for which we want to predict a compositionality score is first matched against the corpus using a MWE identification method (Chapter 4. In our experiments, we focus on 2-word nominal compounds, so this identification method is simple: we locate all occurrences of adjacent lexemes (lemma+POS) matching a compound in our target MWE list.<fnref n="22" /> Once the occurrences are located, we join them into a single token using an underscore (e.g. *ivory tower becomes ivory_tower*). After concatenating the MWE component occurrences, the preprocessed corpus is given to embedding learning software such as word2vec (Mikolov et al. 2013) and GloVe (Pennington et al. 2014), which will treat the MWEs as if they were single words, and generate embeddings for them as well as for their component words when they appear in other contexts.<fnref n="23" />

**Combination function** The combination function creates an artificial embedding $v_{\beta}(w_1, ..., w_n)$ from the individual embeddings of the components of the MWE candidate. Several combination functions exist, but after preliminary tests we decided to focus on the additive model (Mitchell & Lapata 2008), also known

<footnote n="22">²²In practice, we lookup in the MWE list for exact matches of all bigrams in the corpus.</footnote>

<footnote n="23">²³This might be a problem for compositional expressions, since their contexts of occurrence within the MWE are not taken into account as distributional features to build their embeddings.</footnote>
### 3 Fifty shades of compositionality

as continuous bag-of-words (Mikolov et al. 2013). Thus, the composition function is simply a weighted linear combination:

$$v_{\beta}(w_1 \dots w_n) = \sum_{i=1}^{n} \beta_i \frac{v(w_i)}{\|v(w_i)\|}$$

where $\|\cdot\|$ is the euclidean norm and $\beta_i$ are the weights of each component of the MWE to its meaning. The normalisation of the embeddings allows taking only their directions into account, regardless of their norms, which may be proportional to their frequency in corpora, and less relevant to their meanings. Normalisation can be disabled if, for any reason, embedding norms are relevant.

For the special case of 2-word MWE candidates treated here, the combination function can be reformualated as follows:

$$v_{\beta}(w_1 w_2) = \beta \frac{v(w_{\text{head}})}{\|v(w_{\text{head}})\|} + (1 - \beta) \frac{v(w_{\text{mod}})}{\|v(w_{\text{mod}})\|}$$

where $w_{\text{head}}$ and $w_{\text{mod}}$ are the head and modifier of the compound, and $\beta \in [0, 1]$ controls the relative importance of the head to the compound's compositionally constructed embedding. This formulation enables testing our hypotheses about the contribution of the components, instantiated in variants of the combination function used to predict the compositionality scores $pc_{\beta}(w_1 w_2)$:

1. $pc_{\text{head}}(w_1 w_2)$: with $\beta = 1$, the meaning of the whole compound depends on the meaning of its syntactic head (e.g. *crocodile tears* 'simulated *tears*');

2. $pc_{\text{mod}}(w_1 w_2)$: with $\beta = 0$, the meaning of the whole compound depends on the meaning of its modifier (e.g. *busy bee* 'busy person');

3. $pc_{\text{uniform}}(w_1 w_2)$: with $\beta = \frac{1}{2}$, the meaning of the whole compound depends in equal measure on the head and modifier (e.g. *graduate student*);

4. $pc_{\text{maxsim}}(w_1 w_2)$: assumes that the value of $\beta$ can be set individually for each compound, that is: $pc_{\text{maxsim}}(w_1 w_2) = \max_{0 \le \beta \le 1} pc_{\beta}(w_1 w_2)^{24}$

5. $pc_{\text{arith}}(w_1 w_2)$: is the arithmetic mean of $pc_{\text{head}}(w_1 w_2)$ and $pc_{\text{mod}}(w_1 w_2)$; and

6. $pc_{\text{geom}}(w_1 w_2)$: is the geometric mean of $pc_{\text{head}}(w_1 w_2)$ and $pc_{\text{mod}}(w_1 w_2)$, reflecting the tendency that humans have to assign a $hc_{\text{HM}}$ score to the compound closer to the lowest score between $hc_H$ and $hc_M$.

<footnote n="24">²⁴For two words, we do not need to perform parameter search for $\beta$, which has a closed form obtained by solving $\frac{\partial}{\partial \beta} pc_{\beta}(w_1 w_2) = 0$: $\beta = \frac{\cos(w_1 w_2, w_1) - \cos(w_1 w_2, w_2) \times \cos(w_1, w_2)}{(\cos(w_1 w_2, w_1) + \cos(w_1 w_2, w_2)) \times (1 - \cos(w_1, w_2))}$.</footnote>
**Comparison function** The mainstream approach to compare embeddings is the cosine similarity function, which is the normalised dot-product of the vectors. We adopt it to compare the compositionally constructed embedding $v_β(w_1, ..., w_n)$ with the embedding of the whole MWE candidate $v(w_1 ... w_n)$, yielding our predicted compositionality score:

$$ 
\begin{aligned}
\mathrm{pc}_{\beta}(w_1 \dots w_n) &= \cos(v(w_1 \dots w_n), v_{\beta}(w_1, \dots, w_n)) \\
&= \frac{v(w_1 \dots w_n) \cdot v_{\beta}(w_1, \dots, w_n)}{\|v(w_1 \dots w_n)\| \times \|v_{\beta}(w_1, \dots, w_n)\|}
\end{aligned}
$$

**Evaluation metrics** In our experiments on nominal compounds, we calculate Spearman's $\rho$ rank correlation between the predicted compositionality scores $pc_β(w_1, w_2)$ and the human compositionality score $hc_{HM}$ for the compounds that appear in the evaluation datasets *EN-comp*, *FR-comp*, and *PT-comp*. We use the rank correlation instead of linear correlation (Pearson), because we are interested in the framework's ability to order compounds from least to most compositional, regardless of the actual predicted values. For English, besides the evaluation datasets presented in §3.2.3, we also use the datasets of Reddy et al. (2011) (henceforth *Reddy*) and Farahmand et al. (2015) (henceforth *Farahmand*), to enable comparison with related work. For *Farahmand*, since it contains binary judgments instead of graded compositionality scores, results are reported using the best $F_1$ ($BF_1$) score, which is the highest $F_1$ score found using the top $n$ compounds classified as non-compositional, when $n$ is varied (Yazdani et al. 2015).<fnref n="25" />

The framework is summarised in 3.5. The creation of the embeddings of the MWE and of its components depends on the underlying embedding model. The combination and comparison functions were implemented and are freely available in the *mwetoolkit's* script *feat_compositionality.py* (Cordeiro, Ramisch & Villavicencio 2016a). The evaluation metrics are available in the *csv_eval-rank.py* script, released as part of *minimantics*, which also implements the PPMI models (§3.3.3).<fnref n="26" />

### 3.3.3 Experiments and results

The framework presented in §3.3.2 relies on existing models to generate word and MWE embeddings. All models were learnt from the same corpora, containing about 2 billion tokens each: ukWaC for English (Baroni et al. 2009), frWaC

<footnote n="25">²⁵In *Farahmand*, idiomatic compounds are those annotated so by at least 2 out of 4 annotators.</footnote>

<footnote n="26">²⁶https://github.com/ceramisch/minimantics</footnote>
### 3 Fifty shades of compositionality

<figure><img src="image_13.png" /><figcaption>Figure 3.5: Compositionality prediction framework. Thick arrows: corpus-based embeddings of compounds treated as a single token. The schema also covers the evaluation of compositionality prediction (top right). Adapted from Cordeiro et al. (2019)</figcaption></figure>

for French (Ferraresi et al. 2010) and brWaC for Portuguese (Filho et al. 2018). The corpora were pre-processed (tokenisation, lemmatisation, POS tagging), and all occurrences of the compounds in our evaluation datasets were concatenated prior to generating embeddings.

**Embedding models** In our experiments, we evaluate 7 embedding models: 3 of them are correspond to different dimensionality reduction techniques applied to the *co-occurrence counts* matrix, while the other 4 rely on directly optimising a loss function using gradient descent on a word-context *prediction* task:

1. PPMI-SVD first generates a matrix with co-occurrence counts of words within a sliding window. Each cell of this matrix represents the association strength between a word and its contexts through the positive point-wise mutual information (PPMI) score. Then, singular value decomposition (SVD) is applied to the PPMI matrix, keeping a fixed-size number of dimensions per embedding (Dinu et al. 2013).

2. PPMI-topK represents each target word as a list of its top-k co-occurring contexts sorted by descending PPMI values. When comparing two target-word embeddings, we only look at common co-occurring words shared by both target words (Padró et al. 2014a).

3. PPMI-th is similar to PPMI-topK but we filter out all contexts whose PPMI value falls below a fixed threshold t (Padró et al. 2014a).
4. w2v-sg tries to predict whether a pair of target-context words are likely to co-occur. The model randomly samples negative examples, and takes actual co-occurring pairs as positive examples. The optimisation procedure tries to maximise the dot product of positive target-context pairs (Mikolov et al. 2013).

5. w2v-cbow is similar to w2v-sg, but instead of predicting true/false word-context associations, it tries to predict the target word given the average embeddings of the context words (Mikolov et al. 2013).

6. GloVe is a prediction-based model in which the objective function approximates the log-frequency of co-occurring words via the dot product of their embeddings (Pennington et al. 2014).

7. lexvec is similar to GloVe but uses a weighting scheme that penalises more errors on frequent words (Salle et al. 2016).

Table 3.2 shows the evaluation of compositionality prediction for different em-
bedding models (rows) on the 5 datasets. For the *Farahmand* dataset, the evalu-
ation metric is BF₁, whereas for the other datasets we report the Spearman cor-
relation between pc_uni_form and hc_HM. For each model, we report the best results
obtained across all evaluated hyper-parameter configurations.

By the time of these experiments, there was no consensus as to whether count-
based models or prediction-based models were best in general. While some em-
pirical evaluations indicated that prediction-based models were preferable for
many semantic tasks (Baroni et al. 2014), other results suggested that tricks in
their optimisation were actually responsible for their advantages (Levy et al.
2015), and one of our preliminary studies found out that simple count-based tech-
niques were competitive on similarity tasks (Padró et al. 2014b). It turns out that
nowadays count-based models are not so popular, but the results presented in
Table 3.2 do not allows us to designate a winner. Although we obtained the best
results with a count-based model (PPMI-th) for FR-comp and PT-comp, the En-
glish datasets seem to benefit from prediction-based models (w2v-sg and w2v-
cbow). The results obtained for Reddy still are the state of the art on this dataset
in 2022.<sup>27</sup>

<footnote n="27"><sup>27</sup>Notice that Shwartz (2019) report a score of 0.913 on *Reddy*, but they transform it into a binary classification task and do not predict numerical scores.</footnote>
### 3 Fifty shades of compositionality

<table><caption>Table 3.6: Best results for each embedding model: $BF_1$ for *Farahmand*, Spearman $\rho$ for other datasets. For English, the first value is for the compounds found in the corpus, and the second uses fallback for missing compounds. Source: Cordeiro et al. (2019)</caption><thead><tr><th>Model</th><th>Farahmand</th><th>Reddy</th><th>EN-comp</th><th>FR-comp</th><th>PT-comp</th></tr></thead><tbody><tr><td>PPMI-SVD</td><td>.487/.424</td><td>.743/.743</td><td>.655/.666</td><td>.584</td><td>.530</td></tr><tr><td>PPMI-topK</td><td>.435/.376</td><td>.706/.716</td><td>.624/.632</td><td>.550</td><td>.519</td></tr><tr><td>PPMI-th</td><td>.472/.404</td><td>.791/.803</td><td>.688/.704</td><td><strong>.702</strong></td><td><strong>.602</strong></td></tr><tr><td>w2v-cbow</td><td><strong>.512/.471</strong></td><td>.796/.796</td><td>.716/.730</td><td>.652</td><td>.588</td></tr><tr><td>w2v-sg</td><td>.507/.468</td><td><strong>.812/.812</strong></td><td><strong>.726/.741</strong></td><td>.653</td><td>.586</td></tr><tr><td>GloVe</td><td>.400/.358</td><td>.754/.759</td><td>.638/.651</td><td>.680</td><td>.555</td></tr><tr><td>lexvec</td><td>.449/.431</td><td>.774/.773</td><td>.646/.658</td><td>.677</td><td>.570</td></tr></tbody></table>

**Composition functions** Among the numerous evaluations and comparisons performed in this work, we select one analysis to illustrate the performance of the six composition functions described in §3.3.2. Table 3.7 shows that the best results are obtained with different strategies depending on the language: $pc_{uniform}$ for Portuguese, $pc_{arith}$ for French and $pc_{maxsim}$ for English. These three models have similarly good performances across the three languages and clearly outperform the unbalanced functions that ignore the head or the modifier. There seems to be no advantage in using $pc_{geom}$ instead of $pc_{arith}$. It is disappointing that the $pc_{maxsim}$ model does not systematically outperform the others, as it uses an optimised per-compound $\beta$ value to assess the importance of each word. A deeper analysis shows that, on average, the model tends to put more weight on the head than on the modifier ($\beta$ = .55 in English, $\beta$ = .68 in French and Portuguese). When we visualise the compounds that benefit the most from $pc_{maxsim}$, these tend to be the most compositional ones, whereas the predictions for the most idiomatic ones are actually degraded by this strategy. All in all, the simplest function $pc_{uniform}$ provides very decent predictions, being the best or second-best across all languages, and was therefore adopted in all further analyses.

**Further analyses** In total, this work trained 228 embedding models and evaluated mode than 9,000 configurations to study their impact on compositionality prediction. We perform further analyses in Cordeiro et al. (2019), whose conclusions we can summarise as follows:

* The results of Table 3.2 generalise on cross validation and held-out data.
<table><caption>Table 3.7: Spearman ρ for the proposed combination functions, using the best embeddings for each function. Source: Cordeiro et al. (2019)</caption><thead><tr><th>Dataset</th><th>pc<sub>uniform</sub></th><th>pc<sub>maxsim</sub></th><th>pc<sub>geom</sub></th><th>pc<sub>arith</sub></th><th>pc<sub>head</sub></th><th>pc<sub>mod</sub></th></tr></thead><tbody><tr><td>EN-comp</td><td>.726</td><td><strong>.730</strong></td><td>.677</td><td>.718</td><td>.555</td><td>.677</td></tr><tr><td>FR-comp</td><td>.702</td><td>.693</td><td>.699</td><td><strong>.703</strong></td><td>.617</td><td>.645</td></tr><tr><td>PT-comp</td><td><strong>.602</strong></td><td>.590</td><td>.580</td><td>.598</td><td>.558</td><td>.486</td></tr></tbody></table>

* The context window used to learn the embeddings influences compositionality prediction: the optimal values depend on each embedding model, but we observe a general preference for smaller windows.

* Embeddings with more dimensions tend to predict compositionality better, but the observed differences beyond 750 dimensions to not seem to justify the overhead in learning and storing such large vectors.

* Pre-processing the text to learn lemma embeddings slightly degrades the performances for English and significantly improves the performance for French and Portuguese (which have richer morphology).

* For all datasets, results improve when we learn embeddings on corpora up to 1 billion words and remain stable for larger corpora.

* Predicted compositionality correlates with compound frequency, but does not correlate well with conventionality (measured by PMI), suggesting that these are somewhat orthogonal aspects.

* The most difficult compounds for automatic models are often those for which humans show more disagreement (measured by standard deviation).

* Sanity checks indicate little impact on results for the following parameters: number of iterations in prediction-based models, minimum count threshold in count-based models, dimensions larger than 750, intermediate sliding window sizes, variability due to random initialisation in models that rely on gradient descent, and dataset filtering (§3.2.3.2).

### 3.3.4 Going further

One of the limitations of the work presented here is that the embeddings of MWE candidates are obtained from the concatenation of its component words. This requires a specialised corpus pre-processing procedure that has to be repeated for
any new MWE added to the evaluation dataset. We have studied alternatives to create combined phrase embeddings using auto-encoders in the context of the internship of Yannis Coutouly.<fnref n="28" /> The internship of Lucas Pagès consisted in testing whether the concatenation strategy could be applied to verbal discontinuous expressions, by reordering the expressions before concatenation, but this strategy does not seem to work very well in preliminary experiments. The use of pre-trained language models such as BERT to obtain phrase representations sounds promising but was not yet tried. We have studied the use of predicted compositionality as an additional feature in CRFs for the identification of MWEs, but this did not lead to significant improvement (Scholivet et al. 2018).

## 3.4 In short

Multiword expressions are both interesting and challenging, and can be seen as an opportunity to push the limits of language technology. In this chapter, we have looked at methods that try to automatically discover new multiword expressions from corpora, with a particular focus on distinguishing more compositional from more idiomatic expressions. Our journey started with a brief survey of general methods for MWE discovery, including association measures, substitution-based methods, and cross-lingual methods based on translation asymmetries.

Then, we narrowed down our scope to focus on *compositionality*, that is, the varying degree to which the components of MWEs contribute their meaning to the whole expression. Our survey of existing compositionality datasets started with a chronological review. Early motivation came from the evaluation of traditional MWE discovery methods (Lin 1999), giving rise to the idea that compositionality is a continuum that can be modelled as a real number in a range (McCarthy et al. 2003). Early works focused on compositionality datasets with numerical ratings for English nominal and verbal expressions out of context (type-level) (Venkatapathy & Joshi 2005; Piao et al. 2006). Then, token-level datasets started to appear, with discrete compositionality annotations indicating whether an occurrence was compositional or idiomatic (Cook et al. 2008; Hashimoto & Kawahara 2008; Tu & Roth 2011). The very influential work of Reddy et al. (2011) revived the interest in type-level datasets, and similar works were datasets for other languages such as German (im Walde et al. 2016), French and Portuguese (Ramisch, Cordeiro, Zilio, et al. 2016). More recently, multilingual token-level datasets were created with both numerical scores (Madabushi et al. 2021) and discrete fine-grained annotations (Savary, Cordeiro, Lichte, et al. 2019).

<footnote n="28">²⁸https://github.com/Ounaye/TAL_ApprentissageComposition_NADJ</footnote>
In addition to this high-level overview, we have also looked into the creation
of two datasets: type-level numerical annotations for nominal compounds and
token-level discrete annotations for verbal expressions. For the former, we col-
lected ratings from about 15-30 crowdsourcing workers for about 180 nominal
compounds in French and Portuguese and English. The average ratings were
then used as gold scores to evaluate automatic compositionality prediction in
§3.3 (Ramisch, Cordeiro, Zilio, et al. 2016). For the later, we started from the
PARSEME annotations of idiomatic verbal expressions in 5 languages, and auto-
matically collected candidates for literal occurrences. Then, we annotated these
candidates, distinguishing literal from coincidental co-occurrences (in which the
tokens are syntactically incompatible with the MWE), and detailing the nature
of the literal readings characteristics (Savary, Cordeiro, Lichte, et al. 2019).

Finally, we have presented our framework for automatic compositionality pre-
diction based on word embeddings. We start by combining the embeddings of
the MWE components, and the compare it with an embedding generated for the
MWE as a whole. The cosine similarity between these vectors indicates to what
extent the distribution of the MWE is similar to the distributions (as proxies to
meanings) of their components. The framework was evaluated on 5 datasets with
7 embedding models, obtaining state-of-the-art results. We studied many aspects
of the model and report recommendations for embeddings hyper-parameters,
corpus size, combination function, etc. (Cordeiro et al. 2019).

3.5 For the record

The survey of MWE discovery in §3.1 is mostly based on Section 3 of the book
chapter Ramisch & Villavicencio (2018) and on section 2 of the survey paper
Constant et al. (2017), with minor updates. §3.2.3 summarises my work on com-
positionality annotation for nominal compounds in English, French and Brazilian
Portuguese. This work was a collaboration with Silvio Cordeiro and Aline Villav-
icencio, among others. The section contain large parts from the articles that de-
scribe the creation, analysis, and use of the resource (Ramisch, Cordeiro, Zilio,
et al. 2016; Ramisch, Cordeiro & Villavicencio 2016; Wilkens et al. 2017; Cordeiro
et al. 2019). The token-level resource described in §3.2.4 was built in collabora-
tion with Agata Savary, Silvio Cordeiro, Uxoa Iñurrieta, Timm Lichte, and Voula
Giouli. This section was adapted and summarised from Savary, Cordeiro, Lichte,
et al. (2019). We mention as related to these two resources our lexicon for French
complex prepositions and conjunctions (Ramisch, Nasr, et al. 2016). It was not
included in this chapter because the lexicon does not contain compositionality
3 Fifty shades of compositionality

annotations, and because the nature of its entries is quite different from the ones
included in the two datasets presented here.

Compositionality prediction was explored in the PhD thesis of Silvio Cordeiro (Cordeiro 2017). His work was co-supervised by Aline Villavicencio, Alexis Nasr and myself. The implementation of the framework as part of the **mwetoolkit** was published in Cordeiro, Ramisch & Villavicencio (2016a). The first experiments in compositionality prediction were published in Cordeiro, Ramisch, Idiart, et al. (2016). The whole work was summarised in a long journal article, from which most of §3.3 was adapted and summarised (Cordeiro et al. 2019).
# 4 Down-to-earth MWE identification

> *History as well as life itself is complicated – neither life nor history is an enterprise for those who seek simplicity and consistency.*

> — Jared Diamond, *Collapse*

Corpus-based methods are the current mainstream in NLP, playing a central role in modern machine learning, which relies on annotated text as supervision. MWE research, on the other hand, has its origins in lexicography and grammar engineering (Church & Hanks 1990; Sag et al. 2002), with a large literature on unsupervised methods (da Silva et al. 1999; Evert 2004; Fazly et al. 2009; Salehi et al. 2015). Hence, there has been little work on corpus annotation and supervised MWE identification in the 90’s and early 2000’s. Since about 10 years or so, the prominence of MWE-annotated corpora and in-context MWE identification raised significantly, boosted by efforts in creating and freely releasing corpora.

This chapter tells the story of MWE identification with a focus on the role of my research in this story. I start with a contextualisation of the existing research in MWE identification until roughly 2016-2017 (§4.1). Then, I will introduce PARSEME: a collective effort with major impact in the field. Our journey through the PARSEME galaxy will be divided in two parts. First, I will present the global framework and my contributions regarding corpus annotation (§4.2). Second, I will present the PARSEME shared tasks on MWE identification, in which I took part as both an organiser and a participant (§4.3).

## 4.1 Setting the scene

MWE identification takes text as input, and adds MWE annotations on top of it (§2.2.1). Chapter 3 introduced a similar task: token-level compositionality prediction, where input sentences contain a known potentially idiomatic MWE, and we predict a compositionality score for it. In MWE identification, however, the candidate expressions are not known in advance, and it is not even certain that an MWE does occur. Indeed, MWE identification is usually performed on full-text corpora, including sentences with no MWE at all. With respect to token-level
compositionality prediction, MWE identification can be seen as a harder task, in
which we must estimate the compositionality of all possible word combinations,
and group some of them into MWEs. In spite of their similarities, token-level
compositionality prediction and MWE identification developed more or less in-
dependently. While the former derived from its type-level counterpart, the latter
has its roots in parsing and named entity recognition.

Our “pre-PARSEME” literature review starts with the motivations (§4.1.1), and
covers tagging (§4.1.2) and parsing methods (§4.1.3). We then present two influ-
ential shared tasks in the field: DimSum and PARSEME (§4.1.4). The “PARSEME
era” is the topic of the subsequent sections.

4.1.1 Sparks of an idea

Syntactic parsing was probably the main motivation for developing MWE iden-
tification strategies. To some extent, historical parsers addressed some (closed
list of) MWEs, either during tokenisation or with the help of finite-state modules
(Breidt et al. 1996). The influential work of Sag et al. (2002) is among the first to
focus on the phenomenon, putting forward a set of proposals to encode MWEs in
a grammar-based parser (HPSG). Later studies confirmed this hypothesis for sev-
eral MWE categories in the grammar-based framework by noticing an increase
in parsing performance when lists of MWEs were included in the lexicon or as
special symboles and rules in the grammar (Alegria et al. 2004; Villavicencio et al.
2007; Wehrli et al. 2010; Kato et al. 2016).

For data-driven parsing, the seminal work of Nivre & Nilsson (2004) investi-
gates the impact of representing MWEs as subtrees or as words-with-spaces in
a dependency parser. Their experiments based on gold MWE annotations show
that the words-with-spaces approach seems more interesting, suggesting that
better parsing results could be obtained by pre-identifying MWEs. The authors
wonder “how much of this potential can be realized in practice, when relying on
automatic recognition of MWUs rather than manually annotated corpus data”
(Nivre & Nilsson 2004: p. 6). Later work addressed this question, not only for
parsing, but also as a standalone task that can help improve other tasks like
anaphora resolution (Wehrli & Nerima 2013), word sense disambiguation (Fin-
layson & Kulkarni 2011), and MT (Wehrli et al. 2009; Carpuat & Diab 2010).

4.1.2 Sequence tagging

Another NLP task influenced MWE identification: named entity recognition. The
mainstream model for this task was (and still is) a sequence tagging approach
based on BIO encoding (Ramshaw & Marcus 1995), where each token is assigned a single tag indicating whether it is at the beginning (B), inside (I), or outside (O) the entity. This allows us to simulate a segmentation task with a token-level tagger. It is straightforward to consider that continuous MWEs (whose components are adjacent) are like multiword named entities, and can be identified using the same sequence models.

Blunsom & Baldwin (2006) are the first to propose a sequence tagging approach for MWE identification using conditional random fields (CRF). Inspired by supertagging, their work is presented as a method for deep lexical acquisition, that is, to acquire new grammar entries from text. Although they focus on parsing results, they also assess “the ability of the CRF to identify multiword expressions” (Blunsom & Baldwin 2006: p. 170), reaching an accuracy of 75.8% and 53.6% for English and Japanese continuous expressions.

A CRF tagger is also at the core of the work of Constant & Sigogne (2011), who perform joint POS tagging and MWE identification. One of the main contributions of their work is the adaptation the BIO scheme, originally proposed for named entity recognition. Their tagging schemes concatenate lexical segmentation information (B and I tags) with the POS tag of the lexical unit to which the current token belongs. Constant & Sigogne (2011) trained and evaluated their models on the French Treebank, where MWEs of several grammatical categories are marked (Abeillé et al. 2000).

A similar CRF-based system was proposed by Shigeto et al. (2013) for English. However, they evaluate their work on the Penn Treebank, that does not contain MWE annotations. To construct a training and test dataset, they projected grammatical MWEs of the Wiktionary on the treebank, checking syntactic constraints to remove literal and coincidental occurrences. Wiki50 is the first corpus built specifically for the needs of MWE identification (Vincze et al. 2011). It consists in 50 English Wikipedia articles manually annotated for several WME categories and named entities. In their experiments, Vincze et al. (2011) study the impact of various feature sets on their CRF tagger. Scholivet & Ramisch (2017) compare a CRF with a more sophisticated parsing-based model, focusing on highly ambiguous multiword conjunctions and determiners in French.

<table><caption>Figure 4.1: Two-level BIO encoding (Schneider et al. 2014). Nested elements get lowercase bio tags. Adapted from Constant et al. (2019).</caption><tr><td>The</td><td style="color:red;">prime</td><td style="color:red;">minister</td><td>made</td><td>a</td><td style="color:orange;">few</td><td>good</td><td style="color:blue;">decisions</td></tr><tr><td>O</td><td style="color:red;">B</td><td style="color:red;">I</td><td>B</td><td>b</td><td style="color:orange;">i</td><td>o</td><td style="color:blue;">I</td></tr></table>
Since the BIO scheme cannot represent discontinuous MWEs, Schneider et al. (2014) adapt the original scheme by introducing two-level tags, successfully representing MWEs that contain gaps, that is, non-lexicalised tokens occurring in between lexicalised components. Figure 4.1 shows an example of the encoding proposed by Schneider et al. (2014), with the embedded MWE *a few* occurring in between the components of *made decisions*. Moreover, they also propose a structured perceptron for MWE identification, arguing that it is more efficient than linear-chain CRFs. This tagging scheme was then incrementally made more complex, first with the introduction of supersense tags (Schneider & Smith 2015), and later by also combining lexical categories for MWEs (Liu et al. 2021).

In order to tackle the lexical sparsity of MWEs, some studies showed interest in integrating lexicon-based features in sequence tagging models. External MWE lexicons can have a great impact on MWE identification when used as a source of features. Constant & Tellier (2012) develop a generic approach to compute features from MWE lexicons. They use this approach to identify French compounds using CRFs, showing significant gains as compared to settings without lexicon features. This method has also been successfully applied and updated for comprehensive MWE identification in English by Schneider et al. (2014) who performed fine-grained feature-engineering, designing specific features for different MWE lexicons. The use of automatically predicted compositionality scores seems to help in CRF-based MWE identification, although the performance gains are small (Scholivet et al. 2018). The impact of handcrafted vs. automatically acquired lexicons as features for MWE identification may depend on the nature of the target MWE categories Riedl & Biemann (2016).

### 4.1.3 By-product of parsing

After Nivre & Nilsson (2004), several works studied (a) the impact of MWE identification on parsing, and (b) the use of parsing models to identify MWEs. It has been shown that pre-grouping MWEs as words-with-spaces can improve a shallow parser for English (Korkontzelos & Manandhar 2010). Their approach obtains MWE annotations automatically through lookup in the English WordNet used as a MWE repository. Cafferkey et al. (2007) carried out similar experiments with a probabilistic constituency parser. MWEs were automatically identified by applying a named entity recognizer and list of prepositional MWEs. A slight but statistically significant improvement was observed in parsing performances.

Eryiğit et al. (2011) evaluate the impact of gold and automatic MWE annotations on dependency parsing for Turkish. The automatic MWE annotations are also obtained using a dictionary and several rule-based matching strategies. For
some categories, MWE identification downgrades performance, as these are easily recognisable by the parser itself. For some other categories, though, identifying MWEs prior to parsing may improve parsing accuracy up to 1.5% points. These experiments were later extended with a focus on MWE identification performance (Eryiğit et al. 2015). The authors compare a parsing strategy using a special *mwe* dependency with several lexical lookup methods, showing that the latter are more effective than the former on their corpora.

For constituency parsing, Arun & Keller (2005) propose two strategies to represent MWEs in the French Treebank. First, they concatenate the MWE components as words-with-spaces. Second, they keep the internal structure of the MWE, but append a MWE label to the phrase tag. The latter strategy was adapted by Green et al. (2011), who also propose a dedicated tree-substitution grammar to learn and predict MWEs, encoding more lexicalised context within rules. These experiments on French were later extended to Arabic, showing that their model largely outperforms both baseline MWE identification methods and standard constituency parsing models (Green et al. 2013).

Vincze, Zsibrita, et al. (2013) were among the first to use a dependency parser to perform realistic MWE identification, focusing on light-verb constructions in Hungarian. They first automatically match two annotation layers in the Szeged Treebank: syntactic dependencies and LVCs. As a result, the dependency link between a light verb and a predicative noun (e.g. *obj*) is suffixed with a *lvc* tag, whereas regular verb-argument links remain unchanged. An off-the-shelf parser is used to predict the structure of sentences, including LVC links. When compared with a classifier baseline, the parser performs slightly worse on continuous LVCs, but considerably better on discontinuous ones.

Post-processing strategies, which identify MWEs after parsing, have also been proposed. In T. et al. (2013), the text is first parsed with a standard dependency parser for English. Then, a set of rules is applied to identify potential LVCs. A classifier is then trained and applied to distinguish LVCs from regular verbal structures. A contrastive analysis for English and Hungarian using the same method has shown that the approach is quite robust across languages (Vincze, T. & Farkas 2013). The method has also been applied to English verb-particle constructions, indicating that it is portable not only to different languages, but also to different MWE categories (T. & Vincze 2014).

In Nasr et al. (2015), we assess the ability of a transition-based dependency parser trained on the French Treebank to identify, in a dedicated test set, highly ambiguous complex conjunctions such as *fr bien que* (lit. ‘well that’) ‘although’, and complex determiners such as *fr de la* (lit. ‘of the’) ‘some’. We use special dependency label (*morph*) for the MWE occurrences and regular dependencies
## 4 Down-to-earth MWE identification

<figure><img src="image_14.png" /><figcaption>Figure 4.2: Analysis of *fr bien que* as a complex conjunction (left) and as an adverb + conjunction (right). Adapted from Nasr et al. (2015).</figcaption></figure>

for coincidental occurrences, as shown in Figure 4.2. The addition of lexicon-based features explicitly modelling the valency of the verb preceding the candidate MWE considerably improve the performance of the identification scores, although the skewness of the training data instances seems to play a major role in the ability of the system to disambiguate some constructions.

When parsing MWEs, it is interesting to distinguish syntactically regular expressions (e.g. nominal compounds) from syntactically irregular ones (e.g. complex determiners). Candito & Constant (2014) investigate the use of different representations and strategies to identify and parse these two coarse categories. Their experiments on the dependency version of the French Treebank show that it is possible to identify syntactically regular (continuous) MWEs and simultaneously predict their internal syntactic structure. Dual decomposition is an alternative approach that enables combining several elementary systems by optimising a joint objective. It has been successfully applied to simultaneously identify MWEs (with a sequential CRF model) and parse sentences on the dependency French Treebank (Roux et al. 2014).

Up to now, we considered either encoding MWEs as tags, independent or combined with POS, or as special constituents or dependency labels that extend regular syntax trees. Constant et al. (2016) propose a new dedicated tree representation for MWEs, not relying on POS or dependencies, encoding only the MWE structure. When MWEs are represented as trees, any parsing method can be trained to specifically address MWE identification. The authors use an “easy-first” parsing model and evaluate both MWE identification and syntactic parsing. Their experiments show that, while MWE identification helps parsing, the opposite is not verified. A transition-based parser has also been proposed to jointly predict MWE trees and syntactic dependencies (Constant & Nivre 2016). In this framework, the choice of the classifier and hyper-parameters can have a significant impact on the results (Saied et al. 2019).

Before 2017, the only available corpora annotated for MWEs were Wiki50
(Vincze et al. 2011) and STREUSLE (Schneider & Smith 2015), in English.<fnref n="1" /> Thus, most MWE identification methods were evaluated on treebanks, with MWE annotations obtained indirectly from the syntactic trees. MWE identification was studied in the Swedish Talbanken (Nivre & Nilsson 2004), the French Treebank (Candito & Constant 2014), the Arabic Treebank (Green et al. 2013), the MST, IMST, IVS and IWT Turkish treebanks (Eryiğit et al. 2015), the Hungarian Szeged treebank (Vincze, T. & Farkas 2013), the Prague Dependency Treebank (Bejček et al. 2013), the English Penn Treebank (Cafferkey et al. 2007; Kato et al. 2016), and the Universal Dependencies treebanks (Constant & Nivre 2016).

A survey on MWE annotations in treebanks can be found in Rosén et al. (2016). In this section, we covered related work on MWE identification with the narrow scope of statistical, corpus-based methods. Constant et al. (2017: § 3.2.1) includes MWE identification using other techniques such as rules, grammars, finite-state transducers, and symbolic parsers. A survey on statistical MWE-aware parsing methods can be found in Constant et al. (2019). The latter also includes a discussion about orchestration, that is, whether MWE identification should be performed before (Eryiğit et al. 2011), during (Nasr et al. 2015), or after (T. & Vincze 2014) syntactic parsing.

### 4.1.4 DiMSUM and PARSEME: the big bang

The DiMSUM shared task was part of SemEval 2016 (Schneider et al. 2016). The training and test data consisted of a corpus derived from STREUSLE (Schneider & Smith 2015), annotated for strong and weak MWEs. The corpus was also annotated for nominal and verbal supersenses, that is, coarse word sense tags corresponding to WordNet's lexicographer file identifiers. For participants, the goal was to predict both MWE identifiers and supersenses, potentially benefiting from joint approaches to explore the overlap between these two meaning layers. This was the first time that an internationally renowned shared task put a spotlight on MWEs. The task attracted 9 submissions from 6 teams, with most methods relying on sequence taggers, either using CRFs or the averaged perceptron adapted from Schneider et al. (2014). The systems vary in their use of external resources as features, from external MWE lexicons to word embeddings learned using self supervision. The best system scored F1=57.77 on the joint task of predicting MWEs and supersenses.

We have submitted a simple system to the DiMSUM shared task, based on the mwetoolkit and some POS rules for MWE identification, and a “most-frequent-

<footnote n="1">¹Token-level compositionality datasets (§3.2.1) are composed of isolated sentences; we distinguish them from full-text MWE-annotated corpora.</footnote>
supersense” baseline for supersenses (Cordeiro, Ramisch & Villavicencio 2016b). Our system first collected a lexicon of MWEs observed in the training data, keeping track of their POS patterns and whether they appeared as continuous or discontinuous occurrences. Then, using the development set, we designed a set of rules and thresholds to match the entries in this lexicon with those in the test set. Some additional rules such as systematically annotating sequences of proper nouns completed our system. For such a simplistic approach, the obtained performance of F1=50.27% for joint MWE identification and supersense tagging was surprising. Our submission was ranked second, with three systems tied in the first position.

More or less at the same time, the PARSEME COST Action was in full swing. Initially, PARSEME was a networking project funded by the European COST association from 2013 to 2017, coordinated by Agata Savary. It gathered linguistics and NLP experts from 31 countries, mostly in Europe, working on topics related to MWEs and parsing (Savary et al. 2015). The idea of a multilingual shared task on MWE identification was first mentioned during the Action's 5th general meeting in Iași, Romania. Some participants of working group 3, on "statistical, hybrid and multilingual processing of MWEs" initiated the work on the annotation guidelines. This was the beginning of an incredible collective endeavour.

I joined this core group of shared task organisers shortly after its creation, and actively participated in the whole process, from the first pilot annotation in 2016 to the 2022 release of the corpora, taking place as I work on this manuscript. The PARSEME corpora and shared tasks deeply influenced my research in the last 8 years or so, and they deserve a special treatment in this manuscript. Thus, the next two sections are dedicated to the PARSEME galaxy, describing the corpora (§4.2) and the shared tasks associated with them (§4.3).

## 4.2 The PARSEME galaxy: corpora

I had the opportunity to contribute to the creation of several MWE-annotated corpora in the context of PARSEME. In this section, I describe the PARSEME framework from the point of view of the organiser and corpus contributor. I will start with a summary of the PARSEME guidelines for verbal MWEs (4.2.1), the tools and resources in the annotation environment (4.2.2), and then I will present some statistics of the resulting corpora (4.2.3).
4.2.1 Guidelines: finding true North

Multiword expressions may sound like abstract notions that linguists discuss in their ivory towers. However, as soon as we get our nose into the data, we realise that creating annotated MWE corpora, like any NLP resource, is not a fr long
fleuve tranquille (lit. ‘long river calm’) ‘smooth process’. Quite the opposite, it is a chaotic, complex and non-linear collective effort, with its share of ups and downs.
Countless hours of my work went into discussing borderline cases, defining rules, finding the right examples to demonstrate a linguistic notion, etc.

In this section, I will overview the result of this work, presenting two sets of
distinct but related guidelines for the annotation of MWEs. First, I will summarise
the PARSEME annotation guidelines, developed in the international context of
the homonymous COST Action described above. The ambition of the PARSEME
guidelines is to be as universal as possible regarding the covered languages, but
with a narrow scope in terms of MWE categories, focusing on verbal expressions
only (§4.2.1.1). Then, I will summarise the PARSEME-FR guidelines, developed
during the French spin-off project.<fnref n="2" /> These cover all other MWE categories and
also named entities, but were designed specifically for French (§4.2.1.2).

4.2.1.1 The PARSEME guidelines

One of the major decisions taken in the PARSEME working group was to focus
on verbal MWEs. Verbal MWEs are interesting for several reasons, as detailed in
Savary et al. (2018). These include the fact that:

• they tend to be more discontinuous than other MWE categories,

• thus they present long-distance dependencies, interesting for parsing,

• they are challenging to model in cross-lingually consistent framework,

• they are a starting point for future guidelines for non-verbal MWEs.

**Basic definitions** One of the main contributions of the PARSEME guidelines is the refined definition of MWE, introduced in §2.1.2. Remember that MWEs are not only composed of several lexemes, but must also form a connected syntax tree, and display some degree of lexical, morphological, syntactic and/or semantic idiosyncrasy. The guidelines delineate the border between MWEs and related phenomena such as collocations and metaphors.

<footnote n="2">²https://parsemefr.lis-lab.fr</footnote>
Those elements of the MWE that cannot be omitted without losing the id-
iomatic meaning are called **lexicalised components** of the MWE, whereas op-
tional arguments are **open slots**. Selected prepositions are considered to attach
to nominals, so they are not lexicalised components when they introduce open
slots, although always present when the MWE occurs.

We define **verbal MWEs (VMWEs)** as those which, in their canonical forms, have a verb as their syntactic heads. Since VMWEs can occur in various syntactic structures, we need to neutralise variation before applying the decision trees described below. Thus, we define a **canonical form** by listing a set of prototypical syntactic configurations, typically with the verb in finite form. The guidelines can be applied to **meaning-preserving variants** such as analytical tenses and gerunds, as long as a canonical form can be identified. However, nominalisations (e.g. *fr une mise en scène* (lit. ‘a putting in scene’) ‘a direction of a theater play’) and exocentric non-verbal MWEs containing verbs are out of scope (e.g. *pt um faz-de-conta* (lit. ‘a make-as-story’) ‘a make-believe’).

**VMWE categories** The PARSEME typology of verbal MWEs distinguishes so-
called “universal”, “quasi-universal”, language-specific and optional categories:

1. Two *universal* categories are applicable to all covered languages:<fnref n="3" />

a) **Light-verb constructions** (LVC), divided into two subcategories:

i. LVCs in which the verb's meaning is totally bleached (LVC.full),
   [de] *eine Rede halten* (lit. 'hold a speech') 'give a speech',

ii. LVCs in which the verb adds a causative meaning to the noun
   (LVC.cause), e.g. [pl] *narazić na straty* ‘expose to losses’

b) **Verbal idioms** (VID), are VMWEs not belonging to other categories,
most often being semantically non compositional, e.g. **fr prendre à**
**cœur** (lit. ‘take to heart’) ‘take it seriously’

2. Three *quasi-universal* categories are valid for some languages, but not all:

a) **Inherently reflexive verbs** (IRV), pervasive in Romance and Slavic languages, and present in Hungarian and German, where a reflexive clitic (**REFL**) always co-occurs with a verb, or markedly changes its meaning or valency, e.g. **pt se formar** (lit. ‘REFL form’) ‘graduate’

<footnote n="3">³We use the term “universal” to refer to all languages covered in the project. We hope that as this set of languages increases our categories will become more and more truly universal.</footnote>
b) **Verb-particle constructions** (VPC), pervasive in Germanic languages and Hungarian, rare in Romance and absent in Slavic languages, with two subcategories:

i. fully non-compositional VPCs (VPC.full): the particle fully modifies the verb's meaning, e.g. **hu** *berúg* (lit. 'in-kick') 'get drunk'

ii. semi non-compositional VPCs (VPC.semi): the particle adds a partly predictable but non-spatial meaning, e.g. **en** *wake up*

c) **Multi-verb constructions** (MVC) – close to semantically idiomatic serial verbs in Asian languages like Chinese, Hindi, Indonesian and Japanese (but also attested in Spanish), e.g. **hi kar le** (lit. 'do take') 'do (for one's own benefit)'.

3. One *language-specific* category, introduced for Italian:

a) **Inherently clitic verbs** (LS.ICV),<fnref n="4" /> in which at least one non-reflexive clitic (CLI) always accompanies a verb, or markedly changes its meaning or valency, e.g. **it prenderle** (lit. 'take-them') 'get beaten up'.

4. One *optional experimental* category, to be considered at post-annotation:

a) **Inherently adpositional verbs** (IAV) include idiomatic combinations of verbs with post- or prepositions, e.g. **hr ne dođe do usporavanja** (lit. 'not come.FUT to delay') 'no delay will occur'<fnref n="5" />

**Decision trees** The PARSEME guidelines are organised as a set of deterministic decision trees that operationalise the typology above. The goal is that the outcome of these decision trees is as objective and reproducible as possible. The decision trees leave little room for subjective interpretation, and should allow for high inter-annotator agreement.

1. The first phase in this process is the *identification of candidates*, based on the annotator's experience and linguistic intuition. Once a verb and at least one dependent are identified as a potential VMWE, annotators ensure that there are at least two lexicalised components. If the candidate is a meaning-preserving variant, they must find a corresponding canonical form.

<footnote n="4">⁴This category is likely generalisable to other Romance languages such as French.</footnote>

<footnote n="5">⁵This category is considered experimental because, so far, we could not come up with satisfactory tests to clearly distinguish IAVs from regular verbal valency.</footnote>
## 4 Down-to-earth MWE identification

2. The first decision tree contains structural tests, and checks the syntactic structure of the candidate's canonical form. Depending on the POS and dependencies of the verb's dependents, this first decision tree redirects to a second level of tests. Figure 4.3 shows the tests in this decision tree.

3. The final step consists in applying the category-specific tests, which depend on the syntactic structure and allow not only confirming the MWE status of the candidate, but also assigning it a unique category.

<figure><img src="image_15.png" /><figcaption>Figure 4.3: Excerpt of PARSEME 1.2 guidelines, structural decision tree.</figcaption></figure>

Category-specific tests depend on the nature of the verb's dependents. For instance, the decision tree for IRV attempts to identify the semantic/thematic role of the reflexive clitic. If the clitic is used as an expletive in an impersonal or middle-passive (inchoative) alternation, or stands for a pronominalised reflexive or reciprocal argument, then the candidate is *not* an IRV. If, however, the tests for all regular uses fail (answer is NO), then we annotate the candidate as IRV. On the other hand, LVC tests attempt to characterise the nature of the dependent noun or noun phrase as denoting an event or state. LVC tests also check whether the verb is light, that is, only contributes morphological features to the event or state denoted by the dependent. All tests need to pass (answer is YES) before one can confirm that the candidate is an LVC.
Category-specific tests are usually based on the acceptability of paraphrases, such as the LEX test below, employed to identify MWEs of the VID category:

* [LEX] Does a regular *replacement* of one of the components by related words taken from a *relatively large semantic class* lead to ungrammaticality or to an unexpected change in meaning?

↔ YES ⇒ it is a VID

* [pt] eu *quebro* um *galho* (lit. ‘I break a branch’) ‘I help’ → #eu *danifico* um ramo ‘I damage a stem’

↔ NO ⇒ further tests are required

For many categories, the decision trees are complemented by lists of borderline cases and recommendations on how to analyse them. Moreover, the guidelines also contain a glossary that provides more details on some linguistic notions such as “unexpected meaning shift” and “extended noun phrases”. These guidelines are the result of a collective effort: our GitLab project contains 54 members, the guidelines project has 515 commits, and a total of 104 issues were raised, among which 27 are still open and the remainder were discussed and closed after a consensus was reached.<fnref n="6" />

<figure><img src="image_16.png" /><figcaption>Figure 4.4: Number of examples per language in PARSEME guidelines.</figcaption></figure>

**Multilingual examples** One important aspect of the PARSEME guidelines is the database of examples in multiple languages. Currently, the guidelines feature 232 example identifiers, each covering up to 28 languages. However, not all languages have examples for all example identifiers: we have a total of 1,980 examples, whereas in theory we could add up to 232 × 28 = 6,496 examples. In

<footnote n="6">⁶Statistics from https://gitlab.com/parseme on September 14, 2022.</footnote>
edition 1.2, the guidelines contained 1,801 examples; the newly added examples concern mostly Serbian and Arabic. Figure 4.4 shows a histogram with the number of examples per language, ranging from 188 for Spanish to only 1 example for Turkish, Hebrew and Lithuanian.<fnref n="7" />

The examples in the guidelines are complex, including their form in the original language, lexicalised components in bold, literal and idiomatic translations, as well as explanations, comments, negative counter-examples, etc. Their edition by language experts is a time-consuming and error-prone process that required much energy. One of the latest improvements on the PARSEME guidelines is a system for online example edition. The original XML language used to edit the examples on a shared online spreadsheet was recently replaced by an online edition system, developed under my supervision as part of the internship of Quentin Barrouyer and Baptiste Souche, master students at Aix-Marseille University. The screenshot on Figure 4.5 illustrates the example edition module.

<figure><img src="image_17.png" /><figcaption>Figure 4.5: Screenshot of the visual example edition interface.</figcaption></figure>

In this manuscript, we only provide a high-level description of the PARSEME guidelines. For the details, the reader can refer to the freely available online annotation guidelines, available at https://parsemefr.lis-lab.fr/parseme-st-guidelines/.

### 4.2.1.2 The PARSEME-FR guidelines

The PARSEME-FR project was a French spin-off of PARSEME, funded by the National Research Agency (ANR) from 2016 to 2021. In this project, we have de-

<footnote n="7">⁷Statistics based on a dump of the examples database on September 14, 2022.</footnote>
veloped guidelines to annotate non-verbal MWEs, completing the international PARSEME guidelines for VMWEs. In addition, the initial goal of the PARSEME-FR guidelines was to cover MWEs and named entities (NEs) in a single annotation framework. We ended up writing two separate guidelines for MWEs and named entities, with a common top-level entry point.

<figure><img src="image_18.png" /><figcaption>Figure 4.6: Top decision flowchart of the PARSEME-FR annotation guidelines. Source: Candito et al. (2021).</figcaption></figure>

**Top-level decision flowchart** Figure 4.6 illustrates the top decision flowchart, which guides the annotator to the appropriate guidelines. The initial step (CAND) is largely based on the annotator's intuition, which is further confirmed or contradicted by more rigorous tests. In this step, a candidate *c* can be composed of one or more lexemes since single-word NEs are also annotated. The next step (VP_DISTRIB) redirects to the PARSEME VMWE guidelines (§4.2.1.1) if *c* has a distribution of a verbal phrase or a sentence, e.g. *fr il vide son sac* (lit. 'he empties his bag') 'he gets it off his chest'. If *c* is neither verbal nor nominal (NP_-DISTRIB), e.g., *à l'issue de* (lit. 'at the outcome of') 'after', it is tested against our
## 4 Down-to-earth MWE identification

non-verbal MWE guidelines, provided that it is composed of two or more lexemes, and discarded otherwise. If *c* is nominal, it can (in the given context) either be used generically, or refer to a specific entity *e* (SPECIF_REF). In the former case, *c* cannot be a NE, but might be a non-verbal MWE. In the latter case (or if the test is hard to apply), it is necessary to determine the naming convention which links *c* to its referent *e*. If this convention covers the whole concept (CONCEPT_NAMING_CONV), then *c* can (in other contexts) refer to another referent *e'* on the basis of the properties of *e'*. In this case, if *c* is multiword, it might be a non-verbal MWE. Conversely, the naming convention may cover only the link between *c* and *e*, rather than a whole concept. In this case, *c* might be a NE. Thus, if *e* belongs to one of the pre-selected semantic categories (person, organization, location, human product or event), then *c* is tested against the NE guidelines. If their outcome is negative and if *c* is multiword, it might still be a non-verbal MWE.

**Named entities** The scope of the NE annotation in PARSEME-FR covers:

* Persons (PERS), e.g. [*Gutenberg*]PERS, [*Bernard Bonnet*]PERS;

* Locations (LOC), e.g. [*golfe d'[*Ajaccio*]]LOC*LOC* (lit. 'Ajaccio Bay');

* Organisations and human collectives (ORG), e.g. [*Comité départemental d'action touristique*]ORG (lit. 'Departement Committee of Tourism');

* Product names, including titles of works (PROD), e.g. [*Angiox*]PROD, [*Libération*]PROD 'a newspaper';

* Named events (EVE), e.g. [*L'affaire [*Dumas*]PERS]EVE (lit. 'Dumasgate').

Like MWEs, we first identify NE candidates via linguistic intuition, then apply tests to confirm (a) the naming convention and (b) the span of annotation. The naming convention relies on surface clues and external resources:

* [*ObviousProper*]: Is the candidate obviously a proper name, that is, is the annotator confident about its naming convention?

* [*RelevUpper*]: Is the candidate, or its variants in the same text, spelled with an initial uppercase letter to signal a proper name?

* [*Acron*]: Does the candidate sequence have an acronym in the given text?

* [*WebPage*]: Is the candidate the title of a valid website or Wikipedia page?

Two additional tests deal specifically with the annotation span, which can be difficult to determine in the presence of classifiers, titles, abbreviations, etc.:
* **[MinSpan]:** Is the span of the candidate *c* minimal, that is, a shorter span *c* does not refer to the same entity? For instance, in `[la Rochelle]_LOC`, the determiner cannot be omitted.

* **[SpanPerCat]:** This test lists some notoriously difficult cases per category. Classifiers are systematically excluded from person names, events, products, cities (*la ville de* `[Loudun]_LOC 'the city of Loudun'), regions, departments, and some organisations. In other cases, the classifier is systematically included (`[école Notre-Dame]_LOC 'Our Lady's School').

These tests are not applied sequentially, but included within a decision flow-chart, omitted for the sake of brevity. In addition, we provide a way to express whether a named entity's category is primitive or final, to account for metonymical uses. The overlap between nominal MWEs and NEs was thoroughly studied within the project, and our guidelines also propose criteria to annotate sequences that can be seen as both, or nested (e.g. a NE containing a MWE).

### Non-verbal multiword expressions
The main concepts of the PARSEME-FR guidelines are adapted from the PARSEME guidelines. We also rely here on the words vs. tokens distinction, on the notion of lexicalised components, and we also consider that selected prepositions (and complementisers) are not to be included as lexicalised components. The rule for excluding final grammatical markers has an exception, though. For a sequence containing just one component plus a selected preposition, we annotate it as MWE if it satisfies other criteria than the fixedness of the preposition. This is the case, for instance, for *faute de* (lit. 'fault of'): it functions as a sentence modifier, which is normally not the case for a non-temporal noun such as *faute* (lit. 'fault'). The criteria to determine whether *c* is a non-verbal MWE are summarized below:

1. Semantic criteria:

* [ID] the syntactic head of *c* is not its "hypernym"

* [PRED] no predication relation between head and modifier

2. Lexical fixedness criteria:

* [CRAN] *c* contains a cranberry word

* [LEX] no replacement of a content word by a similar word

* [DET] the determiner of a noun is totally fixed

* [ZERO] possible empty determiner, while usually required

3. Morphosyntactic fixedness criteria:

* [MORPHO] no modification of the morphological features
4 Down-to-earth MWE identification

• [IRREG] irregular morphosyntactic structure

• [SYNT] impossibility of syntactic variation for some patterns

• [INSERT] no insertion of modifiers, while usually possible

Here, we only briefly summarise the PARSEME-FR guidelines. For the details,
the reader can refer to the freely available online annotation guidelines.<fnref n="8" /> The
PARSEME-FR corpus (and guidelines) are described in Candito et al. (2021) and
in Candito et al. (2017) (in French).

4.2.2 Environment and tools

One of the assets of PARSEME is its technical environment which makes the
annotation process easier and allows for smooth onboarding for new languages.
The tools used for annotation and corpus processing are shared among all con-
tributors in the community, strengthening their bonds. Below, we describe some
of the tools that are used for corpus creation, management, and enhancement.

**FLAT** The core of the corpus creation work is MWE annotation. FLAT is a generic corpus annotation platform that is open-source freely available, developed by Maarten van Gompel.<fnref n="9" /> The interface allows editing the XML-based FoLiA corpus format (van Gompel & Reynaert 2013). The PARSEME annotation server, also used in the PARSEME-FR project, is hosted on a dedicated web server at the University of Düsseldorf.<fnref n="10" /> We have been in contact with the platform’s developer, especially at the beginning, and new features have been implemented to match the PARSEME needs: support to right-to-left languages, corpus files pagination, and asynchronous uploading for faster annotation.

Figure 4.7 shows an example of corpus annotation on FLAT. A tokenised corpus in CUPT (§4.3) or FoLiA format is uploaded to the platform. If POS information is available, verbs are emphasised by a “V” superscript (since we focus on verbal MWEs). Then, the annotator reads the text sentence by sentence and, whenever a MWE occurs, clicks on its words and selects a category from a drop-down menu. It is possible to add textual comments, a confidence score, edit or delete annotations, and represent overlaps, like the coordinated LVC in the last sentence of Figure 4.7 (**terão apoio e recursos** ‘will have support and resources’).

<footnote n="8">⁸https://gitlab.lis-lab.fr/PARSEME-FR/PARSEME-FR-public/-/wikis/</footnote>

<footnote n="9">⁹https://flat.readthedocs.io</footnote>

<footnote n="10">¹⁰https://mwe.phil.hhu.de/</footnote>
<figure><img src="image_19.png" /><figcaption>Figure 4.7: Screenshot of a FLAT annotation page for Portuguese.</figcaption></figure>

**Consistency checks** High-quality annotations usually rely on a quite stable methodology, with two raters annotating the same data, and a third adjudicator resolving conflicts (Ide & Pustejovsky 2017). In PARSEME, the limited availability of annotators prevents us from systematically adopting this methodology. To compensate for the lack of double annotation, we have developed an original tool referred to as the PARSEME “consistency checks”.

<figure><img src="image_20.png" /><figcaption>Figure 4.8: Screenshot of a consistency checks page for Spanish.</figcaption></figure>

In practice, once MWEs annotation by linguistic experts is performed on corpora, the consistency checks phase takes place. The web interface consists in a vertical visualisation in which all annotated instances (tokens) of the same MWE type are grouped, as illustrated in Figure 4.8. In other words, sentences are not listed in the order they appear in the corpus, but under the lemmas of the annotated MWEs. In the screenshot, the MWE *abrir camino* (lit. ‘open way’)
## 4 Down-to-earth MWE identification

‘open possibility’ is shown in two contexts in which it was annotated as VID. The expert can compare instances and verify if decisions were made consistently.

Inconsistencies may arise not only from subjective interpretation by different annotators, but also due to inattention by a single annotator, or simply because different instances of the same MWE appear far apart in the corpus, and it is impossible to memorise all previous decisions while annotating. Moreover, potentially skipped MWEs are also automatically identified, using the heuristics presented in §3.2.4 with language-specific adaptations. In the example, the expert might, for instance, decide that the first sentence marked as *Skipped* is due to inattention and should have been annotated. These decisions are recorded and a patch is created that can be then applied to the original corpus, to increase the consistency of annotations. My experience in the project has shown that this tool considerably enhances the quality of the annotated corpora.

### Adjudication
In the PARSEME-FR project, we performed double annotation of the Sequoia treebank, containing 3,099 sentences in French (Candito & Seddah 2012). Thus, the consistency checks tool was adapted to take into account inconsistent annotations for the same sentences. The adjudication interface, shown in Figure 4.9, shows sentences in the original corpus order, omitting those with no annotation or when both annotators agree. Disagreements include MWEs annotated by a single annotator, by both annotators but with different labels, or with different spans. For these cases, the interface allows marking one of the annotations as correct, or adding a new custom annotation.

<figure><img src="image_21.png" /><figcaption>Figure 4.9: Screenshot of an adjudication page for French.</figcaption></figure>

### Corpus management and tools
In the international PARSEME initiative, the MWE-annotated corpora of each language are maintained in a dedicated git
repository for version control. Each language team manages its repository according to shared naming conventions. In addition to the corpus files, PARSEME provides a set of useful tools to manage the corpora, including:

* The `cuptlib` library, able to read and manipulate the PARSEME corpus format (called “CUPT”, see §4.3.1), and represent sentences and MWE annotations as Python objects.<fnref n="11" />

* A tool to convert corpus files between the CUPT format and the FoLiA XML format used by the `FLAT` annotation platform.

* A tool which allows joining CUPT files with CoNLL-U files containing morpho-syntactic annotations. The CoNLL-U files can come from gold Universal Dependencies treebanks, or from automatic predictions output by tools such as `UDPipe`.<fnref n="12" />

* A tool to obtain useful statistics from the corpora, such as the distribution (histogram) of MWE categories, lengths (number of tokens), distance (between the first and last lexicalised component) and discontinuities (number of non-lexicalised tokens between first and last lexicalised component).

<figure><img src="image_22.png" /><figcaption>Figure 4.10: Screenshot of Grew-match page showing an MWE in the English corpus.</figcaption></figure>

**Grew-match** Edition 1.2 of the PARSEME shared task introduced the use of Grew-match as a tool to query the corpora (Ramisch et al. 2020). All along the annotation phase, the latest version of the annotated corpora (on its respective git repository) was searchable online via the Grew-match querying tool.<fnref n="13" /> Grew-match is a generic graph-matching tool which was adapted to take into account the MWE annotations, by adding MWE-specific graph nodes and arcs, as shown in Figure 4.10: each MWE gives rise to a fake “token” node, heading arcs to all the components of the MWE. Language teams can thus use Grew-match to query

<footnote n="11">¹¹https://gitlab.com/parseme/cuptlib</footnote>

<footnote n="12">¹²https://gitlab.com/parseme/corpora/wikis/Enhancing-existing-corpora</footnote>

<footnote n="13">¹³http://parseme.grew.fr/</footnote>
4 Down-to-earth MWE identification

and visualise the MWE-annotated corpora, either to find examples or to iden-
tify potential errors and inconsistencies. For example, the VMWE in Figure 4.10
would be retrieved by searching for VMWEs lacking a verbal component. In this
case, the expert can see that the MWE annotation is actually correct, whereas
the (automatically predicted) POS tag of the verb cut is incorrect.

**Documentation** Up to 2020, the release of annotated corpora was coordinated with PARSEME shared tasks. Since then, effort has been put into dissociating corpus annotation from shared tasks. Each language team was given a git repository containing development versions of the corpora. We have created a wiki website containing the “Language Leaders guide”, with instructions to prepare data, recruit and train annotators, use common tools to create and manipulate data (e.g. the centralised annotation platform FLAT), etc.<fnref n="14" /> This documentation evolves as the initiative moves towards more frequent releases of the corpora. We hope that this will allow more flexible resource creation, in accordance with each team’s needs and availability. Moreover, extensions and enhancements in the corpora can be integrated into MWE identification tools faster.

4.2.3 Corpus stats

In this section, we summarise the resulting corpora for both the international
PARSEME initiative and for the PARSEME-FR corpus.

PARSEME The PARSEME international initiative has released three versions of the MWE-annotated corpora, and one release is being prepared at the time of writing this manuscript. These versions include different subsets of languages, with different numbers of annotated sentences.

In 2017, for the edition 1.0 of the PARSEME shared task, we provided data for 18 languages using version 1.0 of the annotation guidelines.¹⁵ This was the first corpus annotated for verbal MWEs in a highly multilingual setting following a set of shared guidelines, annotation tools and formats. Corpora in each language had been split into a training part, provided to the shared task participants in advance, and a test part, used to evaluate the systems, and only released at the end of the evaluation campaign. In total, the 18 language teams produced 62k VMWE annotations on a set of 274k sentences, with an overall average of one expression every 4.4 sentences.

<footnote n="14">¹⁴Available at: https://gitlab.com/parseme/corpora/wikis
¹⁵https://parsemefr.lis-lab.fr/parseme-st-guidelines/1.0/</footnote>
<table><caption>Table 4.1: Number of languages, sentences, tokens, and annotated verbal MWEs (across all languages) for the PARSEME corpora.</caption><thead><tr><th>References</th><th>#lang</th><th>#sent</th><th>#token</th><th>#VMWE</th></tr></thead><tbody><tr><td>v1.0 (Savary et al. 2017)<br><a href="http://multiword.sourceforge.net/sharedtask2017">http://multiword.sourceforge.net/sharedtask2017</a></td><td>18</td><td>274,376</td><td>5.4M</td><td>62,218</td></tr><tr><td>v1.1 (Ramisch, Cordeiro, et al. 2018)<br><a href="http://multiword.sourceforge.net/sharedtask2018">http://multiword.sourceforge.net/sharedtask2018</a></td><td>20</td><td>280,838</td><td>6.1M</td><td>79,326</td></tr><tr><td>v1.2 (Ramisch et al. 2020)</td><td>14</td><td>279,785</td><td>5.5M</td><td>68,503</td></tr></tbody></table>

In 2018, version 1.1 introduced major changes that were largely kept since then: a new data format (CUPT, §4.3.1), new annotation guidelines, new languages, more systematic quality control (consistency checks, §4.2.2), and phenomenon-specific evaluation metrics for identification systems (§4.3.2). Annotation guidelines were modified to improve the clarity of the criteria, rename, add, and remove some categories, and take the phenomena present in more diverse languages into account (e.g. Hindi, Basque).<fnref n="16" /> Three languages from version 1.0 – Maltese, Czech, and Swedish – were not included in version 1.1, but five new languages joined the initiative and were included in this version – Arabic, Basque, Croatian, English, and Hindi. The final release covers 20 languages, and each corpus is split into training, development and test sets.<fnref n="17" /> The test corpora were newly annotated for this edition, since the test data for edition 1.0 had already been released and could not be considered as “blind” any more.

In 2020, version 1.2 was released containing no major changes in the annotation guidelines, but covering only 14 languages among which two new languages from more diverse language families: Chinese and Irish. The reason for this decrease in coverage is that the associated shared task also required the preparation of a large raw corpus (not annotated for MWEs, but automatically annotated for morpho-syntactic information). Thus, many language teams were not available to prepare this raw corpus and their corpora were not included in the release. For this version, all languages use Universal Dependencies for the morpho-syntactic layers, and performed consistency checks. Some languages included newly annotated sentences. For all languages, we re-split the training, development and test sets so that a certain amount of VMWEs in the test set was not seen in the training set (as detailed in §4.3.1).

<footnote n="16">¹⁶https://docs.google.com/document/d/15XPEYCK7tE9pT01yjaqi_oQCtFX_HRszMxCNGjwIDFI/</footnote>

<footnote n="17">¹⁷Except Hindi, English, and Lithuanian, too small to get a reasonably sized development set.</footnote>
4 Down-to-earth MWE identification

Version 1.0 of the corpus has been described in detail by Savary et al. (2018). In Ramisch, Ramisch, et al. (2018), we present a linguistic analysis of version 1.1 of the Brazilian Portuguese corpus. We characterise, for each VMWE category, the most frequent annotated expressions, as well as the distribution of expression length and gap sizes (discontinuities), overlaps and ambiguity. The paper features a list of challenging linguistic phenomena for VMWE annotation in Brazilian Portuguese, with proposals for their consistent annotation.

Analyses of the PARSEME corpora have been published for several other languages, including Arabic (Mohamed et al. 2022), Basque (Iñurrieta et al. 2018), Chinese (Jiang et al. 2018), English (Walsh et al. 2018), Irish (Walsh et al. 2020), Polish (Savary & Waszczuk 2020), Romanian (M mitelu, Cristescu & Onofrei 2019; Mititelu et al. 2022), and Turkish (Ozturk et al. 2022). Finally, there have been multi-lingual analyses of the corpora concerning specific aspects such as the ratio of unseen VMWEs (Maldonado & QasemiZadeh 2018), and the diversity of annotations (Lion-Bouton et al. 2022).

### PARSEME-FR
The PARSEME-FR corpus is significantly different from the multilingual corpora described above. We have annotated the Sequoia treebank, containing 3,099 sentences in French (Candito & Seddah 2012). Annotations cover not only verbal and non-verbal MWEs, but also named entities, as described in §4.2.1.2. In addition, the corpus annotation methodology was different: all sentences were double-annotated independently and then adjudicated by a third person. Verbal MWEs, which had already been annotated by the international PARSEME initiative on the same corpus, were presented as pre-annotations, to be updated or corrected if necessary. Named entities (NEs) include both multi-word and single-word ones.

For non-verbal MWEs, each annotator also indicated one criterion/test among those described in §4.2.1.2, used to determine that a particular instance is indeed a MWE by one of the two annotators. Finally, we employed POS patterns to semi-automatically distinguish irregular from regular MWEs. For irregular MWEs, the POS of the whole cannot be inferred from the POS of the syntactic head of the expression, so we also indicate the POS of the whole.

Figure 4.11 illustrates our fine-grained annotation scheme on an example sentence.<fnref n="18" /> Each annotation label is composed of three parts separated by a vertical pipe (|): the POS of the whole expression (when it cannot be inferred), the category of the entity/expression, and an annotation criterion (for non-verbal MWEs). The example contains the following annotations:

<footnote n="18">¹⁸Adapted from the corpus description website: https://gitlab.lis-lab.fr/PARSEME-FR/PARSEME-FR-public/-/wikis/Corpus-format-description</footnote>
<figure><img src="image_23.png" /><figcaption>Figure 4.11: Sentence annotated according to PARSEME-FR guidelines.</figcaption></figure>

* Three named entity annotations: a two-words person name *Jean Gapé* (EN-PERS.f); and two annotations on the same proper name *Peugeot*: organisation (final type EN-ORG.f) and its original meaning as a person name (primitive type EN-PERS.p). Notice that all named entities get the proper noun part of speech (NPP) and have no associated annotation criterion (\_).

* Three verbal MWE annotations: one discontinuous verbal idiom *fait partie* (lit. 'make part') 'be a member' (MWE-VID); and two overlapping full light-verb constructions (MWE-LVC.full) *effectue travaux* 'performs works' and *effectue tâches* 'performs tasks'. Verbal MWEs come from the international PARSEME annotation campaign and have no associated criterion. Their POS (verb) can be inferred and is not specified either.

* Two non-verbal MWE annotations: one regular common noun *conseil général* (lit. 'council general') 'regional council' annotated thanks to the LEX criterion; and one irregular adverb *tout au plus* (lit. 'all to the more') 'at most' indicated by the ADV part of speech in the first field, which happens to also have IRREG as associated criterion.

Table 4.2 shows the statistics of the PARSEME-FR corpus. It contains 3.1k NEs and 3.4k MWEs, for a total of 6.5k annotations. Annotations occur at a rate of one MWE/NE every 10.5 tokens. Overall, 11.2% and 7.9% of the tokens belong to MWEs and NEs, respectively, 18.9% belong to any of these two categories, and 0.2% belong to both an MWE and an NE. MWEs account for 52.5% of the annotated entities, and are mostly syntactically regular. About one third of them are VMWEs (inherited from the international PARSEME corpus). A VMWE occurs once every 70.1 tokens, with an average length of 2.29 tokens. VMWEs are
## 4 Down-to-earth MWE identification

<table><caption>Table 4.2: Number of annotations (#), number of tokens per annotation (rate), % of discontinuities, average length. Source: Candito et al. (2021).</caption><thead><tr><th></th><th>#</th><th>rate</th><th>discontinuous</th><th>length</th></tr></thead><tbody><tr><td>All</td><td>6,579</td><td>10.5</td><td>9.7%</td><td>2.10</td></tr><tr><td>NEs</td><td>3,128</td><td>22.0</td><td>0.4%</td><td>1.83</td></tr><tr><td>MWEs</td><td>3,451</td><td>19.9</td><td>18.1%</td><td>2.34</td></tr><tr><td>→Regular</td><td>2,764</td><td>24.9</td><td>22.3%</td><td>2.42</td></tr><tr><td>→Verbal</td><td>981</td><td>70.1</td><td>50.6%</td><td>2.29</td></tr><tr><td>→Others</td><td>1,783</td><td>38.6</td><td>6.7%</td><td>2.49</td></tr><tr><td>→Irregular</td><td>687</td><td>100.1</td><td>1.0%</td><td>2.02</td></tr></tbody></table>

much more often discontinuous than other categories (50.6% of the time), with an average gap of 0.9 tokens.

Non-verbal MWEs correspond to 37.5% of all annotations, and occur at a rate of 0.8 per sentence (and one non-verbal MWE every 27.8 tokens). They have an average length of 2.36 tokens but, differently from VMWEs, they are mostly continuous (94.9%). Most non-verbal MWEs are syntactically regular (72.2%). They occur once every 38.6 tokens, have the largest average length (2.49); only 6.7% of them are discontinuous. Only 687 MWEs (all non-verbal) are syntactically irregular. These include all MWEs with a cranberry word. They are almost always continuous (99%) and most of them behave as an adverb (30%) or preposition (27%). The partitive determiner *du* accounts for 5% of irregular MWEs.

In Candito et al. (2021), we provide more detailed analyses: the influence of Sequoia's domain sub-corpora on the MWE/NE distribution; the variability of annotations; the distribution of the criterion annotations; and a comparison with other corpora. These are omitted here for the sake of concision.

## 4.3 The PARSEME galaxy: shared tasks

The creation of the corpora described in §4.2 was motivated by three editions of the PARSEME shared tasks, organised in conjunction with the annual MWE workshop in 2017 (edition 1.0), 2018 (edition 1.1) and 2020 (edition 1.2). The goal of these shared tasks was to stimulate the development of multilingual systems for the automatic identification of *verbal* MWEs in text.

In this section, we will look at the data provided to participant systems (§4.3.1), the evaluation metrics used to assess their predictions (§4.3.2), and at a subset of
the systems in whose development I was involved (§4.3.3). The section concludes with a brief discussion of the results of the most recent edition (§4.3.4).

### 4.3.1 Countdown: data preparation

At each edition of the shared task, participants were given a set of corpora in 14 to 20 languages, depending on the edition. These consist of particular splits of corpora created and maintained by the international PARSEME initiative and described in §4.2.3.<fnref n="19" /> Below, we discuss the corpus format, the splitting strategies, and the tracks for shared task participants.

**Format** The participants of the shared task have access to training corpora that can be used as supervision for machine learning models. They contain not only MWE annotations, but also morpho-syntactic information, obtained from tree-banks or from automatic taggers/parsers. These rely largely on the framework proposed by the Universal Dependencies (UD) project (de Marneffe et al. 2021).

Thus, we provide the text segmented into sentences and tokenised, with processed contractions, e.g. *don't* → *do not*. In addition to the surface form, each token is associated to its numerical ID, lemma, universal POS, morphological features, syntactic head's ID and syntactic relation. This information is present in UD's CoNLL-U file format, with one token per line, and tab-separated columns for linguistic annotations, with blank lines between sentences.<fnref n="20" />

<figure><img src="image_24.png" /><figcaption>Figure 4.12: CUPT format example, MWE annotations in 11th column.</figcaption></figure>

<footnote n="19">¹⁹The PARSEME-FR corpus was not used in these shared tasks and will not be mentioned in the remainder of this section.</footnote>

<footnote n="20">²⁰https://universaldependencies.org/format.html</footnote>
4 Down-to-earth MWE identification

In edition 1.0, we provided CoNLL-U files containing morpho-syntactic annotations, and aligned files in a PARSEME-specific format using tab-separated values (TSV), which we called PARSEME-TSV.<fnref n="21" /> However, file alignment turned out to be cumbersome to manipulate, and in edition 1.1 we coordinated with UD to propose a generic extension to the CoNLL-U format, CoNLL-U Plus.<fnref n="22" /> The CUPT file, currently used to represent the corpora, is an instance of the CoNLL-U Plus specification. CUPT stands for CoNLL-U + PARSEME TSV, and consists of the horizontal concatenation of CoNLL-U files and the last column of the original PARSEME-TSV files, containing MWE annotations.

Figure 4.12 illustrates a French sentence using the CUPT format. The header appears only in the first line of each file, and contains the names of each column. The text meta-data contains the raw (untokenised) text of the sentence. Tokens are listed, one per line, with linguistic annotations in separate columns (morphological features appear truncated in the figure for better readability).

The PARSEME:MWE column encodes information about MWEs. It contains a star '*' for tokens which are not part of a MWE, (or multiword tokens), or an underscore '_' if this information is underspecified. Otherwise, it contains a list of semicolon-separated MWE codes if the current word is a lexicalised component of one or more MWEs. The MWE code of the first lexicalized component in the sentence consists of a VMWE identifier followed by a colon ': ' and a MWE category label, for example: 1:LVC-full. MWE identifiers are integers starting from 1 for each new sentence, and increased by 1 for each new MWE. MWE category labels are strings corresponding to the category of the VMWE, as described in §4.2.1.1. MWE codes of lexicalized components other than the first one contain the VMWE identifier only, and no category label. The full format specification can be found at http://multiword.sourceforge.net/cupt-format/.

**Data splits** In a shared task, all participating systems are evaluated on the same test data, kept secret (blind) during the evaluation phase. As organisers, we had to split the original corpora of each language into two parts. The *training* set (abbreviated as *train*) was shared with participants well in advance, whereas the *test* set was shared at the last moment, for a few days.

In edition 1.0, each corpus was split into train/test sets using a method chosen
individually for each language, given the heterogeneous nature of the corpora.
For all languages, tried to observe the following criteria:

<footnote n="21">²¹https://typo.uni-konstanz.de/parseme/index.php/2-general/184-parseme-shared-task-format-of-the-final-annotation</footnote>

<footnote n="22">²²https://universaldependencies.org/ext-format.html</footnote>
1. The test corpus contained around 500 annotated VMWEs,

2. The test corpus did not overlap with the trial data,

3. Sentences from the end of the corpus are more likely to land in the test set.

In edition 1.1, the splitting strategy was revised, correcting some flaws detected in edition 1.0. First, we decided to split each corpus into training, *development* (abbreviated as *dev*), and (blind) test set. Thus, participants could use a uniform setting to develop their systems and tune/evaluate them on the *dev* set. The only exceptions were Hindi and English, too small to extract a separate *dev* set.

Second, for each language, we took into account the origin of the sentences, that is, their sub-corpus (usually corresponding to a domain/register). Our splitting method ensured that the fraction of each sub-corpus is the same in all corpus parts (train/dev/test). For example, around 59% of all Basque sentences came from UD, while the other 41% came from the sub-corpus Elhuyar. We have made sure that similar proportions are kept in the train/dev/test sets.

Third, we defined rules to split the data depending on the total number of annotated VMWEs. For corpora with 550 VMWEs or less, we took 90% as test set, and 10% as a small training set. For corpora between 1,500 and 5,000 VMWEs, we took sentences such that 500 VMWEs are in the test set, 500 in the dev set, and the rest in the training set. Larger corpora are split using a 80%/10%/10% ratio for the train/dev/test sets. Due to these updates, for most languages, we did not keep the VMWEs in the same split as in edition 1.0.

Analyses of the results of the first two editions indicated that the ratio of unseen VMWEs in the test corpus with respect to the train+dev corpora is highly correlated with the performance of VMWE identification systems (Maldonado & QasemiZadeh 2018; Saied et al. 2018; Savary, Cordeiro, Lichte, et al. 2019).<fnref n="23" /> Thus, edition 1.2 focused on the performance of systems precisely on unseen expressions. However, some datasets in edition 1.1 contained very few unseen VMWEs.<fnref n="24" /> Using them as is would lead to statistically unreliable assessment of systems' performance on unseen VMWEs. As a consequence, we had to design a strategy to re-split the corpora controlling for the distribution of unseen VMWEs.

The most natural candidate for this new splitting criterion would be to homogenise the ratio of unseen VMWEs across languages. Therefore, we performed a preliminary study using data from edition 1.1. Figure 4.13 shows the ratio of unseen VMWEs as a function of the train+dev corpus size. The ratio of unseen VMWEs varies widely across languages, even when controlling for train+dev

<footnote n="23">²³The notion of “unseen” VMWEs is formally defined in §4.3.2.</footnote>

<footnote n="24">²⁴E.g. Romanian, Basque, and Hungarian contained 26, 57, and 62 unseen VMWEs in the test set.</footnote>
## 4 Down-to-earth MWE identification

<figure><img src="image_25.png" /><figcaption>Figure 4.13: Unseen ratios as a function of train+dev size in version 1.1.</figcaption></figure>

size. We hypothesise that it depends not only on the target language and corpus size, but also on other factors such as genre, domain, etc.

This preliminary study dissuaded us from controlling for a “natural” unseen ratio. Instead, we decided to target roughly the same absolute number of unseen VMWEs per language, while test set size and unseen ratio follow naturally. This criterion gives equal weights to each language in system evaluation.

The splitting algorithms has two parameters: the number of unseen VMWEs in test with respect to train+dev, and the number of unseen VMWEs in dev with respect to train. The latter ensures that dev is similar to test, so that systems tuned on dev have similar performances on test. The same procedure is applied to split the whole corpus into test and train+dev, and then to split train+dev into train and dev. The procedure takes as input a set of sentences, a target number of unseen VMWEs $u_t$, and a number $N$ of random splits:

* We use binary search to estimate $s_t$, the size of the target test set leading to the desired value of $u_t$. In this search, for a given test size $s_t^i$, the estimated number of unseen VMWEs is averaged across $N$ random splits of size $s_t^i$.

* Once $s_t$ is found, we compute the average unseen ratio $r_t$ over the $N$ splits.

* Among the $N$ random splits of size $s_t$, we pick the one that best fits $u_t$ and $r_t$ by minimising the cost function $c(u, r, u_t, r_t) = |u_t - u|/u_t + |r_t - r|/r_t$.
## 4.3 The PARSEME galaxy: shared tasks

We applied this algorithm to all languages, with *N*=100, *u*<sub>t</sub>=300 (test) and *u*<sub>t</sub>=100 (dev). This means that all our test sets contain around 300 unseen VMWEs with respect to the train+dev sets. Due the heterogeneous nature of each language's corpora, the unseen ratios vary significantly, from 7% (Romanian) to 69% (Irish). The training, development and test sets of each edition, as well as the evaluation scripts, were published under open licences on LINDAT.<sup>25</sup>

**Tracks** In the three editions shared task, participants could submit results in two tracks: closed and open. Closed-track systems rely on the training corpus, nothing else. Open-track systems can use any extra resource such as MWE lexicons, thesauri, pre-trained embeddings, pre-trained language models, and so on.

In edition 1.2, each language team also prepared a large “raw” corpus, annotated for morphosyntax but not for VMWEs.<sup>26</sup> The raw corpora were part of the data authorised in the closed track. Their sizes range from 12.7 to 2,474 millions of tokens. The genre of the data depends on the language, but efforts were put into making them consistent with the annotated data. The most frequent sources are CoNLL 2017 shared-task raw data,<sup>27</sup> Wikipedia and newspapers. Raw corpora, uniformly released in the UD format, were meant for discovering unseen VMWEs, which were the focus of this edition. However, participants ended up rarely using this resource, preferring pre-trained language models and thus participating in the open track.

### 4.3.2 Evaluation metrics are not rocket science

At the evaluation phase, participants get the blind test set, containing sentences like in Figure 4.12 where the 11th column was replaced by ' _ ' . Their goal is to produce the corresponding codes for identified VMWEs, or asterisks for words not belonging to VMWEs. These predictions are then compared to the reference gold annotations in the corpus, and several evaluation metrics are calculated, as described below. One of the scientific contributions of the PARSEME shared tasks was the definition of new evaluation measures for MWE identification.

All metrics described below are independently calculated for each language, and a cross-language aggregate is obtained by macro-averaging them across languages. That is, overall precision *P* and recall *R* are the averages of precision and

<footnote n="25"><sup>25</sup>edition 1.0: http://hdl.handle.net/11372/LRT-2282, edition 1.1: http://hdl.handle.net/11372/LRT-2842, edition 1.2: http://hdl.handle.net/11234/1-3367.</footnote>

<footnote n="26"><sup>26</sup>http://hdl.handle.net/11234/1-3416</footnote>

<footnote n="27"><sup>27</sup>http://hdl.handle.net/11234/1-1989</footnote>
## 4 Down-to-earth MWE identification

recall across all languages. The global F-score is calculated from these averaged *P* and *R* values. Missing system predictions are assumed to have *P* = *R* = 0.

**VMWE-based evaluation** The quality of predictions is measured with the standard metrics of precision (*P*), recall (*R*) and F-score (*F*). Only the MWE span is considered, categories (e.g. LVC full, VID, IRV) are ignored by the evaluation metrics. Categories are only provided in the training data to guide system design, but we did not require systems to predict them.<fnref n="28" />

<table><caption>Table 4.3: Toy gold corpus with 3 tokens, 2 gold VMWEs, and 3 system predictions. VMWE codes omit categories. Full matches in bold colour, partial matches in light colour. Adapted from (Savary et al. 2017).</caption><thead><tr><th>Token</th><th>Gold</th><th>System 1</th><th>System 2</th><th>System 3</th></tr></thead><tbody><tr><td>word1</td><td>1</td><td>1</td><td>1</td><td>1;4</td></tr><tr><td>word2</td><td>1</td><td>2</td><td>3</td><td>3</td></tr><tr><td>word3</td><td>2</td><td>2</td><td>2</td><td>2;4</td></tr></tbody></table>

To describe the evaluation metrics, we will consider Table 4.3, which presents a toy gold corpus containing one VMWE spanning over 2 tokens and one single-token MWE, and three aligned system predictions. These MWE codes can be seen as simplified versions of the 11th column in a CUPT file (Figure 4.12). If *G* denotes the set of gold VMWEs and *S*<sub>*i*</sub> the set of MWEs predicted by system *i*, then the following holds:<fnref n="29" />

$$
\begin{align*}
G &= \{\{\text{word1}, \text{word2}\}, \{\text{word3}\}\} & |G| &= 2 & ||G|| &= 3 \\
S_1 &= \{\{\text{word1}\}, \{\text{word2}, \text{word3}\}\} & |S_1| &= 2 & ||S_1|| &= 3 \\
S_2 &= \{\{\text{word1}\}, \{\text{word2}\}, \{\text{word3}\}\} & |S_2| &= 3 & ||S_2|| &= 3 \\
S_3 &= \{\{\text{word1}\}, \{\text{word2}\}, \{\text{word3}\}, \{\text{word1}, \text{word3}\}\} & |S_3| &= 4 & ||S_3|| &= 5
\end{align*}
$$

The *MWE-based scores* reward only full matches, considering every expression as an indivisible instance. MWE-based *P*, *R* and *F* correspond to the ratio of full MWEs that were correctly predicted (precision) and retrieved (recall). In Table 4.3, we show full matches in bold using different colours for the matched expressions. The MWE-based *P* and *R* scores for this example are shown below, with *TP*<sub>*i*</sub> being the number of true positives for system *i*:

<footnote n="28">²⁸Per-category results are provided, for discussion, for those systems which did predict them.</footnote>

<footnote n="29">²⁹Let *A* be a set of sets: $|A|$ is its size and $||A|| = \sum_{B_i \in A} |B_i|$ is the sum of the sizes of its elements.</footnote>
4.3 The PARSEME galaxy: shared tasks

$$TP_1 = |G \cap S_1| = |\emptyset| = 0 \quad \Rightarrow \quad R = 0, \quad P = 0$$

$$TP_2 = |G \cap S_2| = |\{\text{word3}\}| = 1 \Rightarrow R = TP_2/|G| = 1/2, P = TP_2/|S_2| = 1/3$$

$$TP_3 = |G \cap S_3| = |\{\text{word3}\}| = 1 \Rightarrow R = TP_3/|G| = 1/2, P = TP_3/|S_3| = 1/4$$

**Token-based evaluation** MWE-based scores were employed in the past to assess MWE identification (Constant & Tellier 2012; Schneider et al. 2014). However, they may be too strict, for instance, for MWEs containing articles and prepositions, whose lexicalisation is also challenging for humans. We would like to reward systems predicting parts of MWEs correctly, as opposed to completely wrong predictions. Such fuzzy-match score must be applicable to all MWE configurations, including discontinuous, single-token, overlapping and nested ones.

Evaluation metrics from the literature cover some of these aspects, but not all. For example, the link-based score of Schneider et al. (2014) is based on word pairs, accounts for discontinuities, but disallows single-token MWEs. The CEAF-M measure, used in coreference resolution, groups mentions into entities and finding the best bijection between gold and system entities (Luo 2005). However, coreference is an equivalence relation, i.e. each mention belongs to exactly one entity, whereas MWEs can exhibit overlapping and nesting.

Our proposed *token-based scores* consider all possible bijections between the MWEs in the gold and system sets, and takes a matching that maximizes the number of correct token predictions (true positives, denoted below as $TP_i^*$ for each system i). Partial matches are shown in light colour in Table 4.3. The application of this metric to the example is as follows:

$$TP_1^* = |\{\text{word1, word2}\} \cap \{\text{word1}\}| + |\{\text{word3}\} \cap \{\text{word2, word3}\}| = 2$$

$$R = TP_1^* / ||G|| = 2/3 \quad P = TP_1^* / ||S1|| = 2/3$$

$$TP_2^* = |\{\text{word1, word2}\} \cap \{\text{word1}\}| + |\{\text{word3}\} \cap \{\text{word3}\}| = 2$$

$$R = TP_2^* / ||G|| = 2/3 \quad P = TP_2^* / ||S2|| = 2/3$$

$$TP_3^* = |\{\text{word1, word2}\} \cap \{\text{word1}\}| + |\{\text{word3}\} \cap \{\text{word3}\}| = 2$$

$$R = TP_3^* / ||G|| = 2/3 \quad P = TP_3^* / ||S3|| = 2/5$$

Formally, let $G = \{g_1, g_2, ..., g_{|G|}\}$ and $S = \{s_1, s_2, ..., s_{|S|}\}$ be the ordered sets of gold and system MWEs in a sentence. Let $B$ be the set of all bijections $b: \{1, 2, .., N\} \to \{1, 2, .., N\}$, where $N = \max(|G|, |S|)$. We define $TP^*$ as maximum number of true positives for any possible bijection: $TP^* = \max_{b \in B} |g_1 \cap s_{b(1)}| + |g_2 \cap s_{b(2)}| + ... + |g_N \cap s_{b(N)}|$. We add up $TP^*$ for all $M$ sentences in a corpus. The global token-based precision is the ratio between the overall token-based true positives $\sum_{j=1}^M TP_{(j)}^*$ and the total number of predicted tokens $\sum_{j=1}^M ||S_{(j)}||$. Recall uses the same numerator, and $G_{(j)}$ instead of $S_{(j)}$ in the denominator.
## 4 Down-to-earth MWE identification

Finding the optimal bijection corresponds to finding the maximum weighted bipartite matching, also called “balanced assignment problem”. While the naive solution has $O(N!)$ complexity, we use the Kuhn-Munkres algorithm (Hungarian method), which has a theoretical complexity of $O(N^3)$.<fnref n="30" /> In practice, the number of MWEs in a sentence $N$ is small (<< 20), so the evaluation is fast. MWE-based and token-based scores are described in detail in Savary et al. (2017).

<figure><img src="image_26.png" /><figcaption>Figure 4.14: A *seen* MWE shares its multi-set of lemmas with an annotated MWE in the training set, regardless of order and inflection.</figcaption></figure>

### Phenomenon-specific scores

Orthogonally to the type of score (MWE-based or token-based), edition 1.1 introduced phenomenon-specific scores, by evaluating the systems only on the subset of expressions which represent a specific (linguistic) phenomenon. That is, phenomenon-specific scores are MWE-based precision, recall and F-scores calculated on a subset of gold and predicted MWEs that present a given characteristic. Our 4 pairs of phenomenon-specific scores focus on challenging MWE characteristics (Constant et al. 2017):

* **Novelty**: We obtain MWE-based scores for two subsets: *seen* and *unseen* MWEs. A MWE from the (gold or prediction) test corpus is considered seen if a MWE with the same multi-set of lemmas is annotated at least once in the training or the development corpus, as shown in Figure 4.14. Otherwise, the MWE is considered unseen. For instance, given the occurrence of *en* *has a new look* in the training or in the development corpus, the test instances *en* *had a look of innocence* and *en* *having a look at this report* would be considered seen and unseen, respectively.

* **Variability**: We calculate MWE-based scores for two subsets of seen MWEs: *identical* forms and *variants* of MWEs from the training set. A MWE is considered a variant if: (i) it is a seen MWE as defined above, and (2) it is not identical to any MWE in the training corpus. Two MWE occurrences

<footnote n="30">³⁰Implemented in this Python library: https://github.com/bmc/munkres</footnote>
## 4.3 The PARSEME galaxy: shared tasks

are considered identical if the strings between their first and last lexicalized components, including non-lexicalized elements in between, are identical. For example, `bg накриво` ли беше `стъпил` is a variant of `стъпя накриво` (lit. 'to step to the side') 'to lose (one's) footing'.

* **Continuity:** We obtain MWE-based scores for two subsets: *discontinuous MWEs*, such as `[sl imajo investicijske načrte` (lit. 'they-have investment plans') 'they have investment plans', and *continuous ones*, like `[tr istifa edecek` (lit. 'resignation will-do') 'they SG will resign'.

* **Length:** We obtain MWE-based scores for two subsets: *single-token MWEs*, e.g., `[de anfangen` (lit. 'at-catch') 'begin', `[es abstenerse` (lit. 'abstain-REFL') 'abstain', and *multi-token ones*, e.g., `[fr je jette un oeil` (lit. 'I throw an eye') 'I look at'.

The phenomenon-specific scores are detailed in Ramisch, Cordeiro, et al. (2018) and on the shared task website.<fnref n="31" /> Unseen scores were used to rank systems in edition 1.2 of the shared task (Ramisch et al. 2020). Specific methods for MWE identification have been proposed targeting discontinuous (Taslimipoor et al. 2019) and variant MWEs (Pasquer, Savary, Antoine & Ramisch 2018).

### 4.3.3 MWE identification systems go into orbit

The PARSEME shared tasks attracted many participants and gave a new impulse to the development of MWE identification. In this section, I will briefly overview a sample of these systems in whose design and development I was involved. First, I will describe Veyn, a neural sequence tagging system (§4.3.3.1). Then, I will present VarIde, a system focusing on variant identification, also (§4.3.3.2).

#### 4.3.3.1 Veyn: mind the gap

Veyn is an MWE identification system based on sequence tagging and recurrent neural networks developed by Nicolas Zampieri during his masters internship under my supervision.<fnref n="32" /> It represents VMWEs using a variant of the begin-inside-outside encoding scheme combined with the VMWE category tag. We previously observed promising results with another in-house system using the sequence tagging paradigm, based on conditional random fields (Scholivet & Ramisch 2017).

<footnote n="31">³¹https://multiword.sourceforge.net/PHITE.php?sitesig=CONF&page=CONF_04_LAW-MWE-CxG_2018 lb_COLING_rb_&subpage=CONF_50_Evaluation_METRICS</footnote>

<footnote n="32">³²Freely available at: https://github.com/zamp13/Veyn</footnote>
## 4 Down-to-earth MWE identification

<figure><img src="image_27.png" /><figcaption>Figure 4.15: Veyn's architecture: lemma and POS embeddings are concatenated and passed through 2 stacked bidirectional GRU layers. BIO-cat labels are predicted using a softmax over the concatenated outputs of the top left and right GRU cells. Credits: Nicolas Zampieri.</figcaption></figure>

We use each token's lemma (CoNLL-U's LEMMA column) and universal part of speech (UPOS) as input features, falling back to surface forms (FORM) and language-specific POS tags (XPOS) if the former are absent. Each token's lemma and POS are represented as embeddings, concatenated and forwarded to a double-stacked bidirectional recurrent layer using gated recurrent units (GRU). Veyn's architecture is illustrated in Figure 4.15. We compare several strategies to preinitialise the input representations with different embedding models (see below).

To speed up training, we crop sentences longer than 128 tokens (400 out of 317,816 sentences – 0.13%). We use the following hyper-parameters in our submission: input embeddings of dimension 250, hidden recurrent state of dimension 512, Nadam optimizer, 10 epochs, batches of size 128, and no drop-out. We used the Python library Keras to implement our system, using Tensorflow as backend.

In the system submitted to the shared task, the output for each token is a probability distribution over the possible BIO-based tags using softmax activation and trained with the categorical cross-entropy loss function (Zampieri et al. 2018). Alternatively, we have later introduced the use of a CRF layer to take into account tag sequence probabilities at the output layer (Zampieri et al. 2019). In both cases, we choose the most probable label in a greedy fashion. VMWEs predictions are then reconstructed on the output based on heuristic rules that group 'B' and 'I' tags with the same category, falling back to 'O' in case of incompatible tags.

As explained in §4.1.2, the BIO scheme uses 'B' tags for the beginning of an
expression, 'I' tags for its subsequent components, and 'O' for tokens outside
the VMWE. BIO was originally designed for continuous sequences, so we use a
special label 'G' for gap tokens, not belonging to an expression, but occurring
in between the VMWE's components, as proposed by Schneider et al. (2014).<fnref n="33" />
BIO does not allow representing overlaps, that is, tokens belonging to more than
one VMWE at the same time. These are very rare (0.34% of the tokens across
all languages). We deal with overlaps by duplicating the sentence containing an
overlap, and adding a different annotation to each copy.

<figure><img src="image_28.png" /><figcaption>Figure 4.16: Three tagging schemes for an example sentence in French.
Adapted from: Zampieri et al. (2018)</figcaption></figure>

One of our goals was to evaluate different tagging schemes and choose the
best one based on the development corpus performances. Therefore, in addition
to the extended BIO scheme, we also tested an adaptation that includes category
labels (BIO+cat) concatenated with 'B' and 'I' labels. Because categories are
quite heterogeneous, it might be a good idea to let parts of the neural network
specialise to predict them separately. This is illustrated in the last row of Fig-
ure 4.16. Finally, we also evaluated our system using a simpler inside-outside
scheme (Klyueva et al. 2017). This scheme does not distinguish the token that
begins an expression from the others (IO+cat).

Figure 4.17 shows the cross-lingual macro-averaged MWE-based and token-
based F-scores for the three tagging schemes. In addition, the fourth bar shows
the performance of the BIO+cat encoding when input lemma and POS represen-
tations are pre-initialised (PI) with 250-dimensional skip-gram embeddings ob-
tained with word2vec applied on the training corpus.<fnref n="34" /> BIO+cat tagging yields
higher average MWE-based F-scores (41.56 for BIO+cat vs. 38.96 for BIO), but
BIO yields higher Token-based F-score (50.88 for BIO+cat vs. 53.90 for BIO). A
similar trend is observed when comparing IO+cat and BIO+cat, with IO+cat
performing better than BIO+cat on Token-based evaluation (52.13 for IO+car
vs. 50.88 for BIO+cat). Both BIO and IO+cat use reduced tagsets with respect
to BIO+cat, and this probably helps recognise words that are parts of an expres-
sion. However, these tagsets are worse at predicting full expressions. Therefore,

<footnote n="33">³³They use underscore 'o' for gaps, with the same interpretation.</footnote>

<footnote n="34">³⁴Pre-training on larger external corpora would prevent us from participating in the closed track.</footnote>
## 4 Down-to-earth MWE identification

we chose BIO+cat as our submission tagging scheme for all languages, assuming that MWE-based evaluation is prioritary. Moreover, pre-initialisation seems to systematically help for both scores, so we also adopt it in our submission.

<figure><img src="image_29.png" /><figcaption>Figure 4.17: Cross-lingual average F-scores for tagging schemes and pre-initialisation (PI) on dev set. Adapted from: Zampieri et al. (2018)</figcaption></figure>

We submitted results with Veyn for 19 out of 20 languages of the PARSEME shared task edition 1.1 (one language was ignored due to licencing issues). The macro-averaged F-scores on the test corpus are 36.94% (vs. 46.68% on dev) for MWE-based scores and 44.9% (vs. 56.32% on dev) for token-based scores. The system was ranked ninth (eighth) on the average MWE-based (Token-based) F-score on the official ranking. More detailed analyses of the system performance are presented in Zampieri et al. (2018).

After the shared task, we continued improving the system, mainly to improve its stability by adding a CRF output layer, early stopping, and drop-out. We also studied the impact of input word representations on MWE identification performance (Zampieri et al. 2019). First, we select 3 languages within the PARSEME 1.1 corpora with more or less rich morphology: French, Polish and Basque. Then, we compare the use of word2vec and FastText embeddings to pre-initialise surface form and lemma representations in Veyn. The latter uses character n-grams to encode sub-lexical units, potentially generalising across morphological variants. We show that subword representations are indeed more efficient for MWE identification in these languages. Moreover, we have highlighted that the use of lemmas always has a positive impact on performance. For languages with high morphological richness like Basque, the concatenation of lemmas and forms outperforms the use of lemmas alone.

### 4.3.3.2 VarIde: separating the wheat from the chaff

The VarIde system is the final operationalisation of an extensive research on MWE variability performed as part of Caroline Pasquer’s PhD thesis, which I
co-advised. The first step in the development of VarIde was a corpus-based analysis of VMWE variation patterns (Pasquer 2017). These patterns depend on the VMWE category: for instance, LVCs present more variability than VIDs in terms of nominal inflection, relative clauses, extraction, and passivisation.

In Pasquer, Savary, Antoine & Ramisch (2018), we propose a metric to characterise the variability of a VMWE type (set of occurrences) based on variant-to-variant similarity. Variants are defined as VMWE occurrences that share the same multi-set of lemmas (“seen” MWEs in §4.3.2). Variant similarity is based on syntactic and linear similarity. Syntactic similarity is defined as the overlap between the outgoing syntactic dependencies of the VMWE component words, measured using the Dice coefficient, and averaged across components. Similarly, linear similarity is the Dice overlap of the set of POS tags that appear within each variant’s “gaps”, that is, the non-lexicalised components linearly inserted between the lexicalised ones. For a given VMWE type *E* with *m* variants, we define its rigidity *R*(*E*) as the average of the pairwise similarity for all (m/2) variant pairs, and its variability as the complement *V*(*E*) = 1 − *R*(*E*). Our experiments on French noun-verb constructions show that:

1. Our variability metrics are highly correlated with a linguistic VMWE typology based on formal fixedness criteria (Tutin 2016).

2. Our metrics can distinguish VMWE categories presenting different variation patterns, e.g. LVC vs. VID.

3. True and false VMWE candidates show different variability distributions, indicating that it may be a useful feature for VMWE identification.

In Pasquer, Savary, Ramisch, et al. (2018), we propose a system designed specifically to identify VMWE variants. We define the subtask of *variant identification* as deciding whether a candidate is a true VMWE variant, or an instance that is not a VMWE at all, although it shares the same multi-set of lemmas with true VMWE instances. Again, our experiments focus on French verb-noun constructions of categories LVC and VID, with an optional lexicalised determiner between the verb and the noun, e.g. *faire une présentation* ‘make a presentation’ (LVC), *tourner la page* ‘make a new start after a difficult period’ (VID). The principle of this system is later generalised for all languages and VMWE categories in VarIde (Pasquer, Ramisch, et al. 2018). Our procedure has the following steps:

1. **Candidate extraction.** Given a training and a test corpus, we retrieve all lexeme combinations that are either annotated as VMWEs (positive candidates), or unannotated but sharing its multi-set of lemmas with at least one
## 4 Down-to-earth MWE identification

annotated VMWE in the training corpus (negative candidates). This step is the same for the training and test corpora, although in the test corpora the positive/negative category is disregarded. We apply POS filters to remove candidates with impossible POS sequences (e.g. VERB-NOUN-DET).

2. **Absolute features.** We extract, for each candidate, its characteristics in the sentence, such the POS of non-lexicalised insertions (gaps), the syntactic distance (number of dependencies) between verb and noun, and the noun's morphological features and outgoing dependencies.

3. **Comparative features.** Half of the training positive candidates are randomly chosen as development set. This is a reference against which we compare positive and negative candidates from the remaining training set, and from the test set. This leads to comparative features to indicate a match (or mismatch) in the syntactic distance, noun outgoing dependencies, etc.

4. **Classification** We learn a simple Naive Bayes classifier using the training candidates annotated with the absolute and comparative features. The idea is that this classifier will be able to infer the types of possible and impossible syntactic and morphological processes underlying VMWE variability, distinguishing them from those involved in non-VMWEs (mainly coincidental cooccurrences, as defined in §3.2.4).

In the VarIde system, we improve this model so that it can be applied to any language and POS sequence (Pasquer, Ramisch, et al. 2018):<fnref n="35" />

* Instead of focusing on verb-noun pairs, we cover all POS patterns.

* We filter out less plausible negative candidates based on length and POS pattern. The length filter excludes candidates whose lexicalised components are more than 20 tokens apart. The POS filter, excludes candidates if a particular POS order has never been observed for candidates that share the same set of POS. For instance, VMWEs whose POS set is {VERB,NOUN} (e.g. *make decisions*) allow both orders VERB-NOUN and NOUN-VERB, and no candidate is filtered out. On the other hand, candidates associated with {VERB,PRON} (e.g. *take it*), only appear in this order, so we will exclude candidates appearing in the PRON-NOUN order.

* Instead of splitting the training data into dev and training sets, comparative features are obtained by comparing a candidate with all other VMWEs in the training set that share the same multi-set of lemmas, except itself.

<footnote n="35">³⁵Freely available at: https://gitlab.com/cpasquer/SharedTask2018_varIDE</footnote>
* Absolute features are adapted to cover all VMWE components and features present in all languages. If a given feature is not applicable to a given MWE candidate, we assign it a special value -1.

VarIde was ranked 5th out of 13 submissions to the closed track of the shared task. It obtained a cross-lingual average F-score of 45.97 (MWE-based) and 47.43 (token-based), whereas the best system in this track obtained an average F-score of 54 (MWE-based) and 59.67 (token-based). Surprisingly, VarIde was ranked second for discontinuous VMWEs ($F = 37.4$). This is a very good result, given that the system only targets seen VMWEs (identical or variants), and cannot in theory predict any unseen VMWE.$^{36}$

<figure><img src="image_30.png" /><figcaption>Figure 4.18: VarIde's F-score for variants on shared task 1.1 languages. Hatched bars: candidate extraction; full bars: final classification.</figcaption></figure>

Figure 4.18 plots the MWE-based F-score of VarIde for the 19 covered languages, focusing only on variants (that is, seen VMWEs whose form is not identical in test and train). We show the F-score both before (hatched bard) and after (full bars) the application of the classifier. If we look at candidate extraction only, we obtain satisfactory coverage, with recall > 0.8, for 17 languages (0.62 and 0.75 for Italian and German). Moreover, extraction recall on variants depends on their proportion in corpora which varies from 12% (Romanian) to 83% (Lithuanian). Final classification performance for variants is sensitive to the reliability of the annotated corpora, being affected by both false positives (e.g. *UV lights.*NOUN *up*.VERB *the temperature* was falsely annotated, probably by analogy to *light*.VERB *up*.ADP) and false negatives. Imbalance between true and false candidates may also have a detrimental impact, either over-representation true VMWE candidates, as in Hungarian (92%) or the contrary, as in Turkish (4%). For

<footnote n="36">³⁶In practice, this may happen due to wrong lemmas and POS tags.</footnote>
## 4 Down-to-earth MWE identification

most languages, classification on top of extraction is beneficial, but sometimes the drop in recall is too drastic, leading to lower F-scores for the final system than for the initial candidate extraction (e.g. in German and Croatian).

### 4.3.4 Results: Houston, we got a problem

<table><caption>Table 4.4: Architecture of the systems, and their use of provided and external resources. Source: Ramisch et al. (2020).</caption><thead><tr><th rowspan="2">System</th><th rowspan="2">Architecture</th><th colspan="2">Use of corpora/resources</th></tr><tr><th>Train+dev corpus</th><th>External resources</th></tr></thead><tbody><tr><td>ERMI</td><td>bidirectional LSTM + CRF</td><td>train model</td><td>–</td></tr><tr><td>FipsCo</td><td>rule-based joint parsing+identification</td><td></td><td>VMWE lexicon</td></tr><tr><td>HMSid</td><td>syntactic patterns, association measures (AMs)</td><td>tune patterns/AMs</td><td>Raw corpus, idiom dataset</td></tr><tr><td>MTLB-STRUCT</td><td>neural language model, fine-tuned for joint parsing+identification</td><td>tune BERT</td><td>multilingual BERT</td></tr><tr><td>MultiVitamin</td><td>neural binary ensemble classifier</td><td>train classifier</td><td>XLM-RoBERTa</td></tr><tr><td>Seen2Seen</td><td>rule-based extraction + filtering</td><td>tune filters</td><td>–</td></tr><tr><td>Seen2Unseen</td><td>+ lexical replacement, translation, AMs</td><td></td><td>Google trans., wiktionary, raw corpus</td></tr><tr><td>TRAVIS-mono</td><td>neural language model, fine-tuned for</td><td>tune BERT</td><td>monolingual BERT</td></tr><tr><td>TRAVIS-multi</td><td>MWE identification</td><td></td><td>multilingual BERT</td></tr></tbody></table>

In this section, for the sake of concision, we only summarise some results of the latest shared task edition 1.2 (Ramisch et al. 2020). Analyses for editions 1.0 and 1.1 can be found in the respective shared task description papers Savary et al. (2017) and Ramisch, Cordeiro, et al. (2018). Moreover, individual system description papers can also offer more focused analyses of the results.

**A closer look at unseen VMWEs** Nine results were submitted to edition 1.2 of the PARSEME shared task, summarised in Table 4.4. They use recurrent neural networks (ERMI, MultiVitamin, MTLB-STRUCT and TRAVIS), candidate extraction based on syntax plus filtering (HMSid, Seen2Seen), and rule-based joint parsing and MWE identification (FipsCo). Annotated corpora are used for model training or fine-tuning, and for tuning patterns and filters. The provided raw corpora has been used by one system only, for pre-training word embeddings (ERMI). We expected that the teams would use the raw corpus for MWE discovery (Chapter 3)), but they may have lacked time to do so. External resources used include morphological and VMWE lexicons, external raw corpora, translation
## 4.3 The PARSEME galaxy: shared tasks

software, pre-trained non-contextual and contextual word embeddings, notably including pre-trained mono- and multi-lingual BERT.

<table><caption>Table 4.5: Unseen MWE-based, global MWE-based, and global token-based Precision (P), Recall (R), F-score (F1) and F1 ranking (#). Closed track above separator, open track below. Source: Ramisch et al. (2020).</caption><thead><tr><th rowspan="2">System</th><th rowspan="2">#Lang</th><th colspan="4">Unseen MWE</th><th colspan="4">Global MWE</th><th colspan="4">Global token</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>#</th><th>P</th><th>R</th><th>F1</th><th>#</th><th>P</th><th>R</th><th>F1</th><th>#</th></tr></thead><tbody><tr><td>ERMI</td><td>14/14</td><td>25.3</td><td>27.2</td><td>26.2</td><td>1</td><td>64.8</td><td>52.9</td><td>58.2</td><td>2</td><td>73.7</td><td>54.5</td><td>62.6</td><td>2</td></tr><tr><td>Seen2Seen</td><td>14/14</td><td>36.5</td><td>00.6</td><td>01.1</td><td>2</td><td>76.2</td><td>58.6</td><td>66.2</td><td>1</td><td>78.6</td><td>57.0</td><td>66.1</td><td>1</td></tr><tr><td>MTLB-STR.</td><td>14/14</td><td>36.2</td><td>41.1</td><td>38.5</td><td>1</td><td>71.3</td><td>69.1</td><td>70.1</td><td>1</td><td>77.7</td><td>70.9</td><td>74.1</td><td>1</td></tr><tr><td>TRAVIS-mu.</td><td>13/14</td><td>28.1</td><td>33.3</td><td>30.5</td><td>2</td><td>60.7</td><td>57.6</td><td>59.1</td><td>3</td><td>70.4</td><td>60.1</td><td>64.8</td><td>2</td></tr><tr><td>TRAVIS-mo.</td><td>10/14</td><td>24.3</td><td>28.0</td><td>26.0</td><td>3</td><td>49.5</td><td>43.5</td><td>46.3</td><td>4</td><td>55.9</td><td>45.0</td><td>49.9</td><td>4</td></tr><tr><td>Seen2Uns.</td><td>14/14</td><td>16.1</td><td>12.0</td><td>13.7</td><td>4</td><td>63.4</td><td>62.7</td><td>63.0</td><td>2</td><td>66.3</td><td>61.6</td><td>63.9</td><td>3</td></tr><tr><td>FipsCo</td><td>3/14</td><td>04.3</td><td>05.2</td><td>05.7</td><td>5</td><td>11.7</td><td>8.8</td><td>10.0</td><td>5</td><td>13.3</td><td>8.5</td><td>10.4</td><td>5</td></tr><tr><td>HMSid</td><td>1/14</td><td>02.0</td><td>03.8</td><td>02.6</td><td>6</td><td>04.6</td><td>04.9</td><td>04.7</td><td>6</td><td>04.7</td><td>04.8</td><td>04.8</td><td>6</td></tr><tr><td>MultiVit.</td><td>7/14</td><td>00.1</td><td>00.1</td><td>00.1</td><td>7</td><td>00.2</td><td>00.1</td><td>00.1</td><td>7</td><td>03.5</td><td>01.3</td><td>01.9</td><td>7</td></tr></tbody></table>

Table 4.5 shows the performance of the systems in the two tracks averaged across the 14 languages, and the number of languages they covered.<fnref n="37" /> Two system results were submitted to the closed track and 7 to the open track. Four systems covered all 14 languages.<fnref n="38" /> As this edition focuses on unseen VMWEs, these scores are presented first. In the open track, the best F-score obtained by MTLB-STRUCT (38.53) is by over 10 points higher the corresponding best score in the edition 1.1 (28.46, by SHOMA). These figures are, however, not directly comparable, due to differences in the languages covered in the two editions, the size and quality of the corpora. The closed-track system ERMI achieves promising results, likely thanks to embeddings pre-trained on the raw corpus.

Global MWE-based F-scores for all, both seen and unseen VMWEs, exceed 66 and 70 for the closed and open tracks, against 54 and 58 in edition 1.1. Like for the unseen scores, it is unclear how much of this difference owes to new/enhanced resources, different language sets, and novel system architectures. The second best score across the two tracks is achieved by a closed-track system (Seen2Seen) using non-neural rule-based candidate extraction and filtering. Global token-based

<footnote n="37">³⁷Full results: http://multiword.sourceforge.net/sharedtaskresults2020/</footnote>

<footnote n="38">³⁸Macro-averages for systems not covering some languages consider P=R=F1=0.</footnote>
## 4 Down-to-earth MWE identification

F-scores are often slightly higher than corresponding MWE-based scores. An interesting opposition appears when comparing the global scores with those for unseen VMWEs. In the former, precision is usually higher than recall, whereas in the latter, recall exceeds precision, except for 2 systems.

<figure><img src="image_31.png" /><figcaption>Figure 4.19: Relation between each language's performance and its unseen ratio (red) and number of VMWEs tokens in the training set (blue). X axis: best MWE-based F1 score. Blue Y axis: Number of VMWEs in training set. Red Y axis: Unseen ratio. Source: Ramisch et al. (2020)</figcaption></figure>

One finding from the previous shared task editions is that performance for a given language is better explained by the unseen ratio than by the size of the training set. This is even truer for edition 1.2, as we could measure a very high negative linear correlation between the highest MWE-based F-score for a given language and the unseen ratio for that language (Pearson coefficient = -0.90). In contrast, the correlation between the best F-score and the size of the corpus in terms of number of annotated VMWEs in the training set is quite poor (Pearson coefficient = 0.23). Appendix Figure 4.19 plots these correlations graphically.

## 4.4 In short

MWE identification is a complex galaxy composed of multiple resources, concepts, systems, documents, results, etc. Our journey through this galaxy started with a chronological overview of the MWE identification literature prior to the birth of PARSEME. We started our timeline by mentioning influential work in grammar engineering and parsing, before addressing sequence tagging models. The core of these models is composed of statistical models like conditional random fields, combined with tagging schemes like begin-inside-outside (Constant & Sigogne 2011). External lexical resources can also play an important role
in tagging approaches (Riedl & Biemann 2016). Parsing-based methods constitute a completely different approach, in which MWE identification and parsing are entangled. They range from simple pre-tokenisation of MWEs as words-with-spaces (Nivre & Nilsson 2004) complex transition-based methods for joint dependency parsing and lexical segmentation (Constant & Nivre 2016).

Two initiatives were game changers for MWE identification: DiMSUM and PARSEME. DiMSUM was the first shared task focusing on MWE identification, releasing annotated corpora for English, and evaluating systems based on their ability to predict MWEs and supersenses (Schneider et al. 2016). PARSEME was originally a COST Action before it evolved into an international community developing MWE-annotated corpora and MWE identification tools.

The PARSEME corpora follow centralised multilingual guidelines targetting verbal MWEs (VMWEs) only. These guidelines describe basic definitions (e.g. lexicalised components), a set of VMWE categories, associated decision trees, and editable multilingual examples. PARSEME-FR is a French spin-off project which extended and completed the PARSEME guidelines other MWE categories, and for named entities. The PARSEME-FR guidelines were designed for French and specify a set of sufficient criteria for MWE annotation. As most members of PARSEME-FR were also involved in the international PARSEME initiative, both projects share a common technical environment: the FLAT annotation platform, a tool for consistency checks, an adjudication interface, tools for corpus format and stats, and the corpus query interface Grew-match. The PARSEME corpora has several releases covering 26 languages in total, with a total of 62K to 79K annotated VMWEs, depending on the release. The PARSEME-FR corpus is much smaller, with 6,579 annotations among which 3,128 are named entities and 3,451 are MWEs, including verbal and non-verbal ones.

The PARSEME shared tasks are evaluation campaigns which happened in 2017, 2018 and 2020, based on the corpora described above. Among some major contributions of the share tasks, we underlined the CUPT file format for MWE-annotated corpora, the data splitting methodology employed for controlling the amount of unseen VMWEs, and the MWE-based, token-based, and phenomenon-specific evaluation metrics and associated software. Systems submitted to the shared task use various models and resources. We described two examples: Veyn, a sequence tagger using recurrent neural networks, and VarIde, a system based on candidate extraction, feature engineering and classification. We presented the results of the latest shared task edition, in which we observe a high correlation between system performance and the ratio of unseen VMWEs. This indicates that there is much room for improvement in terms of generalisation for this task, if we want to reach truly universal MWE identification.
4.5 For the record

The chronological overview in 4.1 was inspired from my contributions to two survey articles (Constant et al. 2017), and Constant et al. (2019) and one system description paper submitted to the DiMSUM shared task (Cordeiro, Ramisch & Villavicencio 2016b).

PARSEME is a collective scientific adventure involving a team of experts. Much of the contents of this chapter are the fruit of joint work with Agata Savary, Silvio Cordeiro, Marie Candito, Jakub Waszczuk, Bruno Guillaume, only to cite a few most prominent collaborators. The PARSEME shared tasks are described in three articles, for editions 1.0, 1.1 and 1.2 (Savary et al. 2017; Ramisch, Cordeiro, et al. 2018; Ramisch et al. 2020), and one article focusing on the corpora (Savary et al. 2018). The PARSEME-FR corpus has been described in a short paper (Candito et al. 2017) and in a longer article (Candito et al. 2021). I co-authored a linguistic analysis of the Brazilian Portuguese PARSEME corpus, not covered in this chapter (Ramisch, Ramisch, et al. 2018). Another work not covered here is our position paper in which we hypothesise that MWE identification generalisation should rely on lexicons complementing (or replacing) annotated corpora (Savary, Cordeiro & Ramisch 2019). Detailed information about PARSEME can be found on the website of the PARSEME corpora.<fnref n="39" />

Veyn was developed by Nicolas Zampieri during his masters, based on Manon Scholivet’s preliminary work (Scholivet & Ramisch 2017). The system was first described in a shared task paper (Zampieri et al. 2018) and later extended and used to study input representations in recurrent neural models (Zampieri et al. 2019). The PhD of Caroline Pasquer was dedicated to studying the variability of MWEs. From a first linguistic characterisation (Pasquer 2017) evolved the idea of objective metrics for MWE variability (Pasquer, Savary, Antoine & Ramisch 2018). Then, we developed a system for variant identification in French (Pasquer, Savary, Ramisch, et al. 2018) and adapted it to other languages (Pasquer, Ramisch, et al. 2018). §4.3.3 adapts and reuses contents from all these publications. Publications not covered in this chapter include a study on feature selection for variant identification (Pasquer, Savary, Antoine, Ramisch, et al. 2020), and a variant identification system in which the classifier was replaced by simple rules that can be turned on/off for each language (Pasquer, Savary, Ramisch, et al. 2020b,a).

<footnote n="39">³⁹https://gitlab.com/parseme/corpora/-/wikis/</footnote>
# 5 The big picture

> *Quand je serai grand j'écrirai moi aussi les misérables parce que c'est ce qu'on écrit toujours quand on a quelque chose à dire.<fnref n="1" />*

> — Romain Gary (Émile Ajar), *La vie devant soi*

When Alice goes down the rabbit hole in Lewis Carroll's famous novel, she uncovers a world of wonders and dangers, a beautiful chaos full of new adventures. Like Alice, I went down the rabbit hole of multiword expressions almost twenty years ago. This manuscript tells a story of what I found there. Now, it is time to look at the big picture to summarise these findings (§5.1 and §5.2). Then, the story ends with an exploration of promising avenues for future research, and speculations on what we will discover after going through the looking-glass (§5.3).

## 5.1 Summary of MWE contributions

Most of my work assumes *explicit* and *linguistically informed* MWE representations in NLP as its underlying hypotheses. Compositionality prediction (Chapter 3) and MWE identification (Chapter 4) are the two main tasks on which I worked, which structured the previous chapters. Here, we adopt an orthogonal perspective, summarising these contributions along the following axes: theoretical framework (§5.1.1), methodological framework (§5.1.2), empirical results (§5.1.3), resources (§5.1.4), and software (§5.1.5).

### 5.1.1 Theoretical framework

My theoretical contributions are summarised here in terms of linguistic concepts, tests and definitions put forward and, to some degree, adopted by the community.

<footnote n="1">¹When I grow up I'm going to write my own *Les Misérables*, because that's what people always write if they have anything to say.</footnote>
**MWE definitions** The MWE definition by Baldwin & Kim (2010) is the basis for the one proposed by PARSEME guidelines (§4.2.1.1) and in my own work (§2.1.2). Except for slight variations, this definition is now quite consensually accepted. In addition, the PARSEME guidelines specify notions such as lexicalised components, meaning-preserving variants, canonical forms, and unexpected meaning change (§4.2.1.1). Our work on MWE ambiguity (§3.2.4) proposes a formal definition of MWE token, type, and variants, relying on more general mathematical objects such as subsequences and subgraphs. We also propose a formalisation of the MWE ambiguity phenomenon via the notions of (coarse) canonical structure, literal occurrence, idiomatic occurrence, and coincidental occurrence. These definitions also clarify the status of MWEs with respect to related phenomena such as metaphors, collocations, constructions, and with respect to underlying syntactic formalisms such as Universal Dependencies (de Marneffe et al. 2021)

**Guidelines** I invested considerable effort in the creation of annotation guidelines for verbal MWEs across languages §4.2.1.1, for MWEs and named entities in French §4.2.1.2, for nominal compound compositionality §3.2.3.1, for literal and coincidental occurrences §3.2.4. Within these guidelines, I highlight the verbal MWE categories in PARSEME, the distinction of MWEs and named entities in PARSEME-FR, and the reformulation of compositionality annotation for crowdsourcing. Furthermore, I participated in a proposal for annotating and representing functional MWEs in French, not covered in this manuscript (Ramisch, Nasr, et al. 2016)

**Task definitions** In Constant et al. (2017), we define a foundational framework for *MWE processing tasks*, distinguishing identification and discovery (§2.2). This distinction was influential at the time of publication because terminology was unstable concerning tasks. Moreover, in Cordeiro et al. (2019) we formalise the task of compositionality prediction, contributing to more systematic modelling of this complex phenomenon (§3.1). Both works were published in the Computational Linguistics journal, benefiting from its visibility.

Finally, this manuscript itself includes original theoretical contributions in Chapter 1, such as new MWE typology (§2.1.4) and motivations for MWE research (§2.3). Put together, these theoretical proposals contribute to more accurate linguistic descriptions, which in turn can be applied in MWE resource construction and system results analyses.
5.1.2 Methodological framework

A methodological framework, with established conventions and available tools,
is of utmost importance to assess progress in NLP. In practice, such frameworks
often emerge from shared tasks. As a co-organiser of the PARSEME shared tasks,
my work collaborates to building a methodological framework in MWE identifi-
cation, covering data and evaluation.

**Data** In collaboration with UD, we defined a standard to extend CoNLL-U files, of which the CUPT format is an instance (§4.3.1). MWE annotations are seen as sets of (potentially non adjacent) token indices within a sentence. This flexible representation allows for overlapping, single-token, and nested MWEs, which were not possible in the past. A set of tools are provided to deal with this data (FLAT for corpus annotation, consistency checks, CUPT python library, etc.). Moreover, the PARSEME shared tasks encourage participants to model MWE identification using supervised or semi-supervised machine learning. Hence, corpora in several languages are are uniformly split into training, development and test parts. A focus on data splitting stategies allows studying the generalisation of MWE identification with respect to unseen MWEs (Ramisch, Cordeiro, et al. 2018).

**Evaluation** The PARSEME shared tasks propose evaluating MWE identifica-
tion using MWE-based (full match) and token-based (approximate) metrics. The
latter take into account the fact that CUPT allows representing single-token,
overlapping, and nested MWEs (§4.3.2). Besides, we defined focused metrics to
study the performance of systems specifically for discontinuous, single-token,
variable and unseen MWEs. These metrics, along with the script that implements
them, contributes to more rigorous system comparisons and deeper error analy-
ses. In my current work, not included in this manuscript because of its recent pub-
lication, we survey methodological choices in MWE identification experiments
and propose a tool to estimate the statistical significance of system differences
(Ramisch et al. 2023). Evaluation is an important aspect of MWE processing and
of NLP in general, and is part of my interests for future work (5.3.3).

5.1.3 Empirical results

Experiments are a central component of nowadays mainstream data-intensive
NLP. Here, I highlight empirical results which are representative of my work.
5 *The big picture*

* In Nasr et al. (2015), we show that a *dependency parser* can identify highly ambiguous *functional MWEs* in French using special dependency labels. Adding the valency of the preceding verb as a feature considerably improves the performance of MWE identification scores. A similar result, although with smaller performance gains, was obtained when adding automatically predicted compositionality scores to a CRF to identify the same kind of constructions (Scholivet et al. 2018).

* Our experiments in *compositionality prediction* confirm that idiomaticity can be modelled accurately using word embeddings (§3.3.3). However, the choice of embedding model has a noticeable impact on results (Cordeiro et al. 2019). Lemmatisation is important for morphologically richer languages, but can be skipped for English. A billion-word corpus is sufficient to obtain top results for our datasets. Compositionality was found to be highly correlated with frequency, but not with conventionality (PMI).

* Variability can be used as a feature to distinguish MWEs from regular word combinations §4.3.3.2. In Pasquer, Savary, Antoine & Ramisch (2018), we propose metrics to characterise the variability of a VMWE type based on variant-to-variant similarity. Our metrics are highly correlated with a linguistic VMWE typology based on formal fixedness criteria, and can distinguish VMWE categories presenting different variation patterns, e.g. LVC vs. VID. This idea was implemented in VarIDE, a system for MWE identification which was ranked 5th out of 13 submissions in the closed track of the PARSEME shared task edition 1.1 (Pasquer, Ramisch, et al. 2018).

* Recently, we investigated the identification of MWEs in *non-standard language*: tweets in English (Zampieri et al. 2022). We found out that a fine-tuned transformer using a custom tagset outperforms a dictionary lookup baseline. Automatically identified MWEs are then used as features for hate speech detection, improving the performance of the downstream task.

5.1.4 Resources

My work contributed to the creation of freely available resources such as:

* Compositionality datasets containing 180 noun-noun and adjective-noun compounds in English, French, and Portuguese (§3.2.3). These were later extended to include single-word and multiword substitutes (§3.2.3.3).<fnref n="2" />

<footnote n="2">²https://doi.org/10.5281/zenodo.8296689</footnote>
* Corpora annotated for verbal multiword expressions in 26 languages as part of the PARSEME initiative (§4.2.3). My contribution, in addition to coordinating the language teams, includes the annotation of the Brazilian Portuguese corpora (Ramisch, Ramisch, et al. 2018).<fnref n="3" />

* Annotation of the French treebank Sequoia not only for verbal MWEs but also for other MWE categories, plus named entities (§4.2.1.2).<fnref n="4" />

* Fine annotation of literal, coincidental and idiomatic MWE occurrences in 6 languages, including Brazilian Portuguese (§3.2.4). This dataset was used to study the prevalence of literal readings and could be used in the future for in-context compositionality prediction.<fnref n="5" />

5.1.5 Software

We conclude this overview with a sample of open-source tools developed mostly
by master and PhD students under my (co-)supervision:

* The **mwetoolkit** was developed during my PhD to implement MWE discovery methods (Ramisch 2015).<fnref n="6" /> The tool was extended by Silvio Cordeiro to include the methods for compositionality prediction described in §3.3.2. Additionally, two MWE identification systems were added: a rule-based one, developed by Silvio Cordeiro and submitted to the DiMSUM shared task (Cordeiro, Ramisch & Villavicencio 2016b), and a CRF-based sequence tagging system, developed by Manon Scholivet (Scholivet & Ramisch 2017).

* Veyn is a deep learning system for MWE identification using stacked recurrent neural networks. It was developed by Nicolas Zampieri and participated at the PARSEME shared task edition 1.1 (§4.3.3.1).<fnref n="7" />

* VarIDE identifies MWEs by first selecting candidate expressions using POS patterns, then classifying them based on absolute and relative variability features §4.3.3.2. The system was developed by Caroline Pasquer and participated at edition 1.1 of the PARSEME shared task.<fnref n="8" />

<footnote n="3">³https://gitlab.com/parseme/corpora/-/wikis/home</footnote>

<footnote n="4">⁴https://deep-sequoia.inria.fr/</footnote>

<footnote n="5">⁵http://hdl.handle.net/11372/LRT-2966</footnote>

<footnote n="6">⁶https://mwetoolkit.sourceforge.net/</footnote>

<footnote n="7">⁷https://github.com/zamp13/Veyn</footnote>

<footnote n="8">⁸https://gitlab.com/cpasquer/SharedTask2018_varIDE</footnote>
* Seen2Seen and Seen2Unseen are two variants of a rule-based interpretable MWE identification system, also developed by Caroline Pasquer (Pasquer, Savary, Ramisch, et al. 2020b,a). It focuses on seen expressions and uses a set of 8 rules that can be turned on/off to match MWEs found in the training corpus.<fnref n="9" />

## 5.2 Summary of other NLP contributions

Throughout the years, other research questions also raised my scientific interest. MWEs were often the starting point for these questions, but the work belong to other subfields of NLP. This manuscript omits the details about these related contributions, but briefly mentions them below for the sake of completeness.<fnref n="10" />

**Word representations** Representing the semantics of word combinations is a challenge at the core of MWE research. Word embeddings (or vector space models, as we used to call them) are not only the dominant representation for (lexical) semantics, but are also pervasive in NLP. Thus, our work on compositionality prediction led us to studying the characteristics of the vectors used as word and MWE representations. In Padró et al. (2014a), we studied three aspects of count-based embedding models: frequency thresholds, similarity measures and target-context association scores. The study shows that frequency thresholds applied to contexts have a great impact on the models' stability, whereas models are quite insensitive to the choice of similarity score. This work was later extended, showing that keeping the top *k* most frequent target-context pairs is more effective than more sophisticated filters based on mutual information (Padró et al. 2014b).

More recently, we proposed a method to build lightweight interpretable contextualised embeddings using minimal supervision (Aloui et al. 2020). In this method, supervision takes the form of small lists of seed monosemous words for each coarse-grained Wordnet supersense. These are then used to train context classifiers, one per supersense. Once a new word is given to the model, each classifier assesses the context, giving it a score for each supersense. The union of all supersense scores forms the embedding for the word occurrence. Our evaluation on supersense tagging shows that we can get useful insights by looking at the dimensions of the embeddings associated with the predicted and gold supersenses.

<footnote n="9">⁹https://gitlab.com/cpasquer/st_2020</footnote>

<footnote n="10">¹⁰Another goal of this section is to adhere to the formal requirements of the present exercise, that is, enumerate the research contributions of the author to obtain an academic title.</footnote>
**Specialised frame extraction** In addition to computational models for lexical semantics, I explored compositional (frame) semantics for specialised language. Together with Beatriz Sánchez Cárdenas, we developed a methodology and tool to elicit verb-argument combinations from automatically parsed corpora. We rely on the **mwetoolkit** to extract recurrent co-occurring noun-verb-noun triples in various syntactic configurations. The triples are then manually annotated for thematic roles and semantic categories, and grouped into specialised semantic frames. When corpus co-occurrence queries return incomplete or insufficient results, word embeddings help capturing similar fillers. An analysis of specialised corpora in the environmental sciences domain in English and in Spanish illustrates our methodology (Cárdenas & Ramisch 2019).

**Multilingual dependency parsing** The PARSEME initiative currently covers VMWE-annotated corpora for 26 languages. Universal Dependencies is a similar larger-scape project, covering morphosyntax (POS, lemmas, morphology, syntactic dependencies) for more than 100 languages of the world. Both communities rely on a backbone consisting of multilingual annotation guidelines, designed to be as universal as possible, and then declined in different languages. Thus, multilingualism and cross-lingual generalisation is also part of my research interests.

Manon Scholivet's thesis studied deeply multilingual models for three sub-tasks of morphosyntactic analysis: POS tagging, morphological feature prediction, and dependency parsing (Scholivet et al. 2019). The underlying hypothesis is that abstract descriptions of each language, like those contained in the World Atlas of Language Structures (WALS), could be provided to machine learning models along with training corpora in several languages. These high-level descriptions would then guide the model to (a) generalise across languages with similar characteristics and (b) associate concrete linguistic descriptions (POS tags, syntactic relations) with their corresponding abstract features in the WALS.

We looked at different ways to use the WALS, but also at different representations for words: cross-lingual word embeddings and character models. Experiments were performed on a set of 41 languages from the UD collection, in a monolingual setting, a multilingual setting (concatenation of all languages) and a zero-shot setting (concatenation of all languages *except* the one on which the model was tested). Some interesting findings of this work are:

* In the zero-shot setting, the WALS is useful to analyse isolated languages, that is, those languages which do not share characteristics (lexicon, morphology, syntax) with any other language in the collection.
* For non isolated languages, the presence of one or more similar languages (e.g. from the same linguistic genus) is often more useful than the WALS.

* Character-based representations can harm zero-shot performance for un-related languages sharing the same writing system (e.g. Arabic and Urdu).

**Epidemiological event extraction** On the more applied side, I co-supervised the thesis of Léo Bouscarrat, in partnership with a company (CIFRE).<fnref n="11" /> The goal was to develop a prototype system for epidemiological surveillance based on the automatic extraction of epidemiological events from news feeds. In a first moment, the work focused on creating resources: multilingual ontologies (Bouscarrat et al. 2020) and a specialised corpus annotated for the target events (article under review). Then, we studied the feasability and stability of fine-tuning multilingual pre-trained language models, namely mBERT, on a similar task, that is, the extraction of political events from news texts (Bouscarrat et al. 2021).

## 5.3 To infinity and beyond!

To conclude this adventure in the wonderland of MWEs, I discuss my projects, interests, and dreams for my future research. First, I list concrete ideas for the next steps in the PARSEME initiative (§5.3.1). Then, I describe ongoing work on the front of semantic lexicon induction (§5.3.2). Finally, I suggest that the field should take linguistic diversity more seriously in its models and data, and share exploratory ideas on how this could be made possible (§5.3.3)

### 5.3.1 PARSEME 2030: keeping the ball rolling

In the last years, PARSEME allowed me to keep getting my hands dirty with highly enjoyable data annotation, corpus processing, code writing, experiments, etc. Moreover, I learned much about project management and community building. Both aspects (concrete data and human interaction) bring me satisfaction in my work, motivating me to continue. In addition, many interesting research questions emerged from this apparently more technical work, and there is no reason to believe that this should stop.

However, *fr on ne peut pas courir plusieurs lièvres à la fois* (lit. ‘one not can not run several hares at the same time’) ‘if you run after two hares, you will catch neither’, so I list below some of my priorities for PARSEME, from short-term

<footnote n="11">¹¹The thesis has not been defended, it has been interrupted for personal reasons.</footnote>
to long-term goals, and from more concrete applied tasks to more exploratory fundamental research.

**Resource creation** In spite of the lack of prestige associated with this task, creating resources is still important in the era of large language models. First, although NLP models nowadays tend to build upon self-supervision from raw text, human supervision is often much more effective than increasing model or raw data size (Ouyang et al. 2022). Second, model evaluation is essential to understand and improve language technology, and can only be performed with the help of annotated datasets. How can we know if our NLP models are able to understand and generate MWEs? Without annotated corpora and datasets, this would be impossible, e.g. Madabushi et al. (2021); Haviv et al. (2023). Finally, creating resources informs linguistic theory and enriches the description of linguistic phenomena by grounding them on actual data (as opposed to toy examples). Concretely, the next steps for PARSEME are the following:

1. We need to organise the development and release of MWE-annotated corpora in multiple languages in a more homogeneous, systematic, and automatised way. Taking inspiration from the UD community, we would like to use continuous integration tools to automatically validate the corpora, and package them for regular releases. Building a **software infrastructure** that allow scaling up the number of languages covered is essential to make the release of annotated corpora less dependent on human intervention.

2. **Community management** is key to ensure the longevity of PARSEME. First, we need more effective training materials for onboarding, such as recorded video tutorials, quick start manuals, readable diagrams, etc. Second, keeping the current members engaged requires organising recurrent meetings, not only to make collective decisions but also to set goals and deadlines. Third, documentation (e.g. websites, git, readmes) must be up-to-date and easily accessible.

3. Massively multilingual initiatives such as PARSEME and UD should ensure that resource development is compatible across annotation layers. Moreover, projects can benefit from each other's experience and tools. Proposals for **synergies between PARSEME and UD** have been proposed (Savary et al. 2023), and concrete discussions take place within the UniDive Action and events such as the UNLID Dagstuhl Seminar.<fnref n="12" />

<footnote n="12">¹²https://gitlab.com/unlid-dagstuhl-seminar/unlid-2023</footnote>
All in all, the expected outcome of these steps is to increase both the quality and the quantity of annotated corpora, as well as attracting new languages to join the initiative. On the long run, these resources will not only allow the development of more accurate and robust automatic MWE identification systems, but could also be used to fine-tune and evaluate large language models, and inform linguistic typology studies and cross-lingual descriptions of this complex phenomenon. Many of the tasks above are already planned or in progress, in collaboration with Agata Savary and many other PARSEME members.

**Enhanced and extended MWE description** The PARSEME multilingual guidelines cover only verbal MWEs, but other MWE categories have been described in language specific projects, e.g. for French (§4.2.1.2). Thus, extending the current guidelines to nominal, modifier, and functional MWE categories is now both necessary and possible. A draft of the overall categorisation of MWEs across morphosyntactic categories has been proposed in §2.1.4, and could serve as a basis for collaborative guidelines writing, annotation, and incremental development of full-coverage MWE-annotated corpora in many languages. This process could help clarify the scope of MWEs with respect to related linguistic phenomena such as named entities, domain-specific terms, metaphors, collocations, verbal and non-verbal valency, semi-productive syntactic irregularity, and constructions (in the sense of construction grammar).

**In-context MWE semantics** Corpora annotated for MWEs encode binary distinctions between idiomatic and compositional readings of word combinations. However, more realistic semantic representations also involve linking textual units to meaning representations, such as Wordnet synsets, lexicon entries, and semantic frames. Token-based MWE annotations such as those in the PARSEME corpora are a first step towards preventing word-by-word interpretation, but remain quite basic and of little practical usefulness if not used to foster meaning representations that take their idiosyncrasies into account.

A first step would consist in modeling compositionality scores in context, as opposed to type-level predictions addressed in Chapter 3. Such scores could be useful for automatic MWE identification and vice versa, thus connecting the work presented in Chapter 3 and Chapter 4. In-context compositionality prediction could rely on contextual word embeddings extracted from language models. Related work indicates that, although language models can predict the presence of idiomatic combinations, they seem to be unable to encode their meaning, e.g. with respect to paraphrases (Shwartz 2019; Madabushi et al. 2022).
Another possibility consists in assigning lexical functions to annotated MWEs, in the sense of the meaning-text theory (Mel'čuk 2023). Automatic discovery and prediction of lexical functions has shown promising results (Anke et al. 2016; Rodríguez-Fernández et al. 2016; Garcia, Salido, Sotelo, et al. 2019; Anke et al. 2021; 2022). Combining this kind of approach to the PARSEME view could enhance the current flat representation with richer lexical functions that, in turn, guide the mapping between MWEs and higher-level representations (e.g. senses, frames, AMR graphs). This mapping between MWEs and meaning representations is an ambitious research goal that can be addressed in a more general context via word sense and frame induction (see §5.3.2 below).

**Cognitive processing of multiword units** MWEs are complex linguistic objects whose study can take inspiration from research in other fields. In psycholinguistics, multiword units and idiomaticity are studied mostly in the perspective of usage-based approaches to language acquisition and statistical learning (Goldberg 2005; Tomasello 2015). For instance, Conklin & Carrol (2020) use an eye tracking protocol to study how we process conventional (e.g. en bread and butter) and new binomials. Siyanova-Chanturia et al. (2011) find a processing advantage for idiomatic over non idiomatic combinations for native speakers, which was not observed for non native speakers. Computational models have been proposed to explain the formation of multiword chunks in the mental lexicon, both at perception and production (McCauley & Christiansen 2019).

I believe that cross-fertilisation of ideas between computational linguistics and cognitive psycholinguistics can be mutually beneficial. On the one hand, findings about the basic associative and memory mechanisms that influence language acquisition can inspire NLP models, e.g. as inductive biases in neural architectures. On the other hand, computational simulation can be used to study language acquisition, e.g. in child-directed speech corpora. Psychological models are usually very simple and could benefit from more complex representations such as word embeddings to model semantic proximity, for example. The multidisciplinary study of the acquisition of multiword sequences is the research topic of Leonardo Pinto-Arata, whose PhD I co-supervise with Arnaud Rey.

### 5.3.2 Without lexicons, NLP cannot fly

In Savary, Cordeiro & Ramisch (2019), we conjecture that lexicons are necessary for accurate and robust MWE identification. One of the arguments to support this claim is the nature of MWE idiosyncrasies: limited morphosyntactic flexibility, used as a proxy for semantic non compositionality, can only be observed across
several occurrences. Thus, most MWEs can be more accurately be modeled at
the level of *types*, that is, as sets of token occurrences sharing the same idiomatic
meaning and associated fixedness properties. Since MWEs’ theoretical ambigu-
ity seems quite low in practice (Savary, Cordeiro, Lichte, et al. 2019), the use
of supervised corpus-based methods may be suboptimal for their identification.
Conversely, lexicons can describe the properties of MWE types concisely and
be used instead of or in complement to supervised MWE identification methods
(Schneider et al. 2014; Riedl & Biemann 2016; Scholivet et al. 2018).

Moreover, as discussed in §5.3.1, the link between MWE identification and
more general semantic parsing is not yet fully laid out. In semantic parsing, mean-
ing representations usually adopt some annotation scheme like UCCA (Abend &
Rappoport 2013), AMR (Banarescu et al. 2013), or DRS (Bos et al. 2017). These
schemes rely on lexical resources that describe word senses, such as Wordnets
(Bond & Foster 2013), and predicative structure, such as PropBank (Pradhan et al.
2022) and FrameNet (Baker et al. 1998).

Since most MWEs are semantically idiosyncratic, they have often been studied
in the context of semantic annotation projects like PropBank (Hwang et al. 2010;
Bonial & Palmer 2016), FrameNet (Petruck & Ellsworth 2016), Wordnet (Mitlelu,
Stoyanova, et al. 2019; Maziarz et al. 2022) and UMR (Sun et al. 2023). Much em-
phasis has been put on light-verb constructions, given the complex interaction
between the semantic arguments of predicative nouns and the syntactic com-
plements of light verbs. Nonetheless, more comprehensive MWE models like
PARSEME are still largely disconnected from meaning representations.

**Inducing semantic lexicons** SELEXINI is a collaborative project which I coor-
dinate, and whose ambitious goal it to connect the partners’ expertise on MWE
modeling with the world of meaning representations.<fnref n="13" /> To this end, we assume
that *semantic lexicons* must be developed, not only for MWE identification, but
also for other NLP tasks involving some degree of semantic interpretation. In
particular, we believe that lexicons can take MWEs into account from scratch, as
well as confer robustness and interpretability to NLP systems, complementary
to pre-trained language models that are nowadays mainstream.

Handcrafted semantic lexicons abound for English (WordNet, FrameNet, Verb-
Net, PropBank) and exist for some other languages. Babelnet (Navigli & Ponzetto
2010) is a highly multilingual lexical network, with version 5.0 covering 500
languages. Wiktionary is a collaborative multilingual lexicon built by and for
humans, covering 182 languages with a relatively high coverage and quality.

<footnote n="13">¹³https://selexini.lis-lab.fr</footnote>
For French, numerous projects aimed at (semi-)automatically creating similar resources, such as the WOLF (Sagot & Fišer 2008), and a French version of VerbNet (Danlos et al. 2016). Manually annotated FrameNet corpora for French include ASFALDA (Djemaa et al. 2016), and CALOR (Marzinotto et al. 2018), covering 105 and 50 frames (i.e. about 1/10 of the English FrameNet). Finally, there is the Réseau Lexical du Français (RL-fr), a handcrafted lexicon based on meaning-text theory, notably modelling MWEs (Polguère 2014).

As they allow for fine-grained encoding of phenomena (e.g. RL-fr, FrameNet),
handcrafted lexicons favour linguistic precision. Still, their granularity is fixed
and often too fine for practical NLP (Lacerra et al. 2020). Moreover, huge effort
is required to reach decent coverage: FrameNet has been ongoing for 20+ years,
with still insufficient coverage for English, and even more so for French. The RL-
fr described about 28K lexical units. MWE coverage is even weaker, e.g. 55.2%
of the MWEs annotated in the PARSEME-FR corpus are absent from Wiktionary.
Finally, most handcrafted lexicons build upon discrete labels, whereas recent NLP
breakthroughs lie in using continuous representations.

<figure><img src="image_32.png" /><figcaption>Figure 5.1: SELEXINI’s framework illustrating how lemmas and in-
duced frames relate to occurrences. Source: ANR SELEXINI proposal.</figcaption></figure>

Rather than manually building a semantic lexicon, SELEXINI aims at devel-
oping methods for MWE-aware and semi-supervised *sense and frame induction*.
Induction is understood as automatic lexicon construction by learning from dis-
tributional and structural regularities in raw large corpora. The induced lexicon
consists of clustered sentences associated to (predicted) explicit labels, as shown
in Figure 5.1. Although clusters are induced from opaque embeddings and pre-
trained neural language models, the lexicon’s structure will make them more
interpretable than (contextualised) embeddings alone. The lexicon covers single
and MWE entries, encoding their syntactic and semantic idiosyncrasies.

However, it would be a pity to induce lexicons from scratch and completely
ignore existing handcrafted resources. Based on our experience on weak super-
vision to induce semantic representations (Aloui et al. 2020), we will leverage
Wiktionary, using it as weak supervision to inform and constrain the clustering
process. We prefer Wiktionary over other resources because it is open, it has
a larger coverage as compared to other handcrafted resources like the French
Open Wordnet (Bond & Foster 2013), and because it is more suitable than Babel-
net for French WSD according to Segonne et al. (2019). Moreover, Wiktionary is
large (24 languages have 50,000 entries or more), so our methods are applicable
to other languages. This topic is currently being studied within the PhD of Anna
Mosolova, which I co-advise with Marie Candito.

**Evaluating semantic lexicons** In SELEXINI, we believe that the notion of *semantic lexicon* is central to attain interpretable and robust semantic processing of texts. In a semantic lexicon, linguistic objects such as induced senses and frames act as trade-off aggregate between static word embeddings, which tend to conflate the different meanings of words (Mikolov et al. 2013; Bojanowski et al. 2017; Camacho-Collados & Pilehvar 2018), and the opposite extreme of contextual embeddings obtained via pre-trained language models, in which each occurrence has a distinct representation (Devlin et al. 2019). We hypothesise that the semantic lexicon will be usable both within robust NLP models for downstream tasks and in tasks requiring human interpretation.

We will design evaluation protocols to drive our approach towards high robustness, as compared to supervised methods alone. In particular, we will evaluate the lexicon's usefulness extrinsically on the downstream task of machine reading comprehension. Our goals are (1) to devise new strategies to inject (induced) lexical-semantic knowledge into machine reading comprehension systems, and (2) to assess whether the lexicon helps improve their generalization and explainability. For example, for the question *q=who robbed the diamonds?* and the passage *p=the theft of the diamonds was committed by the queen*, a system having access to the induced frame [*steal, rob, commit theft*] could “explain” why the deep subject of *p* is the correct answer.

In short, we intend to design a sound lexicon model, inspired by the sophisti-
cation reached in handcrafted lexicons, taking into account the particularities of
MWEs. In contrast to standard word embeddings, we will induce structured se-
mantic units including syntactic and semantic valency, that is, semantic frames
and their slots. As a by-product, this procedure will generate an automatically
sense-annotated corpus, which can bootstrap large-coverage WSD. One weak-
ness of word sense induction, also touching contextualized word embeddings,
is their low interpretability. SELEXINI includes the generation of interpretable
textual descriptions for the induced units. The resulting hybrid lexicon will link
dense embeddings to symbolic descriptions, thus proposing a trade-off between
practical usefulness and explicit labels. Its evaluation will be based on applicabil-
ity, putting special emphasis on its integration within downstream applications,
the interpretability of results, and the diversity of the phenomena covered.

5.3.3 Diversity in NLP: the more the merrier

To conclude this roadmap, I would like to *start a fresh hare* ‘start a new topic for discussion’ and mention the issue of *linguistic diversity*. While the recent progress in the field has been impressive, we have been witnessing an overwhelming dominance of English (Bender 2011) as the main language of study and, to a lesser extent, of a few other languages spoken in occidental societies (Joshi et al. 2020). As a result, many models and methods are suboptimal or not at all adapted to the remaining, mostly morphologically rich, languages, even those with large numbers of speakers.

Linguistic diversity can also be looked at from an intra-linguistic perspective.
On the one hand, most linguistic phenomena are Zipfian, with a large number of
rare events (Baayen 2001). On the other hand, statistical machine learning algo-
rithms, especially discriminative ones like neural networks, are often optimised
to minimise some loss function averaged over many training instances. Hence,
rare linguistics phenomena are not correctly modeled, and benchmark-oriented
evaluation will not properly assess their impact on results. This sensitivity of sta-
tistical NLP models to data sparseness creates a bias: aspects which are not well
covered by technology might be gradually abandoned by speakers.

As stated in the UniDive memorandum of understanding: *Endangered diversity is known to be a major risk in domains of life studied by biology, genetics, medicine (Forschungsverbund Berlin 2018), sociology (Phillips 2014), etc. Linguistic diversity is closely connected to these aspects and should be regarded, from a holistic perspective, as part of biocultural diversity, as put by the Terralingua initiative.*<fnref n="14" /> In §2.3.4, I argue that the idiosyncratic behaviour of MWEs makes them an intrinsically interesting phenomenon for intra-linguistic diversity. Studying them in a multilingual context such as PARSEME favours a more diverse point

<footnote n="14">¹⁴https://unidive.lisn.upsaclay.fr/</footnote>
of view in their linguistic description and derived computational models. Within
the UniDive COST Action, I intend to foster research that favours both intra- and
inter-linguistic diversity, that is:

* Pursue the discussion and convergence between PARSEME and related multilingual initiatives such as Universal Dependencies, Unimorph, and Uniform Meaning Representation (Savary et al. 2023).

* Clarify the relation between MWEs and construction grammar, which in turn can provide a powerful tool to look at different cross-linguistic strategies to convey meaning (Croft 2022).

* Ground language technology on typology research and resources, and contributing to their development (Scholivet et al. 2019; Ponti et al. 2019).

Assessing and favouring intra- and inter-language diversity is a much more
generic and vague research project than the two previous ones, discussed in more
detail in §5.3.1 and in §5.3.2. It is a pervasive and traversal goal, a *fr* *fil directeur*
(lit. ‘thread director’) ‘guiding principle’, a political position statement about our
mission as NLP researchers in a time when language technology becomes incredi-
bly influential in human societies. In addition to developing language technology
in the form of parsers, language models, MT systems, this includes collaborating
with marginalised linguistic communities to identify their needs in a more hori-
zontal fashion (Bird 2020).

Thus, I conclude this manuscript with an invitation to reflect on our role as
academic researchers. We benefit from the privilege of being less subject to the
pressure of the markets, so our goal cannot be only to beat state-of-the-art perfor-
mances on highly standardised benchmarks. The current hype on neural meth-
ods and large language models is exciting, and opens many new research oppor-
tunities. However, these innovations are driven by only a few companies from
the Silicon Valley, and the speed at which they are adopted could lead to massive
homogenisation and loss of diversity in terms of languages, approaches, linguis-
tic phenomena covered, etc.

I intend to pursue my work on multiword expressions and semantic lexicons
in a highly multilingual environment, with further interactions with universal
annotation projects, linguistic typology, and construction grammar. Moreover, I
am very much interested in exploring new ways to communicate about compu-
tational linguistics research, both to fellow researchers and to the general public,
relying on scientific mediation, artistic expression and partnerships. In this way,
I intend to contribute to a more diverse landscape in NLP, because, as put by
Lewis Carroll, “imagination is the only weapon in the war with reality”.
References

Abeillé, Anne, Lionel Clément & Alexandra Kinyon. 2000. Building a treebank for French. In *Proceedings of the second international conference on language resources and evaluation (LREC’00)*. Athens, Greece: European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2000/pdf/230.pdf.

Abend, Omri & Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In *Proceedings of the 51st annual meeting of the association for computational linguistics (volume 1: long papers)*, 228–238. Sofia, Bulgaria: ACL. https://aclanthology.org/P13-1023.

Acosta, Otavio, Aline Villavicencio & Viviane Moreira. 2011. Identification and treatment of multiword expressions applied to information retrieval. In *Proceedings of the workshop on multiword expressions: from parsing and generation to the real world*, 101–109. Portland, Oregon, USA: ACL. https://aclanthology.org/W11-0815.

Adams, Douglas. 1979. *The hitchhiker's guide to the galaxy*. Pan Books.

Alegria, Iñaki, Olatz Ansa, Xabier Artola, Nerea Ezeiza, Koldo Gojenola & Ruben Urizar. 2004. Representation and treatment of multiword expressions in Basque. In *Proceedings of the workshop on multiword expressions: integrating processing*, 48–55. Barcelona, Spain: ACL. https://aclanthology.org/W04-0407.

Aloui, Cindy, Carlos Ramisch, Alexis Nasr & Lucie Barque. 2020. SLICE: supersense-based lightweight interpretable contextual embeddings. In *Proceedings of the 28th international conference on computational linguistics*, 3357–3370. Barcelona, Spain (Online): International Committee on Computational Linguistics. DOI: 10.18653/v1/2020.coling-main.298. https://aclanthology.org/2020.coling-main.298.

Anke, Luis Espinosa, Jose Camacho-Collados, Sara Rodríguez-Fernández, Horacio Saggion & Leo Wanner. 2016. Extending wordnet with fine-grained collocaational information via supervised distributional learning. In *Proceedings of coling 2016, the 26th international conference on computational linguistics: technical papers*, 3422–3432. Osaka, Japan: The COLING 2016 Organizing Committee. http://www.aclweb.org/anthology/C16-1323.
Anke, Luis Espinosa, Joan Codina-Filba & Leo Wanner. 2021. Evaluating language models for the retrieval and categorization of lexical collocations. In *Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume*, 1406–1417. Online: ACL. DOI: 10.18653/v1/2021.eacl-main.120. https://aclanthology.org/2021.eacl-main.120.

Anke, Luis Espinosa, Alexander Shvets, Alireza Mohammadshahi, James Henderson & Leo Wanner. 2022. Multilingual extraction and categorization of lexical collocations with graph-aware transformers. In *Proceedings of the 11th joint conference on lexical and computational semantics*, 89–100. Seattle, Washington: ACL. DOI: 10.18653/v1/2022.starsem-1.8. https://aclanthology.org/2022.starsem-1.8.

Araujo, Vitor De, Carlos Ramisch & Aline Villavicencio. 2011. Fast and Flexible MWE Candidate Generation with the mwetoolkit. In *Proceedings of the Workshop on MWEs: from Parsing and Generation to the Real World*, 134–136. http://aclweb.org/anthology/W11-0822. Portland, OR, USA: ACL.

Artstein, Ron & Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. *Computational Linguistics* 34(4). 555–596.

Arun, Abhishek & Frank Keller. 2005. Lexicalization in crosslinguistic probabilistic parsing: the case of French. In *Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL’05)*, 306–313. Ann Arbor, Michigan: ACL. DOI: 10.3115/1219840.1219878. https://aclanthology.org/P05-1038.

Attia, Mohammed, Antonio Toral, Lamia Tounsi, Pavel Pecina & Josef van Genabith. 2010. Automatic extraction of Arabic multiword expressions. In Éric Laporte, Preslav Nakov, Carlos Ramisch & Aline Villavicencio (eds.), *Proceedings of the coling workshop on multiword expressions: from theory to applications (mwe 2010)*, 18–26. Beijing, China: Association for Computational Linguistics.

Baayen, R. Harald. 2001. *Word frequency distributions*. Vol. 18 (Text, Speech and Language Technology). Springer.

Baker, Collin F., Charles J. Fillmore & John B. Lowe. 1998. The Berkeley FrameNet project. In *36th annual meeting of the association for computational linguistics and 17th international conference on computational linguistics, volume 1*, 86–90. Montreal, Quebec, Canada: ACL. DOI: 10.3115/980845.980860. https://aclanthology.org/P98-1013.

Baldwin, Timothy, Colin Bannard, Takaaki Tanaka & Dominic Widdows. 2003. An empirical model of multiword expression decomposability. In *Proceedings of the ACL 2003 workshop on multiword expressions: analysis, acquisition and*
*treatment*, 89–96. Sapporo, Japan: ACL. DOI: 10.3115/1119282.1119294.
https://aclanthology.org/W03-1812.

Baldwin, Timothy & Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya & Fred J. Damerau (eds.), *Handbook of natural language processing*, 2nd edn., 267–292. Boca Raton, FL, USA: CRC Press, Taylor & Francis Group.

Banarescu, Laura, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer & Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In *Proceedings of the 7th linguistic annotation workshop and interoperability with discourse*, 178–186. Sofia, Bulgaria: ACL. https://aclanthology.org/W13-2322.

Bannard, Colin. 2007. A measure of syntactic flexibility for automatically identifying multiword expressions in corpora. In *Proceedings of the workshop on a broader perspective on multiword expressions*, 1–8. Prague, Czech Republic: ACL. https://aclanthology.org/W07-1101.

Baroni, Marco, Silvia Bernardini, Adriano Ferraresi & Eros Zanchetta. 2009. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. *Language Resources & Evaluation* 43(3). 209–226. DOI: 10.1007/s10579-009-9081-4. http://www.springerlink.com/content/C348PU7321GX5081.

Baroni, Marco, Georgiana Dinu & Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In *Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: long papers)*, 238–247. Baltimore, Maryland: ACL. DOI: 10.3115/v1/P14-1023. https://aclanthology.org/P14-1023.

Barque, Lucie, Pauline Haas, Richard Huyghe, Delphine Tribout, Marie Candito, Benoit Crabbé & Vincent Segonne. 2020. FrSemCor: annotating a French corpus with supersenses. English. In *Proceedings of the 12th language resources and evaluation conference*, 5912–5918. Marseille, France: European Language Resources Association. https://aclanthology.org/2020.lrec-1.724.

Barreiro, Anabela, Johanna Monti, Brigitte Orliac & Fernando Batista. 2013. When Multiwords Go Bad in Machine Translation. *MT Summit workshop Proceedings on Multi-word Units in Machine Translation and Translation Technology*. 10.

Bejček, Eduard, Pavel Straňák & Pavel Pecina. 2013. Syntactic identification of occurrences of multiword expressions in text using a lexicon with dependency structures. In *Proceedings of the 9th workshop on multiword expressions*, 106–115. Atlanta, Georgia, USA: ACL. https://aclanthology.org/W13-1016.
Bender, Emily M. 2011. On achieving and evaluating language-independence in NLP. *Linguistic Issues in Language Technology* 6. DOI: 10.33011/lilt.v6i.1239. https://journals.colorado.edu/index.php/lilt/article/view/1239.

Bergsma, Shane, Aditya Bhargava, Hua He & Grzegorz Kondrak. 2010. Predicting the semantic compositionality of prefix verbs. In *Proceedings of the 2010 conference on empirical methods in natural language processing*, 293–303. Cambridge, MA: ACL. https://aclanthology.org/D10-1029.

Biemann, Chris & Eugenie Giesbrecht. 2011. Distributional semantics and compositionality 2011: shared task description and results. In *Proceedings of the workshop on distributional semantics and compositionality*, 21–28. Portland, Oregon, USA: ACL. https://aclanthology.org/W11-1304.

Bird, Steven. 2020. Decolonising speech and language technology. In *Proceedings of the 28th international conference on computational linguistics*, 3504–3519. Barcelona, Spain (Online): International Committee on Computational Linguistics. DOI: 10.18653/v1/2020.coling-main.313. https://aclanthology.org/2020.coling-main.313.

Bird, Steven, Ewan Klein & Edward Loper. 2009. *Natural language processing with python: analyzing text with the natural language toolkit*. O’Reilly. http://www.nltk.org/book.

Birke, Julia & Anoop Sarkar. 2006. A clustering approach for nearly unsupervised recognition of nonliteral language. In *11th conference of the European chapter of the association for computational linguistics*, 329–336. Trento, Italy: ACL. https://aclanthology.org/E06-1042.

Blaheta, Don & Mark Johnson. 2001. Unsupervised learning of multi-word verbs. In *Proceedings of the acl workshop on collocations*, 54–60. Toulouse, France.

Blunsom, Phil & Timothy Baldwin. 2006. Multilingual deep lexical acquisition for HPSGs via supertagging. In *Proceedings of the 2006 conference on empirical methods in natural language processing*, 164–171. Sydney, Australia: ACL. https://aclanthology.org/W06-1620.

Bogantes, Diana, Eric Rodríguez, Alejandro Arauco, Alejandro Rodríguez & Agata Savary. 2016. Towards lexical encoding of multi-word expressions in Spanish dialects. In *Proceedings of the tenth international conference on language resources and evaluation (LREC’16)*, 2255–2261. Portorož, Slovenia: European Language Resources Association (ELRA). https://aclanthology.org/L16-1358.

Bojanowski, Piotr, Edouard Grave, Armand Joulin & Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. *Transactions of the Associa-
tion for Computational Linguistics 5. 135–146. DOI: 10.1162/tacl_a_00051.
https://doi.org/10.1162/tacl%5C_a%5C_00051.

Bond, Francis & Ryan Foster. 2013. Linking and extending an open multilingual Wordnet. In *Proceedings of the 51st annual meeting of the association for computational linguistics (volume 1: long papers)*, 1352–1362. Sofia, Bulgaria: ACL. https://www.aclweb.org/anthology/P13-1133.

Bonial, Claire & Martha Palmer. 2016. Comprehensive and consistent PropBank light verb annotation. In *Proceedings of the tenth international conference on language resources and evaluation (LREC’16)*, 3980–3985. Portorož, Slovenia: European Language Resources Association (ELRA). https://aclanthology.org/L16-1628.

Bos, Johan, Valerio Basile, Kilian Evang, Noortje J. Venhuizen & Johannes Bjerva. 2017. The Groningen Meaning Bank. In *Nancy Ide & James Pustejovsky (eds.), Handbook of Linguistic Annotation*, 463–496. Dordrecht: Springer Netherlands. DOI: 10.1007/978-94-024-0881-2_18. https://doi.org/10.1007/978-94-024-0881-2_18.

Bott, Stefan, Nana Khvtisavrishvili, Max Kisselew & Sabine Schulte im Walde. 2016. GhoSt-PV: A representative gold standard of German particle verbs. In *Proceedings of the 5th workshop on cognitive aspects of the lexicon (CogALex - v)*, 125–133. Osaka, Japan: The COLING 2016 Organizing Committee. https://aclanthology.org/W16-5318.

Bouamor, Dhouha, Nasredine Semmar & Pierre Zweigenbaum. 2012. Identifying bilingual Multi-Word Expressions for Statistical Machine Translation. In *Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)*. Istanbul, Turkey: European Language Resources Association (ELRA).

Bouscarrat, Léo, Antoine Bonnefoy, Cécile Capponi & Carlos Ramisch. 2020. Multilingual enrichment of disease biomedical ontologies. English. In *Proceedings of the lrec 2020 workshop on multilingual biomedical text processing (multilingualbio 2020)*, 21–28. Marseille, France: European Language Resources Association. https://aclanthology.org/2020.multilingualbio-1.4.

Bouscarrat, Léo, Antoine Bonnefoy, Cécile Capponi & Carlos Ramisch. 2021. AMU-EURANOVA at CASE 2021 task 1: assessing the stability of multilingual BERT. In *Proceedings of the 4th workshop on challenges and applications of automated extraction of socio-political events from text (case 2021)*, 161–170. Online: ACL. DOI: 10.18653/v1/2021.case-1.21. https://aclanthology.org/2021.case-1.21.
References

Breidt, Elisabeth, Frederique Segond & Giuseppe Valetto. 1996. Formal description of multi-word lexemes with the finite-state formalism IDAREX. In *COLING 1996 volume 2: the 16th international conference on computational linguistics*. https://aclanthology.org/C96-2182.

Buarque, Chico. 2003. *Budapeste*. 176 p. Companhia das Letras.

Butnariu, Cristina, Su Nam Kim, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz & Tony Veale. 2010. SemEval-2 task 9: the interpretation of noun compounds using paraphrasing verbs and prepositions. In *Proceedings of the 5th international workshop on semantic evaluation*, 39–44. Uppsala, Sweden: ACL. https://aclanthology.org/S10-1007.

Cafferkey, Conor, Deirdre Hogan & Josef van Genabith. 2007. Multi-word units in treebank-based probabilistic parsing and generation. In *Proc. of RANLP 2007*. Borovets.

Calzolari, Nicoleta, Charles Fillmore, Ralph Grishman, Nancy Ide, Alessandro Lenci, Catherine Macleod & Antonio Zampolli. 2002. Towards best practice for multiword expressions in computational lexicons. In *Proceedings of the third international conference on language resources and evaluation (Irec 2002)*, 1934–1940. Las Palmas, Canary Islands, Spain: European Language Resources Association.

Camacho-Collados, Jose & Mohammad Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. *Journal of Artificial Intelligence Research* 63(1). 743–788. DOI: 10.1613/jair.1.11259. https://doi.org/10.1613/jair.1.11259.

Candito, Marie, Mathieu Constant, Carlos Ramisch, Agata Savary, Bruno Guil-laume, Yannick Parmentier & Silvio Cordeiro. 2021. A french corpus annotated for multiword expressions and named entities. *Journal of Language Modelling* 8(2). 415–479. DOI: 10.15398/jlm.v8i2.265. https://jlm.ipipan.waw.pl/index.php/JLM/article/view/265.

Candito, Marie, Mathieu Constant, Carlos Ramisch, Agata Savary, Yannick Parmentier, Caroline Pasquer & Jean-Yves Antoine. 2017. Annotation d'expressions polylexicales verbales en français. In *Actes de la 24e conférence sur le traitement automatique des langues naturelles (taln 2017) : articles courts*, 1–9. http://taln2017.cnrs.fr/wp-content/uploads/2017/06/actes_TALN_2017-vol2.pdf. Orléans, France: ATALA.

Candito, Marie & Matthieu Constant. 2014. Strategies for contiguous multiword expression analysis and dependency parsing. In *Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: long papers)*, 743–753. Baltimore, Maryland: ACL. DOI: 10.3115/v1/P14-1070. https://aclanthology.org/P14-1070.