## 5 Conclusions

In our chapter, we presented our system for verbal MWE recognition. The system uses POS tagging and dependency parsing as a means of finding verbal MWEs in multiple languages.

Apart from parsing-based solutions (Al Saied et al. (2017), Nerima et al. (2017) and our system), the shared task hosted a number of other approaches, like neural networks (Klyueva et al. (2017)) or sequence labeling based models (Boroş et al. (2017), Maldonado et al. (2017)). In the final results, parsing-based systems achieved the best results for almost all languages, showing that this approach works very well for language independent MWE identification.

Our chapter further shows that it is possible to build a highly language independent MWE detection methodology that makes use of a limited amount of language-specific data and achieve reasonable results.

## Acknowledgements

We would like to thank the anonymous reviewers and the volume editors for their invaluable comments on our work.

Veronika Vincze was supported by the UNKP-17-4 New National Excellence Program of the Ministry of Human Capacities, Hungary.

## Abbreviations

ID idiom

IREFLV inherently reflexive verb

LVC light-verb construction

MWE multiword expression

OTH other

POS part of speech

VPC verb-particle construction

VMWE verbal multiword expression

## References

Al Saied, Hazem, Matthieu Constant & Marie Candito. 2017. The ATILF-LLF system for parseme shared task: A transition-based verbal multiword expression tagger. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 127–132. Association for Computational Linguistics. DOI:10.18653/v1/W17-1717
Bohnet, Bernd. 2010. Top accuracy and fast dependency parsing is not a contradiction. In *Proceedings of the 23rd International Conference on Computational Linguistics* (COLING 2010), 89–97. http://www.aclweb.org/anthology/C10-1011.

Boroş, Tiberiu, Sonia Pipa, Verginica Barbu, Mititelu & Dan Tufiş. 2017. A data-driven approach to verbal multiword expression detection. PARSEME shared task system description paper. In *Proceedings of the 13th Workshop on Multi-word Expressions* (MWE ’17), 121–126. Association for Computational Linguistics. DOI:10.18653/v1/W17-1716

Candito, Marie & Matthieu Constant. 2014. Strategies for contiguous multiword expression analysis and dependency parsing. In *Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers)*, 743–753. Association for Computational Linguistics. http://www.aclweb.org/anthology/P14-1070.

Constant, Matthieu & Joakim Nivre. 2016. A transition-based system for joint lexical and syntactic analysis. In *Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers)*, 161–171. Association for Computational Linguistics. http://www.aclweb.org/anthology/P16-1016.

Green, Spence, Marie-Catherine de Marneffe, John Bauer & Christopher D. Manning. 2011. Multiword expression identification with tree substitution grammars: A parsing tour de force with French. In *Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing*, 725–735. Association for Computational Linguistics. http://www.aclweb.org/anthology/D11-1067.

Green, Spence, Marie-Catherine de Marneffe & Christopher D. Manning. 2013. Parsing models for identifying multiword expressions. *Computational Linguistics* 39(1). 195–227. DOI:10.1162/COLI_a_00139

Klyueva, Natalia, Antoine Doucet & Milan Straka. 2017. Neural networks for multi-word expression detection. In *Proceedings of the 13th Workshop on Multi-word Expressions* (MWE ’17), 60–65. Association for Computational Linguistics. April 4, 2017. DOI:10.18653/v1/W17-1707

Maldonado, Alfredo, Lifeng Han, Erwan Moreau, Ashjan Alsulaimani, Koel Dutta Chowdhury, Carl Vogel & Qun Liu. 2017. Detection of verbal multi-word expressions via conditional random fields with syntactic dependency features and semantic re-ranking. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 114–120. Association for Computational Linguistics. DOI:10.18653/v1/W17-1715
Nasr, Alexis, Carlos Ramisch, José Deulofeu & André Valli. 2015. Joint dependency parsing and multiword expression tokenization. In *Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, 1116–1126. Association for Computational Linguistics. http://www.aclweb.org/anthology/P15-1108.

Nerima, Luka, Vasiliki Foufi & Eric Wehrli. 2017. Parsing and MWE detection: Fips at the PARSEME shared task. In *Proceedings of The 13th Workshop on Multi-word Expressions* (MWE ’17), 54–59. Association for Computational Linguistics. DOI:10.18653/v1/W17-1706

Nivre, Joakim. 2015. Towards a universal grammar for natural language processing. In Alexander Gelbukh (ed.), *Computational Linguistics and intelligent text processing*, 3–16. Cham: Springer. http://stp.lingfil.uu.se/~nivre/docs/nivre15cicling.pdf.

Nivre, Joakim, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty & Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the Tenth International Conference on Language Resources and Evaluation* (LREC 2016), 1659–1666. European Language Resources Association (ELRA). 23-28 May, 2016.

Savary, Agata, Marie Candito, Verginica Barbu Mititelu, Eduard Bejček, Fabienne Cap, Slavomír Čeplö, Silvio Ricardo Cordeiro, Gülşen Eryiğit, Voula Giouli, Maarten van Gompel, Yaakov HaCohen-Kerner, Jolanta Kovalevskaitė, Simon Krek, Chaya Liebeskind, Johanna Monti, Carla Parra Escartín, Lonneke van der Plas, Behrang QasemiZadeh, Carlos Ramisch, Federico Sangati, Ivelina Stoyanova & Veronika Vincze. 2018. PARSEME multilingual corpus of verbal multiword expressions. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 87–147. Berlin: Language Science Press. DOI:10.5281/zenodo.1471591

Savary, Agata, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova & Antoine Doucet. 2017. The PARSEME Shared Task on automatic identification of verbal multiword expressions. In *Proceedings of the 13th
*Workshop on Multiword Expressions* (MWE '17), 31–47. Association for Computational Linguistics. DOI:10.18653/v1/W17-1704

Simkó, Katalin Ilona, Viktória Kovács & Veronika Vincze. 2017. USzeged: Identifying verbal multiword expressions with POS tagging and parsing techniques. In *Proceedings of The 13th Workshop on Multiword Expressions* (MWE '17), 48–53. Association for Computational Linguistics.

Vincze, Veronika, János Zsibrita & István Nagy T. 2013. Dependency parsing for identifying Hungarian light verb constructions. In *Proceedings of the 6th International Joint Conference on Natural Language Processing*, 207–215. Nagoya, Japan: Asian Federation of Natural Language Processing. http://www.aclweb.org/anthology/I13-1024.

Waszczuk, Jakub, Agata Savary & Yannick Parmentier. 2016. Promoting multi-word expressions in a* TAG parsing. In Nicoletta Calzolari, Yuji Matsumoto & Rashmi Prasad (eds.), *Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers* (COLING-16), 429–439. Association for Computational Linguistics. http://aclweb.org/anthology/C/C16/C16-1042.pdf. December 11-16, 2016.

Wehrli, Eric, Violeta Seretan & Luka Nerima. 2010. Sentence analysis and collocation identification. In *Proceedings of the Workshop on Multiword Expressions: From Theory to Applications* (MWE '10), 27–35. Association for Computational Linguistics. http://www.aclweb.org/anthology/W/W10/W10-??04.
<figure><img src="image_51.png" /></figure>
# Chapter 9

## Semi-automated resolution of inconsistency for a harmonized multiword-expression and dependency-parse annotation

Julian Brooke

The University of Melbourne

King Chan

The University of Melbourne

Timothy Baldwin

The University of Melbourne

This chapter presents a methodology for identifying and resolving various kinds of inconsistency in the context of merging dependency and multiword expression (MWE) annotations, to generate a dependency treebank with comprehensive MWE annotations. Candidates for correction are identified using a variety of heuristics, including an entirely novel one which identifies violations of MWE constituency in the dependency tree, and resolved by arbitration with minimal human intervention. Using this technique, we identified and corrected several hundred inconsistencies across both parse and MWE annotations, representing changes to a significant percentage (well over 10%) of the MWE instances in the joint corpus and a large difference in MWE tagging performance relative to earlier versions.

## 1 Introduction

The availability of gold-standard annotations is important for the training and evaluation of a wide variety of Natural Language Processing (NLP) tasks, includ-

<figure><img src="image_52.png" /></figure>

<footnote>Julian Brooke, King Chan & Timothy Baldwin. 2018. Semi-automated resolution of inconsistency for a harmonized multiword-expression and dependency-parse annotation. In *Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 245–262. Berlin: Language Science Press. DOI:10.5281/zenodo.1469565</footnote>
ing the evaluation of dependency parsers (Buchholz & Marsi 2006). In recent years, there has been a focus on multi-annotation of a single corpus, such as joint syntactic, semantic role, named entity, coreference and word sense annotation in Ontonotes (Hovy et al. 2006) or constituency, semantic role, discourse, opinion, temporal, event and coreference (among others) annotation of the Manually Annotated Sub-Corpus of the American National Corpus (Ide et al. 2010). As part of this, there has been an increased focus on harmonizing and merging existing annotated data sets as a means of extending the scope of reference corpora (Ide & Suderman 2007; Declerck 2008; Simi et al. 2015). This effort sometimes presents an opportunity to address conflicts among annotations, a worthwhile endeavour since even a small number of errors in a gold-standard syntactic annotation can, for example, result in significant changes in downstream applications (Habash et al. 2007). This chapter presents the results of a harmonization effort for the overlapping STREUSLE annotation (Schneider, Onuffer, et al. 2014) of multiword expressions (MWEs: Baldwin & Kim 2010) and dependency parse structure in the English Web Treebank (EWT: Bies et al. 2012), with the long-term goal of building reliable resources for joint MWE/syntactic parsing (Constant & Nivre 2016).

As part of merging these two sets of annotations, we use analysis of cross-annotation and type-level consistency to identify instances of potential annotation inconsistency, with an eye to improving the quality of the component and combined annotations. It is important to point out that our approach to identifying and handling inconsistencies does not involve re-annotating the corpus; instead we act as arbitrators, resolving inconsistency in only those cases where human intervention is necessary. Our three methods for identifying potentially problematic annotations are:

* a cross-annotation heuristic that identifies MWE tokens whose parse structure is incompatible with the syntactic annotation of the MWE;

* a cross-type heuristic that identifies *n*-grams with inconsistent token-level MWE annotations; and

* a cross-type, cross-annotation heuristic that identifies MWE types whose parse structure is inconsistent across its token occurrences.

The first of these is specific to this harmonization process, and as far as we are aware, entirely novel. The other two are adaptions of an approach to improving syntactic annotations proposed by Dickinson & Meurers (2003). After applying these heuristics and reviewing the candidates, we identified hundreds of errors in MWE annotation and about a hundred errors in the original syntactic annotations. We make available a tool that applies these fixes in the process of joining the two annotations into a single harmonized, corrected annotation, and release
the harmonized annotations in the form of HAMSTER (the HArmonized Multiword and Syntactic TreE Resource): https://github.com/eltimster/HAMSTER. This chapter goes beyond the MWE2017 paper that first introduced HAMSTER (Chan et al. 2017) to show that the application of these and other corpus fixes has a major effect on MWE identification performance: we find that almost a quarter of the error originally assumed to be tagger error is actually attributable to errors in the corpus.

## 2 Related work

Our long-term goal is building reliable resources for joint MWE/syntactic parsing. Explicit modelling of MWEs has been shown to improve parser accuracy (Nivre 2004; Seretan & Wehrli 2006; Finkel & Manning 2009; Korkontzelos & Manandhar 2010; Green et al. 2013; Vincze et al. 2013; Wehrli 2014; Candito & Constant 2014; Constant & Nivre 2016). Treatment of MWEs has typically involved parsing MWEs as single lexical units (Nivre 2004; Eryiğit et al. 2011; Foptopoulou et al. 2014), but this flattened, “words with spaces” (Sag et al. 2002) approach is inflexible in its coverage of MWEs where components have some level of flexibility.

The English Web Treebank (Bies et al. 2012) represents a gold-standard annotation effort over informal web text. The original syntactic constituency annotation of the corpus was based on hand-correcting the output of the Stanford Parser (Manning et al. 2014); for our purposes we have converted this into a dependency parse using the Stanford Typed Dependency converter (de Marneffe et al. 2006). We considered the use of the Universal Dependencies representation (Nivre et al. 2016), but we noted that several aspects of that annotation (in particular the treatment of all prepositions as case markers dependent on their noun) make it inappropriate for joint MWE/syntactic parsing since it results in large numbers of MWEs that are non-continuous in their syntactic structure (despite being continuous at the token level).<fnref n="1" /> As such, the Stanford Typed Dependencies is the representation which has the greatest currency for joint MWE/syntactic parsing work (Constant & Nivre 2016).

The STREUSLE corpus (Schneider, Onuffer, et al. 2014) is based entirely on the Reviews subset of the EWT, and comprises 3,812 sentences representing 55,579 tokens. The annotation was completed by six linguists who are native English

<footnote n="1">¹An example of this would be a phrase such as *think of home*, where we consider *think of* to be an MWE, but the Universal Dependencies framework would treat *of* as a syntactic dependent of *home*, not *think*.</footnote>
<figure><img src="image_53.png" /><figcaption>Figure 1: An example where the arc count heuristic is breached. *Deep tissue* has been labeled in the sentence here as an MWE in STREUSLE. *Deep* and *tissue* act as modifiers to *massage*, a term that has not been included as part of the MWE.</figcaption></figure>

speakers. Every sentence was assessed by at least two annotators, which resulted in an average inter-annotator F1 agreement of 0.7. The idiosyncratic nature of MWEs lends itself to challenges associated with their interpretation, and this was readily acknowledged by those involved in the development of the STREUSLE corpus (Hollenstein et al. 2016). Two important aspects of the MWE annotation are that it includes both continuous and non-continuous MWEs (e.g., **check* *out**), and that it supports both **WEAK** and **STRONG** annotation. With regards to the latter, a variety of cues are employed to determine associative strength. The primary factor relates to the degree in which the expression is semantically opaque and/or morphosyntactically idiosyncratic. An example of a strong MWE would be **top notch**, as used in the sentence: *We stayed at a top notch hotel*. The semantics of this expression are not immediately predictable from the meanings of *top* and *notch*. On the other hand, the expression **highly recommend** is considered to be a weak expression as it is largely compositional – one can **highly recommend** a product – as indicated by the presence of alternatives such as **greatly recommend** which are also acceptable though less idiomatic. A total of 3,626 MWE instances were identified in STREUSLE, across 2,334 MWE types.

Other MWE-aware dependency treebanks include the various UD treebanks (Nivre et al. 2016), the Prague Dependency Treebank (Bejček et al. 2013), the Redwoods Treebank (Oepen et al. 2002), and others (Nivre 2004; Eryiğit et al. 2011; Candito & Constant 2014). The representation of MWEs, and the scope of types covered by these treebanks, can vary significantly. For example, the internal syntactic structure may be flattened (Nivre 2004), or in the case of Candito & Constant (2014), allow for distinctions in the granularity of syntactic representation for regular vs. irregular MWE types.

The identification of inconsistencies in annotation requires comparisons to be made between similar instances that are labeled differently. Boyd et al. (2007) employed an alignment-based approach to assess differences in the annotation
of *n*-gram word sequences in order to establish the likelihood of error occurrence. Other work in the syntactic inconsistency detection domain includes those related to POS tagging (Loftsson 2009; Eskin 2000; Ma et al. 2001) and parse structure (Ule & Simov 2004; Kato & Matsubara 2010). Dickinson & Meurers (2003) outline various approaches for detecting inconsistencies in parse structure within treebanks.

In general, inconsistencies associated with MWE annotation fall under two categories: (1) annotator error (i.e. false positives and false negatives); and (2) ambiguity associated with the assessment of hard cases. While annotation errors apply to situations where a correct label can be applied but is not, hard cases are those where the correct label is inherently difficult to assign. We address both these categories in this work.

# 3 Error candidate identification

## 3.1 MWE syntactic constituency conflicts

The hypothesis that drives our first analysis is that for nearly all MWE types, the component words of the MWE should be syntactically connected, which is to say that every word is a dependent of another word in the MWE, except one word which connects the MWE to the rest of the sentence (or the root of the sentence). We can realise this intuition by using an arc-count heuristic: for each labeled MWE instance we count the number of incoming dependency arcs that are headed by a term outside the MWE, and if the count is greater than one, we flag it for manual analysis. Figure 1 gives an example where the arc count heuristic is breached since both terms of the MWE *deep tissue* act as modifiers to the head noun that sits outside the MWE.

## 3.2 MWE type inconsistency

Our second analysis involves first collecting a list of all MWE types in the STREUSLE corpus, corresponding to lemmatized *n*-grams, possibly with gaps. We then match these *n*-grams across the same corpus, and flag any MWE type which has at least one inconsistency with regards to the annotation. That is, we extract as candidates any MWE types where there were at least two occurrences of the corresponding *n*-gram in the corpus that were incompatible with respect to their annotation in STREUSLE, including discrepancies in weak/STRONGISH designation. For non-continuous MWE types, matches containing up to 4 words of intervening context between the two parts of the MWE type were included as candidates
for further assessment. Some examples of many *n*-gram types which showed in-
consistency in their MWE annotation include *interested in, high quality, ask
for, in town, pizza place, even though*, and *easy to work with*.

3.3 MWE type parse inconsistency

The hypothesis that drives our third analysis is that we would generally expect the internal syntax of an MWE type to be consistent across all its instances.<fnref n="2" /> For each MWE type, we extracted the internal dependency structure of all its labeled instances, and flagged for further assessment any type for which the parse structure (including typed dependency label) varied between at least two of those instances. Note that although this analysis is aimed at fixing parse errors, it makes direct use of the MWE annotation provided by STREUSLE to greatly limit the scope of error candidates to those which are most relevant to our interest. Some MWE types which showed syntactic inconsistency include *years old, up front, set up, check out, other than*, and *get in touch with*.

4 Error arbitration

Error arbitration was carried out by the authors (all native English speakers with
experience in MWE identification), with at least two authors looking at each er-
ror candidate in most instances, and for certain difficult cases, the final annota-
tion being based on discussion among all three authors. One advantage of our
arbitration approach over a traditional token-based annotation was that we could
enforce consistency across similar error candidates (e.g., *disappointed with* and
*happy with*) and also investigate non-candidates to arrive at a consensus; where
at all possible, our changes relied on precedents that already existed in the rele-
vant annotation.

Arbitration for the MWE syntax conflicts usually involved identifying an error
in one of the two annotations, and in most cases this was relatively obvious. For
instance, in the candidate ... *the usual lady called in sick hours earlier, called in
sick* was correctly labeled as an MWE, but the parse incorrectly includes *sick* as
a dependent of *hours*, rather than *called in*. An example of the opposite case is
... *just to make the appointment* ..., where *make the* had been labeled as an MWE,
an obvious error which was caught by our arc count heuristic. There were cases
where our arc count heuristic was breached due to what we would view as a

<footnote n="2">²Noting that we would not expect this to occur between MWE instances of a given combination of words, and non-MWE combinations of those same words.</footnote>
general inadequacy in the syntactic annotation, but we decided not to effect a
change because the impact would be too far reaching; examples of this were
certain discourse markers (e.g., *as soon as*), and infinitives (e.g., *have to complete
where the to* is considered a dependent of its verb rather than of the other term in
the MWE *have to*). The most interesting cases were a handful of non-continuous
MWEs where there was truly a discontinuity in the syntax between the two parts
of the MWE, for instance *no amount of * can*. This suggests a basic limitation
in our heuristic, although the vast majority of MWEs did satisfy it.

For the two type-level arbitrations, there were cases of inconsistency upheld
by real usage differences (e.g., *a little house* vs. *a little tired*). We identified clear
differences in usage first and divided the MWE types into sets, excluding from
further analysis non-MWE usages of MWE type *n*-grams. For each consistent
usage of an MWE type, the default position was to prefer the majority annotation
across the set of instances, except when there were other candidates that were
essentially equivalent: for instance, if we had relied on majority annotation for
*job* * *do* (e.g., *the job that he did*) it would have been a different annotation than
*do* * *job* (e.g., *do a good job*), so we considered these two together. We treated
continuous and non-continuous versions of the same MWE type in the same
manner.

In the MWE type consistency arbitration, for cases where majority rules did
not provide a clear answer and there was no overwhelming evidence for non-
compositionality, we introduced a special internal label called *hard*. These corre-
spond to cases where the usage is consistent and the inconsistency seems to be
a result of the difficulty of the annotation item (as discussed earlier in Section 2),
which extended also to our arbitration. Rather than enforce a specific annotation
without strong evidence or allow the inconsistency to remain when there is no
usage justification for it, the corpus merging and correction tool gives the user
the option to treat hard annotated MWEs in varying ways: the annotation may
be kept unchanged, removed, converted to weak, or converted to hard for the
purpose of excluding it from evaluation. Examples of hard cases include *go back*,
*go in*, *more than*, *talk to*, *speak to*, *thanks guys*, *not that great*, *pleased with*,
*have* * *option*, *get* * *answer*, *fix* * *problem*. On a per capita basis, inconsistencies
are more common for non-continuous MWEs relative to their continuous coun-
terparts, and we suspect that this is partially due to their tendency to be weaker,
in addition to the challenges involved in correctly discerning the non-continuous
parts, which are sometimes at a significant distance from each other.

Table 1 provides a summary of changes to MWE annotation at the MWE type
and token levels. *Mixed* refer to MWEs that are heterogeneous in the associative
strength between terms in the MWE (between weak and strongish). Most of the changes in Table 1 (98% of the types) were the result of our type consistency analysis. Almost half of the changes involved the use of the *hard* label, but even excluding these (since only some of these annotations required actual changes in the final version of the corpus) our changes involve over 10% of the MWE tokens in the corpus, and thus represent a significant improvement to the STREUSLE annotation.

Relative to the changes to the MWE annotation, the changes to the parse annotation were more modest, but still not insignificant: for 161 MWE tokens across 72 types, we identified and corrected a dependency and/or POS annotation error. The majority of these (67%) were identified using the arc count heuristic. Note we applied the parse relevant heuristics after we fixed the MWE type consistency errors, ensuring that MWE annotations that were added were duly considered for parse errors.

<table><caption>Table 1: Summary of changes to MWE annotation at the MWE type and token level.</caption><thead><tr><th></th><th></th><th>No MWE</th><th>Weak</th><th>Strong</th><th>Mixed</th><th>Hard</th><th>TOTAL</th></tr></thead><tbody><tr><td rowspan="4">Token</td><td>No MWE</td><td>—</td><td>55</td><td>136</td><td>6</td><td>151</td><td>348</td></tr><tr><td>Weak</td><td>35</td><td>—</td><td>22</td><td>4</td><td>46</td><td>107</td></tr><tr><td>Strong</td><td>44</td><td>42</td><td>—</td><td>9</td><td>70</td><td>165</td></tr><tr><td>Mixed</td><td>2</td><td>4</td><td>3</td><td>12</td><td>2</td><td>23</td></tr><tr><td></td><td>TOTAL</td><td>81</td><td>101</td><td>161</td><td>31</td><td>269</td><td>643</td></tr><tr><td rowspan="4">Type</td><td>No MWE</td><td>—</td><td>31</td><td>74</td><td>5</td><td>64</td><td>174</td></tr><tr><td>Weak</td><td>31</td><td>—</td><td>13</td><td>4</td><td>35</td><td>83</td></tr><tr><td>Strong</td><td>34</td><td>28</td><td>—</td><td>7</td><td>43</td><td>112</td></tr><tr><td>Mixed</td><td>2</td><td>4</td><td>3</td><td>7</td><td>2</td><td>18</td></tr><tr><td></td><td>TOTAL</td><td>67</td><td>63</td><td>90</td><td>23</td><td>144</td><td>387</td></tr></tbody></table>

# 5 Experiments

In this section we investigate the effect of the HAMSTER MWE inconsistency fixes on the task of MWE identification. For this we use the AMALGr MWE identification tool of Schneider, Danchik, et al. (2014), which was developed on
the initial release of the STREUSLE (called then the CMWE).<fnref n="3" /> AMALGr is a su-
pervised structured perceptron model which makes use of external resources
including 10 MWE lexicons as well as Brown cluster information. For all our
experiments we use the default settings from Schneider, Danchik, et al. (2014),
including the original train/test splits and automatic part-of-speech tagging pro-
vided by the ARK TweetNLP POS tagger (Owoputi et al. 2013) trained on the
all non-review sections of the English Web Treebank. We note that in contrast
to typical experiments in NLP, here we are holding *the approach* constant while
varying the quality of the dataset, which provides a quantification of the extent
to which errors in the dataset interfered with our ability to build or accurately
evaluate models. Following Schneider, Danchik, et al. (2014), we report an F-score
which is calculated based on links between words: a true positive occurs when
two words which are supposed to appear together in an MWE do so as expected.

<table><caption>Table 2: AMALGr F-scores for various versions of MWE annotation of EWT Reviews.</caption>
  <thead>
    <tr>
      <th>Dataset</th>
      <th>F1-score (%)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>CMWE (Schneider, Danchik, et al. 2014)</td>
      <td>59.4</td>
    </tr>
    <tr>
      <td>STREUSLE 3.0</td>
      <td>64.6</td>
    </tr>
    <tr>
      <td>HAMSTER-original</td>
      <td>69.1</td>
    </tr>
    <tr>
      <td>HAMSTER-notMWE</td>
      <td>68.2</td>
    </tr>
    <tr>
      <td>HAMSTER-weak</td>
      <td>69.4</td>
    </tr>
    <tr>
      <td>HAMSTER-original-noeval</td>
      <td>70.2</td>
    </tr>
    <tr>
      <td>HAMSTER-weak-noeval</td>
      <td>69.3</td>
    </tr>
    <tr>
      <td>HAMSTER-original-test</td>
      <td>67.1</td>
    </tr>
    <tr>
      <td>HAMSTER-original-train</td>
      <td>65.7</td>
    </tr>
  </tbody>
</table>

There are two baselines in Table 2: the first is the original performance of AMALGr as reported in Schneider, Danchik, et al. (2014) using CMWE (version 1.0 of this annotation), and the second is its performance using STREUSLE (version 3.0). Note that these involve exactly the same texts: the difference between these two numbers reflects other fixes to this dataset that have happened in the

<footnote n="3">³The key difference between the CMWE and STREUSLE is the inclusion of supersense tags. Though we hope to eventually include supersense information in the output of HAMSTER, supersenses are beyond the scope of the present work.</footnote>
years since its initial release. The difference between the two is quite substantial, at roughly 5% F-score.

The rest of the table makes use of HAMSTERized versions of STREUSLE, which we refer to as simply HAMSTER. The options here mostly refer to our treatment of the *hard* cases, which must be removed to make use of AMALGr. *-original* indicates that we apply all fixes which result in the creation or removal of a standard STREUSLE label (i.e., weak and strongish), but leave *hard* annotations as they were in the original corpus. *-notMWE* and *-weak* create versions of the corpus where all *hard* labels have been mapped to either nothing (no MWE) or weak MWEs, respectively. Another option we consider is *-noeval*, which involved tweaking the AMALGr evaluation script to exclude particular annotations (in this case *hard*) from evaluation altogether; that is, it does not matter what the model predicted for those words which are considered *hard*. Finally, *-test* and *-train* refer to the situation where we apply our fixes to texts only in the test or training sets, respectively; this gives us a sense of whether the improved performance of the model over the HAMSTER datasets is primarily due to the removal of errors from the test set, or whether improving the consistency of the training set is playing a major role as well.

Our fixes result in roughly another 5% increase to F-score relative to STREUSLE 3.0, for a total of about 10% F-score difference relative to results using the original CMWE annotation of this corpus. With respect to options for phrases labeled as *hard*, treating them as nonMWEs seems to be a worse option than simply leaving them alone; the best explanation for this is probably that these hard cases are generally more similar to labelled MWEs. Treating them as weak appears to a better strategy. Even better, though, might be to leave *hard* inconsistencies in the training set but exclude them from consideration during testing. The results using mixed training/test datasets indicate that the fixes to the test data are clearly more important, but the consistency across the two sets also accounts for a major part of the performance increase seen here.

Our second round of experiments looks at exact match recall with respect to various subsets of the MWEs in the test set. Here we consider only the original STREUSLE and HAMSTERized version with *hard* MWEs unchanged. *N* is the number of MWEs labeled as that type in that version of the dataset. Our goal here is to get a sense of how our changes have affected the identification of specific kinds of MWE. Weak versus strongish is an obvious distinction (mixed MWE were considered strongish), but even more relevant to what we have done here is whether or not the MWE appears in both the training and test sets. We are also interested in the status of multiword named entities (identified fairly reliably
using proper noun tags in the gold-standard POS tags), which occur numerously in a corpus of reviews, but often as singletons, i.e., with a frequency of one. We would expect MWEs which neither appear in our corpus nor are named entities (NEs) to be relatively unaffected by our fixes, and among the most challenging MWEs to identify in general.

<table><caption>Table 3: AMALGr exact recall for different MWE subsets in original and HAMSTERized STREUSLE.</caption><thead><tr><th rowspan="2">MWE types</th><th colspan="2">STREUSLE</th><th colspan="2">HAMSTER</th></tr><tr><th>N</th><th>Recall (%)</th><th>N</th><th>Recall (%)</th></tr></thead><tbody><tr><td>All</td><td>423</td><td>59.7</td><td>444</td><td>63.4</td></tr><tr><td>strongish</td><td>352</td><td>63.2</td><td>368</td><td>66.3</td></tr><tr><td>weak</td><td>71</td><td>24.0</td><td>76</td><td>35.5</td></tr><tr><td>In training</td><td>178</td><td>77.7</td><td>208</td><td>80.1</td></tr><tr><td>Not in training</td><td>247</td><td>47.4</td><td>238</td><td>49.4</td></tr><tr><td>Named entity (NE)</td><td>52</td><td>73.5</td><td>52</td><td>71.6</td></tr><tr><td>Not NE, not in training</td><td>195</td><td>40.3</td><td>186</td><td>43.9</td></tr></tbody></table>

In Table 3, AMALGr does better with the HAMSTER dataset for most of the MWE subtypes considered here. The most striking difference occurs for the weak tag, reflecting a disproportionate amount of inconsistency, enough that the model built on the earlier version was apparently hesitant to apply the tag at all. Not only are MWEs with training instances tagged better after our fixes, but the set of such MWE tokens has noticeably increased. There is a corresponding drop in those test instances without training data, which are clearly the most difficult to identify, particularly when named entities are excluded. The recall of named entities has actually dropped slightly, though since there are only 52 of these in the test set, this corresponds to a single missed example and is probably not meaningful. Though the rationale in terms of higher-level semantics is clear, we wonder whether including NER as part of MWE identification may result in a distorted view of the importance of MWE lexicons in token-level MWE identification. Here, we can see that among test-set-only MWEs, they stand out as being significantly easier than the rest, probably because in English they can be identified fairly reliably using only capitalization.
# 6 Discussion

Our three heuristics are useful because they identify potential errors with a high degree of precision. For the MWE type consistency analysis, 77% of candidate types were problematic, and for parse type consistency, the number was 63%. For the arc count heuristic, 54% of candidate types were ultimately changed: as mentioned earlier, many of the breaches involved systematic issues with annotation schema that we felt uncomfortable changing in isolation. By bringing these candidate instances to our attention, we were able to better focus our manual analysis effort, including in some cases looking across multiple related types, or even searching for specialist knowledge which could resolve ambiguities: for instance, in the example shown in Figure 1, though a layperson without reference material may be unsure whether it is tissue or massage which is considered to be deep, a quick online search indicates that the original EWT syntax is in error (deep modifies tissue).

However, it would be an overstatement to claim to have fixed all (or even almost all) the errors in the corpus. For instance, our type consistency heuristics only work when there are multiple instances of the same type, yet it is worth noting that 82% of the MWE types in the corpus are represented by a singleton instance. Our arc count heuristic can identify issues with singletons, but its scope is fairly limited. We cannot possibly identify missing annotations for types that were not annotated at least once. We might also miss certain kinds of systematic annotation errors, for instance those mentioned in De Smedt et al. (2015), though that work focused on the use of MWE dependency labels which are barely used in the EWT, one of the reasons a resource like STREUSLE is so useful.

Our experiments with the AMALGr tool show that our fixes result in a major improvement in MWE identification. One particularly striking result is the fact that the errors identified in the annotation since its original release account for about a quarter of all error (as measured by F-score) in the original model trained on it. This error may affect relative comparisons between systems, and we should be skeptical of results previously drawn based on relatively small differences in MWE identification in earlier versions of the corpus (e.g., Qu et al. 2015). This amount of error is also unacceptable simply in terms of the obfuscation relative to the degree of absolute progress on the task. Beyond this specific effort, we believe, for annotation efforts in general and for MWEs in particular, we should move beyond a singular focus on achieving sufficient annotator agreement in the initial annotation – the agreement in the original CWME was impressively high – and instead develop protocols for semi-automated, type-level inconsistency detection as a default step before any annotation is released.
# 7 Conclusion

We have proposed a methodology for merging MWE and dependency parse annotations, to generate HAMSTER: a gold-standard MWE-annotated dependency treebank with high consistency. The heuristics used to enforce consistency operate at the type- and cross-annotation level, and affected well over 10% of the MWEs in the new resource, resulting in a downstream change in MWE identification of roughly 5% F-score. More generally, we have provided here a case study in how bringing together multiple kinds of annotation done over the same corpus can facilitate rigorous error correction as part of the harmonization process.

# Abbreviations

<table><tr><td>AMALGr</td><td>A Machine Analyzer of Lexical Groupings</td></tr><tr><td>CMWE</td><td>The Comprehensive Multiword Expression Corpus</td></tr><tr><td>EWT</td><td>The English Web Treebank</td></tr><tr><td>HAMSTER</td><td>The Harmonized Multiword and Syntactic Tree Resource</td></tr><tr><td>MWE</td><td>multiword expression</td></tr><tr><td>NE</td><td>named entity</td></tr><tr><td>NER</td><td>named entity recognition</td></tr><tr><td>NLP</td><td>Natural Language Processing</td></tr><tr><td>STREUSLE</td><td>Supersense-Tagged Repository of English with a Unified Semantics for Lexical Expressions</td></tr></table>

# References

Baldwin, Timothy & Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya & Fred J. Damerau (eds.), *Handbook of Natural Language Processing*, Second edition, 267–292. Boca Raton: CRC Press.

Bejček, Eduard, Eva Hajičová, Jan Hajič, Pavlína Jínová, Václava Kettnerová, Veronika Kolářová, Marie Mikulová, Jiří Mírovský, Anna Nedoluzhko, Jarmila Panevová, Lucie Poláková, Magda Ševčíková, Jan Štěpánek & Šárka Zikánová. 2013. *Prague dependency treebank 3.0*. Charles University in Prague, UFAL.

Bies, Ann, Justin Mott, Colin Warner & Seth Kulick. 2012. *English web treebank*. Tech. rep. LDC2012T13. Linguistic Data Consortium. http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId=LDC2012T13.
Boyd, Adriane, Markus Dickinson & Detmar Meurers. 2007. Increasing the recall of corpus annotation error detection. In *Proceedings of the Sixth Workshop on Treebanks and Linguistic Theories* (TLT 2007), 19–30. http://decca.osu.edu/publications/boyd-et-al-07b.html.

Buchholz, Sabine & Erwin Marsi. 2006. CoNLL-x shared task on multilingual dependency parsing. In *Proceedings of the 10th Conference on Computational Natural Language Learning* (CoNLL-X’06), 149–164. http://dl.acm.org/citation.cfm?id=1596276.1596305.

Candito, Marie & Matthieu Constant. 2014. Strategies for contiguous multiword expression analysis and dependency parsing. In *Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers)*, 743–753. Association for Computational Linguistics. http://www.aclweb.org/anthology/P14-1070.

Chan, King, Julian Brooke & Timothy Baldwin. 2017. Semi-automated resolution of inconsistency for a harmonized multiword expression and dependency parse annotation. In *Proceedings of the 13th Workshop on Multiword Expressions (MWE 2017)*, 187–193. Valencia, Spain. http://aclweb.org/anthology/W17-1726.

Constant, Matthieu & Joakim Nivre. 2016. A transition-based system for joint lexical and syntactic analysis. In *Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers)*, 161–171. Association for Computational Linguistics. http://www.aclweb.org/anthology/P16-1016.

de Marneffe, Marie-Catherine, Bill MacCartney & Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In *Proceedings of the 5th International Conference on Language Resources and Evaluation* (LREC 2006). European Language Resources Association (ELRA).

De Smedt, Koenraad, Victoria Rosén & Paul Meurer. 2015. Studying consistency in UD treebanks with INESS-Search. In *Proceedings of the 14th Workshop on Treebanks and Linguistic Theories* (TLT14), 258–267.

Declerck, Thierry. 2008. A framework for standardized syntactic annotation. In *Proceedings of the 6th international on language resources and evaluation* (LREC 2008), 3025–3028.

Dickinson, Markus & W. Detmar Meurers. 2003. Detecting inconsistencies in treebanks. In *Proceedings of the Second Workshop on Treebanks and Linguistic Theories* (TLT 2003), 45–56. 14-15 November, 2003.

Eryiğit, Gülşen, Tugay İlbay & Ozan Arkan Can. 2011. Multiword expressions in statistical dependency parsing. In *Proceedings of IWPT Workshop on Statistical*
Parsing of *Morphologically-Rich Languages* (SPMRL 2011), 45–55. http://dl.acm.org/citation.cfm?id=2206359.2206365. October 6, 2011.

Eskin, Eleazar. 2000. Detecting errors within a corpus using anomaly detection. In *Proceedings of the First North American Chapter of the Association for Computational Linguistics Conference*, 148–153. http://dl.acm.org/citation.cfm?id=974305.974325. April 29 - May 04, 2000.

Finkel, Jenny Rose & Christopher D. Manning. 2009. Joint parsing and named entity recognition. In *Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics* (NAACL ’09), 326–334. Association for Computational Linguistics. http://dl.acm.org/citation.cfm?id=1620754.1620802. May 31 - June 05, 2009.

Fotopoulou, Angeliki, Stella Markantonatou & Voula Giouli. 2014. Encoding MWEs in a conceptual lexicon. In *Proceedings of the 10th Workshop on Multiword Expressions* (MWE ’14), 43–47. Association for Computational Linguistics.

Green, Spence, Marie-Catherine de Marneffe & Christopher D. Manning. 2013. Parsing models for identifying multiword expressions. *Computational Linguistics* 39(1). 195–227. DOI:10.1162/COLI_a_00139

Habash, Nizar, Ryan Gabbard, Owen Rambow, Seth Kulick & Mitchell P. Marcus. 2007. Determining case in Arabic: Learning complex linguistic behavior requires complex linguistic features. In *Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning 2007* (EMNLP-CoNLL 2007), 1084–1092.

Hollenstein, Nora, Nathan Schneider & Bonnie Webber. 2016. Inconsistency detection in semantic annotation. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the 10th International Conference on Language Resources and Evaluation* (LREC 2016), 3986–3990. European Language Resources Association (ELRA).

Hovy, Eduard, Mitchell P. Marcus, Martha Palmer, Lance Ramshaw & Ralph Weischedel. 2006. OntoNotes: The 90% solution. In *Proceedings of the Main Conference of Human Language Technology of the North American Chapter of the Association for Computiational Linguistics*, 57–60.

Ide, Nancy, Collin Baker, Christiane Fellbaum & Rebecca Passonneau. 2010. The Manually Annotated Sub-Corpus: A community resource for and by the people. In *Proceedings of the 48th annual meeting of the ACL (ACL 2010)- short papers*, 68–73. Uppsala, Sweden.
Ide, Nancy & Keith Suderman. 2007. GrAF: A Graph-based format for linguistic annotations. In *Proceedings of the Linguistic Annotation Workshop*, 1–8. Prague, Czech Republic. http://dl.acm.org/citation.cfm?id=1642059.1642060.

Kato, Yoshihide & Shigeki Matsubara. 2010. Correcting errors in a treebank based on synchronous tree substitution grammar. In *Proceedings of the ACL 2010 Conference-Short Papers*, 74–79. Association for Computational Linguistics. July 11 - 16, 2010.

Korkontzelos, Ioannis & Suresh Manandhar. 2010. Can recognising multiword expressions improve shallow parsing? In *Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics* (HLT ’10), 636–644. http://dl.acm.org/citation.cfm?id=1857999.1858088.

Loftsson, Hrafn. 2009. Correcting a POS-tagged corpus using three complementary methods. In *Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics* (EACL ’09), 523–531. Association for Computational Linguistics.

Ma, Qing, Bao-Liang Lu, Masaki Murata, Michnori Ichikawa & Hitoshi Isahara. 2001. On-line error detection of annotated corpus using modular neural networks. In *International Conference on Artificial Neural Networks*, 1185–1192.

Manning, Christopher D., Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard & David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In *Proceedings of the 52nd annual meeting of the association for computational linguistics: System demonstrations*, 55–60.

Nivre, Joakim. 2004. Incrementality in deterministic dependency parsing. In Frank Keller, Stephen Clark, Matthew Crocker & Mark Steedman (eds.), *Proceedings of the ACL Workshop on Incremental Parsing: Bringing Engineering and Cognition together*, 50–57. Association for Computational Linguistics.

Nivre, Joakim, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty & Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the Tenth International Conference on Language Resources and Evaluation* (LREC 2016), 1659–1666. European Language Resources Association (ELRA). 23-28 May, 2016.
Oepen, Stephan, Dan Flickinger, Kristina Toutanova & Christoper D. Manning.
2002. LinGO Redwoods: A rich and dynamic treebank for HPSG. In *Proceedings
of The First Workshop on Treebanks and Linguistic Theories* (TLT2002).

Owoputi, Olutobi, Brendan O'Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider & Noah A. Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In *Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics*.

Qu, Lizhen, Gabriela Ferraro, Liyuan Zhou, Weiwei Hou, Nathan Schneider & Timothy Baldwin. 2015. Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: the impact of word representations on sequence labelling tasks. In *Proceedings of the SIGNLL Conference on Computational Natural Language Learning* (CoNLL 2015), 83–93.

Sag, Ivan A., Timothy Baldwin, Francis Bond, Ann A. Copestake & Dan Flickinger.
2002. Multiword expressions: A pain in the neck for NLP. In *Proceedings of the
3rd International Conference on Computational Linguistics and Intelligent Text
Processing*, vol. 2276/2010 (CICLing '02), 1–15. Springer-Verlag.

Schneider, Nathan, Emily Danchik, Chris Dyer & Noah A. Smith. 2014. Discriminative lexical semantic segmentation with gaps: Running the MWE gamut. *Transactions of the Association for Computational Linguistics* 2(1). 193–206. https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/281.

Schneider, Nathan, Spencer Onuffer, Nora Kazour, Nora Emily Danchik, Michael T. Mordowanec, Henrietta Conrad & Smith Noah A. 2014. Comprehensive annotation of multiword expressions in a social web corpus. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the 9th International Conference on Language Resources and Evaluation* (LREC 2014), 455–461. European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2014/pdf/521_Paper.pdf.

Seretan, Violeta & Eric Wehrli. 2006. Accurate collocation extraction using a multilingual parser. In *Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics*, 953–960. 17-18 July 2006.

Simi, Maria, Simonetta Montemagni & Cristina Bosco. 2015. Harmonizing and merging Italian treebanks: Towards a merged Italian dependency treebank and beyond. In Roberto Basili, Cristina Bosco, Rodolfo Delmonte, Alessandro Moschitti & Maria Simi (eds.), *Harmonization and development of resources and tools for Italian Natural Language Processing within the PARLI project*, 3–23. Heidelberg, Germany: Springer. DOI:10.1007/978-3-319-14206-7_1
Ule, Tylman & Kiril Simov. 2004. Unexpected productions May well be errors. In Maria Teresa Lino, Maria Francisca Xavier, Fátima Ferreira, Rute Costa, Raquel Silva, Carla Pereira, Filipa Carvalho, Milene Lopes, Mónica Catarino & Sérgio Barros (eds.), *Proceedings of the 4th international conference on language resources and evaluation* (LREC 2004), 1795–1798.

Vincze, Veronika, János Zsibrita & István Nagy T. 2013. Dependency parsing for identifying Hungarian light verb constructions. In *Proceedings of the 6th International Joint Conference on Natural Language Processing*, 207–215. Nagoya, Japan: Asian Federation of Natural Language Processing. http://www.aclweb.org/anthology/I13-1024.

Wehrli, Eric. 2014. The relevance of collocations for parsing. In *Proceedings of the 10th Workshop on Multiword Expressions* (MWE ’14), 26–32. Association for Computational Linguistics. 26-27 April, 2014.
# Chapter 10

## Sequence models and lexical resources for MWE identification in French

Manon Scholivet

Aix Marseille Univ, Université de Toulon, CNRS, LIS, Marseille, France

Carlos Ramisch

Aix Marseille Univ, Université de Toulon, CNRS, LIS, Marseille, France

Silvio Cordeiro

Institute of Informatics, Federal University of Rio Grande do Sul, Brazil

Aix Marseille Univ, Université de Toulon, CNRS, LIS, Marseille, France

We present a simple and efficient sequence tagger capable of identifying continuous multiword expressions (MWEs) of several categories in French texts. It is based on conditional random fields (CRF), using as features local context information such as previous and next word lemmas and parts of speech. We show that this approach can obtain results that, in some cases, approach more sophisticated parser-based MWE identification methods without requiring syntactic trees from a treebank. Moreover, we study how well the CRF can take into account external information coming from both high-quality hand-crafted lexicons and MWE lists automatically obtained from large monolingual corpora. Results indicate that external information systematically helps improving the tagger's performance, compensating for the limited amount of training data.

## 1 Introduction

Identifying multiword expressions (MWEs) in running texts with the help of lexicons could be considered as a trivial search-and-replace operation. In theory, one could simply scan the text once and mark (e.g. join with an underscore)

<figure><img src="image_54.png" /></figure>
all sequences of tokens that appear as headwords in the MWE lexicons. Direct matching and projection of lexical entries onto the corpus can be employed as a simple yet effective preprocessing step prior to dependency parsing (Nivre & Nilsson 2004) and machine translation (Carpuat & Diab 2010). Upon recognition, the identified member words of an MWE can be concatenated and treated as single token, that is, a “word with spaces”, as suggested by Sag et al. (2002).

However, this simple pipeline will fail when dealing with frequent categories of MWEs that present some challenging characteristics such as **variability** and **ambiguity**. For many MWE categories, **variability** due to morphological inflection may pose problems. For instance, if a lexicon contains the idiom *to make a face*, string matching will fail to identify it in *children are always making faces* because the verb and the noun are inflected.<fnref n="1" /> Since lexicons usually contain canonical (lemmatised) forms, matching must take inflection into account. This can be carried out by (a) pre-analysing the text and matching lemmas instead of surface-level word forms (Finlayson & Kulkarni 2011), or by (b) looking up lexicons containing inflected MWEs (Silberztein et al. 2012).

Things get more complicated when the target MWEs are ambiguous, though. An MWE is **ambiguous** when its member words can co-occur without forming an expression. For instance, *to make a face* is an idiom meaning ‘to show a funny facial expression’, but it can also be used literally when someone is making a snowman (Fazly et al. 2009). Additionally, the words in this expression can co-occur by chance, not forming a phrase (Boukobza & Rappoport 2009; Shigeto et al. 2013). This is particularly common for multiword function words such as prepositions (e.g. *up to*), conjunctions (e.g. *now that*) and adverbials (e.g. *at all*). For example, *up to* is an MWE in *they accepted up to 100 candidates* but not in *you should look it up to avoid making a typo*. Similarly, *at all* is an adverbial in *they accepted no candidates at all*, but not in *this train does not stop at all stations*. Context-dependent statistical methods (Fazly et al. 2009; Boukobza & Rappoport 2009) and syntax-based methods (Candito & Constant 2014; Nasr et al. 2015) are usually employed to deal with semantic ambiguity and accidental co-occurrence, respectively.

In addition to variability and ambiguity, an additional challenge stems from the absence or limited coverage of high-quality hand-crafted lexical resources containing MWEs for many languages. Therefore, it is not always possible to em-

<footnote n="1">¹In addition, the determiner *a* is not mandatory. However, discontinuous expressions containing optional intervening words are out of the scope of this work because our method is based on sequence models and our corpora only contain continuous MWEs. An adaptation of sequence models to discontinuous expressions has been proposed by Schneider, Danchik, et al. (2014).</footnote>
ploy purely symbolic look-up methods for MWE identification. Statistical meth-
ods are an interesting alternative, since one can learn generic models for MWE
identification based on corpora where MWEs have been manually annotated. If
enough evidence is provided and represented at the appropriate level of granular-
ity, the model can make generalizations based on commonly observed patterns.
It may then be able to identify MWE instances that have never occurred in an-
notated training data. However, annotated corpora often do not contain enough
training material for robust MWE identification. Complementary evidence can
be obtained with the help of unsupervised MWE discovery methods that create
MWE lists from raw corpora, which are then considered as if they were hand-
crafted lexicons. In short, the heterogeneous landscape in terms of available re-
sources (annotated corpora, hand-crafted lexicons) motivates the development
of statistical MWE identification models that can exploit external hand-crafted
and automatically constructed lexicons as a complementary information source
(Constant & Sigogne 2011; Schneider, Danchik, et al. 2014; Riedl & Biemann 2016).

We propose a simple, fast and generic sequence model for tagging continuous MWEs based on conditional random fields (CRF). It cannot deal with discontinuous expressions, but is capable of modelling variable and highly ambiguous expressions. Moreover, we propose a simple adaptation to integrate information coming from external lexicons. Another advantage of our CRF is that we do not need syntactic trees to train our model, unlike methods based on parsers (Le Roux et al. 2014; Nasr et al. 2015; Constant & Nivre 2016). Moreover, for expressions that are syntactically fixed, it is natural to ask ourself if we really need a parser for this task. Parsers are good for non-continuous MWEs, but we hypothesise that continuous expressions can be modelled by sequence models that take ambiguity into account, such as CRFs. Regardless of the syntactic nature of these ambiguities, we expect that the highly lexicalised model of the CRF compensates for its lack of structure.

The present chapter contains three significant extensions with respect to our
previous publication at the MWE 2017 workshop (Scholivet & Ramisch 2017).
First, we train and test our models on two complementary datasets containing
nominal expressions and general MWEs in French. Second, we study the inte-
gration of automatically constructed MWE lexicons obtained with the help of
MWE discovery techniques. Third, we study the performance of our system on
particularly hard MWE instances such as those including variants and those that
do not occur in the training corpora.

In short, we demonstrate that, in addition to being well suited to identifying
highly ambiguous MWEs in French (Scholivet & Ramisch 2017), the proposed
model and its corresponding free implementation<fnref n="2" /> can also be applied to identify other MWE categories and use other types of external lexicons. We believe that this approach can be useful (a) when no treebank is available to perform parsing-based MWE identification, (b) when large monolingual corpora are available instead of hand-crafted lexical resources, and (c) as a preprocessing step to parsing, which can improve parsing quality by reducing attachment ambiguities (Candito & Constant 2014; Nivre & Nilsson 2004).

## 2 Related work

Token identification of MWEs in running text can be modelled as a machine learning problem, building an identification model from MWE-annotated corpora and treebanks. To date, it has been carried out using mainly two types of models: sequence taggers and parsers. Sequence taggers such as CRFs, structured support vector machines and structured perceptron allow disambiguating MWEs using local feature sets such as word affixes and surrounding word and POS n-grams. Parsers, on the other hand, can take into account longer-distance relations and features when building a parse tree, at the expense of using more complex models.

Sequence taggers have been proven useful in identifying MWEs. MWE identification is sometimes integrated with POS tagging in the form of special tags. Experiments have shown the feasibility of sequence tagging for general expressions and named entities in English (Vincze et al. 2011), verb-noun idioms in English (Diab & Bhutada 2009) and general expressions in French (Constant & Sigogne 2011) and in English (Schneider, Danchik, et al. 2014; Riedl & Biemann 2016). Shigeto et al. (2013) tackle specifically English function words and build a CRF from the Penn Treebank, additionally correcting incoherent annotations. We develop a similar system for French, using the MWE annotation of the French Treebank as training data and evaluating the model on a dedicated dataset.

Parsing-based MWE identification requires a treebank annotated with MWEs. Lexicalised constituency parsers model MWEs as special non-terminal nodes included in regular rules (Green et al. 2013). In dependency parsers, it is possible to employ a similar approach, using special dependency labels to identify relations between words that make up an expression (Candito & Constant 2014).

The work of Nasr et al. (2015) is a parsing-based approach evaluated on highly ambiguous grammatical MWEs in French (§5.1). In their work, they link word

<footnote n="2">²The CRF-MWE tagger described in this chapter is included in the mwetoolkit in the form of 2 scripts: *train_crf.py* and * annotate_crf.py*, freely available at http://mwetoolkit.sf.net/</footnote>
sequences belonging to complex conjunctions such as *bien que* 'well that' ⇒ 'al-
though' and partitive determiner such as *de la* 'of the' ⇒ 'some', using a special
dependency link called *morph*, similar to Universal Dependencies' *compound re-
lation* (Nivre et al. 2016). On the other hand, these word sequences can occur
by chance, such as in *Je pense bien que je suis malade*. 'I think well that I am
sick.' ⇒ 'I really think that I am sick'. Then, the adverb *well* modifies the verb
*think*, which in turn has a complement introduced by *that*. Nasr et al. (2015) train
a second-order graph-based dependency parser to distinguish *morph* from other
syntactic relations, implicitly identifying MWEs. In addition to standard features,
they extract features from a valence dictionary specifying whether a given verb
licences complements introduced by *que* 'that' or *de* 'of'.

Our hypothesis is that parsing-based techniques like this are not required to
obtain good performances on continuous expressions. Our paper adapts a stan-
dard CRF model inspired on the ones proposed by Constant & Sigogne (2011),
Riedl & Biemann (2016) and Shigeto et al. (2013) to deal with continuous MWEs.

Concerning external lexical resources, Nasr et al. (2015) have shown that their
features extracted from a valence dictionary can significantly improve identifica-
tion. Moreover, most systems based on sequence taggers also integrate additional
hand-crafted lexicons to obtain good results (Constant & Sigogne 2011; Schnei-
der, Danchik, et al. 2014). Nonetheless, the integration of automatically discov-
ered lexicons of MWEs has not been explored by many authors, with the notable
exception of Riedl & Biemann (2016). We show that our CRF can naturally han-
dle automatically and manually constructed lexicons and that, in both cases, the
system benefits from the extra information present in the lexicons.

3 A CRF-based MWE tagger

Linear-chain conditional random fields (CRFs) are an instance of stochastic mod-
els that can be employed for sequence tagging (Lafferty et al. 2001). Each input
sequence *T* is composed of *t*₁... *t*ₙ tokens considered as an observation. Each ob-
servation is tagged with a sequence *Y* = *y*₁... *y*ₙ of tags corresponding to the
values of the hidden states that generated them. CRFs can be seen as a discrimi-
nant version of hidden Markov models, since they model the conditional proba-
bility *P*(*Y*|*T*). This makes them particularly appealing since it is straightforward
to add customised features to the model. In first-order linear-chain CRFs, the
probability of a given output tag *y*ᵢ for an input word *x*ᵢ depends on the tag of
the neighbour token *y*ᵢ₋₁, and on a rich set of features of the input *ϕ*(*T*), that
can range over any position of the input sequence, including but not limited to
the current token $t_i$. CRF training consists in estimating individual parameters proportional to $p(y_i, y_{i-1}, \phi(T))$.

The identification of continuous MWEs is a segmentation problem. We use a tagger to perform this segmentation, employing the well-known Begin-Inside-Outside (BIO) encoding (Ramshaw & Marcus 1995). In BIO, every token $t_i$ in the training corpus is annotated with a corresponding tag $y_i$ with values B, I or O. If the tag is B, it means the token is the beginning of an MWE. If it is I, this means the token is inside an MWE. I tags can only be preceded by another I tag or by a B. Finally, if the token's tag is O, this means the token is outside the expression, and does not belong to any MWE. An example of such encoding for the 2-word expression *de la* 'some' in French is shown in Figure 1.

<table><caption>Figure 1: Example of BIO tagging of a French sentence containing a *De+determiner* expression, assuming that the current word ($w_0$) is *de*.</caption><thead><tr><th><i>i</i></th><th>-2</th><th>-1</th><th>0</th><th>1</th><th>2</th><th>3</th></tr></thead><tbody><tr><td>w<sub>i</sub></td><td>Il</td><td>jette</td><td>de</td><td>la</td><td>nourriture</td><td>périmée</td></tr><tr><td>y<sub>i</sub></td><td>O</td><td>O</td><td>B</td><td>I</td><td>O</td><td>O</td></tr><tr><td></td><td>He</td><td>discards</td><td>some</td><td>food</td><td>expired</td><td></td></tr></tbody></table>

For our experiments, we have trained a CRF tagger with the CRFSuite toolkit<fnref n="3" /> (Okazaki 2007). We used a modified version of the French treebank (Abeillé et al. 2003) as training, development, and test data, and the MORPH dataset<fnref n="4" /> (Nasr et al. 2015) as development and test data. We additionally include features from an external valence lexicon, DicoValence<fnref n="5" /> (van den Eynde & Mertens 2003), and from an automatically constructed lexicon of nominal MWEs obtained automatically from the frWaC corpus (Baroni & Bernardini 2006) with the help of the mwetoolkit (Ramisch 2014).

## 3.1 CRF features

Our set of features $\phi(T)$ contains 37 different combinations of values (henceforth referred to as the ALL feature set). Our features are inspired on those proposed by Constant & Sigogne (2011), and are similar to those used by Schneider, Danchik, et al. (2014) and Riedl & Biemann (2016). The feature templates described below

<footnote n="3">³http://www.chokkan.org/software/crfsuite/</footnote>

<footnote n="4">⁴http://pageperso.lif.univ-mrs.fr/~carlos.ramisch/?page=downloads/morph</footnote>

<footnote n="5">⁵http://bach.arts.kuleuven.be/dicovalence/</footnote>
consider that the current token $T_0$ has surface form $w_0$, lemma $L_0$ and POS $P_0$. In addition to output tag bigrams (CRF's first-order assumption), we consider the following feature templates in our model, to be regarded in conjunction with the current tag to predict:

* Single-token features ($T_i$):<fnref n="6" />

- $w_0$: wordform of the current token

- $L_0$: lemma of the current token

- $P_0$: POS tag of the current token

- $w_i$, $L_i$ and $P_i$: wordform, lemma or POS of previous ($i \in \{-1, -2\}$) or next ($i \in \{+1, +2\}$) tokens

* N-gram features (bigrams $T_{i-1}T_i$ and trigrams $T_{i-1}T_iT_{i+1}$):

- $w_{i-1}w_i$, $L_{i-1}L_i$, $P_{i-1}P_i$: wordform, lemma and POS bigrams of previous-current ($i=0$) and current-next ($i=1$) tokens

- $w_{i-1}w_iw_{i+1}$, $L_{i-1}L_iL_{i+1}$, $P_{i-1}P_iP_{i+1}$: wordform, lemma and POS trigrams of previous-previous-current ($i=-1$), previous-current-next ($i=0$) and current-next-next ($i=+1$) tokens

* Orthographic features (ORTH):

- **HYPHEN and DIGITS**: the current wordform $w_i$ contains a hyphen or digits

- **F-CAPITAL**: the first letter of the current wordform $w_i$ is uppercase

- **A-CAPITAL**: all letters of the current wordform $w_i$ are uppercase

- **B-CAPITAL**: the first letter of the current word $w_i$ is uppercase, and it is at the beginning of a sentence.

* Lexicon features (LF), described in more detail in §4.3:

- **QUEV**: the current wordform $w_i$ is *que*, and the closest verb to the left licences a complement introduced by *que* according to the valence dictionary DicoValence.<fnref n="7" />

- **DEV**: the current wordform $w_i$ is *de*, and the closest verb to the left licences a complement introduced by *de* according to the valence dictionary DicoValence.

<footnote n="6">⁶$T_i$ is a shortcut denoting the group of features $w_i$, $L_i$ and $P_i$ for a token $T_i$. In other words, each token $T_i$ is a tuple $(w_i, L_i, P_i)$. The same applies to n-grams.</footnote>

<footnote n="7">⁷**QUEV** and **DEV** are sequential versions of the *subcat* features proposed by Nasr et al. (2015).</footnote>
- Association measures (AM) between the current token's lemma $L_i$ and the previous tokens' lemmas:

* MLE: probability of the lemma sequence estimated using maximum likelihood estimation

* PMI: pointwise mutual information of the lemma sequence.

* DICE: Dice's coefficient of the lemma sequence

* T-MEAS: Student's t-score of the lemma sequence

* LL: log-likelihood ratio between the current lemma and the previous lemma

Our proposed feature set is similar to previous work, with only minor differences (Constant & Sigogne 2011; Schneider, Onuffer, et al. 2014; Riedl & Biemann 2016). Like all previous models, we encode output tags with BIO, and we consider as features the surface form of the current token, of surrounding tokens, and bigrams of those. Our orthographic features are practically identical to related work, but all previously proposed models include 4- to 5-character prefixes and suffixes, which we do not. The features proposed by Constant & Sigogne (2011) are only based on surface forms of words, given that their task is to jointly predict POS and MWE tags. On the other hand, the features of Schneider, Onuffer, et al. (2014) and Riedl & Biemann (2016) are based on current and surrounding lemmas and POS tags, and so are ours. Differently from these two articles, we rely on token trigram features and we do not use mixed lemma+POS features. The lemma-based features of Schneider, Onuffer, et al. (2014) are quite different from ours, because they are conditioned on particular POS tags. The main differences between previous models and ours are in the lexicon features: Constant & Sigogne (2011) and Schneider, Onuffer, et al. (2014) employ hand-crafted lexicons and extract more detailed information from them than we do. Riedl & Biemann (2016) cover both hand-crafted and automatically built lexicons. Their feature set has one feature in common with ours: Student's t-score. In short, the features are similar in nature, but present some arbitrary variation in their implementations, in addition to some variation due to the nature of the available lexicons and corpora.

Our training corpora contain syntactic dependency information. However, we decided not to include it as CRF features for two main reasons. First, we wanted to evaluate the hypothesis that sequence-based methods can perform MWE identification without resorting to treebanks, as opposed to parser-based identification. Second, representing syntactic structure in a CRF is tricky as the linear-chain model in our experiments is not adequate for representing general graphs.
Nonetheless, adding features based on simplified syntactic information (e.g. the
dependency label of each word with respect to its parent) is feasible and repre-
sents an interesting idea for future work.

4 Experimental setup

In order to evaluate our systems, we test them on four MWE categories in French:

* Adverb+*que* expressions (AQ): in French, adverbs (such as *bien* 'well') and the subordinating conjunction *que* 'that' are frequently combined to build complex conjunctions such as *bien que* 'well that' ⇒ 'although'. This category was chosen because (a) these expressions present little variability,<fnref n="8" /> and (b) they are highly ambiguous, since their components can co-occur by chance, as in *il sait bien que tu mens*. 'he knows well that you lie.' ⇒ 'he really knows that you are lying'. Thus, we can focus on ambiguity as a challenging problem to model.

* De+determiner expressions (DD): in French, partitive and plural determiners are formed by the word de 'of' followed by a definite article, for instance, *il mange de la salade, du pain et des fruits*. 'he eats of the.SG.FEM salad, of-the.SG.MASC bread and of-the.PL fruit.' ⇒ 'he is eating some salad, bread and fruit'. Similarly to AQ, these constructions present little variation<fnref n="9" /> and are ambiguous with preposition+article combinations such as *il parle de la salade* (lit. *he talks of the salad*) 'he talks about the salad'. Their disambiguation is challenging because it relies on the argumental structure of the verb governing the noun. Moreover, these are among the most frequent tokens in a corpus of French (Nasr et al. 2015).

* Nominal expressions: at a first moment, we focus on the identification of nominal expressions for two reasons. First, they present morphological variability but are syntactically fixed, making CRFs particularly suitable to model them. Second, we test the inclusion of automatically calculated association measures as features in the identification model, and our lexicon of pre-calculated association measures contains only nominal MWEs.

* General MWEs: finally, we evaluate our model on a corpus containing several categories of continuous MWEs. These include nominal expressions,

<footnote n="8">⁸The only variability that must be taken into account is that *que* is sometimes written as *qu*’ when the next word starts with a vowel.</footnote>

<footnote n="9">⁹Except for contractions *de+le=du* and *de+les=des*</footnote>
complex conjunctions and determiners such as AQ and DD combinations, fixed prepositional phrases, multiword named entities, some limited continuous verbal expressions such as *avoir lieu* (lit. *have place*) ‘take place’, and so on. Our training and test corpora do not contain any labels distinguishing these MWE categories. Therefore the only category-based analysis we perform relies on the POS tags of the component words (for nominal MWEs).

In our experiments, we used two annotated corpora: the French treebank and the MORPH dataset. Other corpora annotated with MWEs in French do exist (Laporte et al. 2008; Savary et al. 2017). However, we chose to evaluate our model on a dataset for which, at the time of writing this chapter, many studies on MWE identification methods have been reported (the French treebank) and on an in-house dataset focusing on ambiguous MWEs (MORPH). Hence, we can compare our sequence model with state-of-the art results and verify whether they are adequate to recognise ambiguous MWEs. Evaluation on other corpora is left for future work.

## 4.1 The French treebank

We train and test our models on the MWE-annotated French treebank (FTB), available in CONLL format and automatically transformed into the CRFsuite format. The FTB is traditionally split into three parts: train, dev and test. We train our systems systematically on the training part of the FTB, that we adapted to keep only the MWEs we are interested in. For the experiments where we considered general MWEs and nominal MWEs, we used the FTB version of the SPMRL shared task (Seddah et al. 2013). The FTB dev and test corpora were employed respectively for feature engineering and evaluation. For each word, the corpus contains its wordform, lemma, POS (15 different coarse POS tags), and syntactic dependencies (that were ignored).

In the original corpus, MWE information is represented as words with spaces. For instance, *bien que* appears as a single token with underscores when it is a complex conjunction, whereas accidental co-occurrence is represented as two separate tokens *bien* and *que*. We argue that using such gold tokenisation is unrealistic, especially in the case of ambiguous MWEs. We thus systematically split single-token MWEs and added an extra column containing MWE tags using BIO encoding (§3). Even though this preprocessing might sound artificial, we believe that it provides a more uniform treatment to ambiguous constructions, closer to their raw-text form. This assumption is in line with the latest developments in
the dependency parsing community, which has evolved from parser evaluation
on gold tokenisation (Buchholz & Marsi 2006) to evaluation on raw text (Zeman
et al. 2017).

The MWE-BIO tags were generated using the following transformation heuristics in the case of ambiguous AQ and DD MWEs:

* For AQ expressions:

1. We scan the corpus looking for the lemmas *ainsi_que*, *alors_que*, *autant_que*, *bien_que*, *encore_que*, *maintenant_que* and *tant_que*.

2. We split them into two tokens and tag the adverb as B and que as I.

• For DD expressions:

1. We scan the corpus looking for the wordforms *des*, *du*, *de_la* and *de_l'*. Due to French morphology, *de* is sometimes contracted with the articles *les* (determinate plural) and *le* (determinate singular masculine). Contractions are mandatory for both partitive and preposition+determiner uses. Therefore, we systematically separate these pairs into two tokens.<fnref n="10" />

2. If a sequence was tagged as a determiner (D), we split the tokens and tag *de* as B and the determiner as I.

3. Contractions (*des*, *du*) tagged as P+D (preposition+determiner) were split in two tokens, both tagged as O.

• All other tokens are tagged as O, including all other categories of MWEs.

For the newly created tokens, we assign individual lemmas and POS tags. The
word *de* is systematically tagged as P (preposition), not distinguishing partitives
from prepositions at the POS level. The input to the CRF is a file containing one
word per line, BIO tags as targets, and FEATURENAME=VALUE pairs including *n*-grams of wordforms, lemmas and POS tags, as described in §3.1.

In the case of nominal MWEs, we applied the same procedure as for AQ pairs
to the MWEs matching certain sequences of POS tags<fnref n="11" />. We accept that tokens
can be separated by punctuation marks, as in the proper noun *Bouches-du-Rhône*.

<footnote n="10">¹⁰An alternative to this preprocessing would be to keep contractions untokenised, and to assign a single B tag to those representing determiners. However, this would actually move the task of MWE identification to the POS tagger, which would need to choose whether the token is a determiner or a contracted preposition before MWE identification.</footnote>

<footnote n="11">¹¹The exact regular-expression pattern is: `(A.N) | (N.(PONCT.)?(A | (P+D.(PONCT.)?N) | (P.(PONCT.)?(D.)?(PONCT.)?N) | N)+)`.</footnote>
When the MWE starts with a noun (N), it can be followed by one or more adjectives (A), nouns (N), or nouns preceded by prepositions (P) optionally including determiners (D) between the preposition and the noun. The matched nominal MWEs include combinations composed of:

* adjective noun: *premier ministre* 'prime minister';

* noun adjective: *corps médical* 'medical community';

* noun-noun: *maison mère* 'parent company';

* noun preposition noun: *motion de censure* 'motion of censure';

* noun preposition determiner noun: *impôt sur le revenu* 'income tax';

* noun preposition+determiner noun: *ironie du sort* 'twist of fate'.

## 4.2 MORPH dataset

We used the MORPH dataset introduced by Nasr et al. (2015) as test and development corpora for ambiguous AQ and DD expressions. It contains a set of 1,269 example sentences, each containing one of 7 ambiguous AQ constructions and 4 ambiguous DD constructions. To build this corpus, around 100 sentences for each of the 11 target constructions were extracted from the frWaC corpus and manually annotated as to whether they contain a multiword function word (MORPH) or accidental cooccurrence (OTHER). We have preprocessed the raw sentences as follows:

1. We have automatically POS tagged and lemmatized all sentences using an off-the-shelf POS tagger and lemmatizer independently trained on the FTB.<fnref n="12" /> This information is given to the CRF as part of its features.

2. We have located the target construction in the sentence and added BIO tags according to the annotation provided: target pairs annotated as MORPH were tagged B + I, target pairs annotated as OTHER were tagged O.

3. For each target construction, we have taken the first 25 sentences as development corpus (dev, 275 sentences) and the remaining sentences for testing (test, 994 sentences).

<footnote n="12">¹²http://macaon.lif.univ-mrs.fr/</footnote>
4. We created four targeted datasets: `DEV<sub>AQ</sub>`, `DEV<sub>DD</sub>`, `FULL<sub>AQ</sub>` and `FULL<sub>DD</sub>`, where the different construction classes are separated, in order to perform feature selection.

Table 1 summarises the corpora covered by our experiments in terms of number of tokens and MWEs. We trained all systems on the training portion of the FTB with different tokenisation choices, depending on the target MWE.<fnref n="13" /> The density of AQ and DD being too low in FTB-dev and FTB-test, we tune and evaluate our model for AQ and DD constructions on the MORPH dataset. For nominal and general MWEs, however, we use the FTB-dev and FTB-test portions.

<table><caption>Table 1: Number of tokens and MWEs in each corpus of our experiments.</caption><thead><tr><th>Corpus</th><th>Portion</th><th>Target MWEs</th><th>#tokens</th><th>#MWEs</th></tr></thead><tbody><tr><td>FTB</td><td>train</td><td>AQ</td><td>285,909</td><td>216</td></tr><tr><td>FTB</td><td>train</td><td>DD</td><td>285,909</td><td>1,356</td></tr><tr><td>FTB</td><td>train</td><td>Nominal</td><td>443,115</td><td>6,413</td></tr><tr><td>FTB</td><td>train</td><td>General</td><td>443,115</td><td>23,522</td></tr><tr><td>FTB</td><td>dev</td><td>Nominal</td><td>38,820</td><td>686</td></tr><tr><td>FTB</td><td>dev</td><td>General</td><td>38,820</td><td>2,117</td></tr><tr><td>FTB</td><td>test</td><td>Nominal</td><td>75,217</td><td>1,019</td></tr><tr><td>FTB</td><td>test</td><td>General</td><td>75,217</td><td>4,041</td></tr><tr><td>MORPH</td><td>FULL<sub>AQ</sub></td><td>AQ</td><td>11,839</td><td>730</td></tr><tr><td>MORPH</td><td>FULL<sub>DD</sub></td><td>DD</td><td>8,319</td><td>539</td></tr></tbody></table>

## 4.3 External lexicons

The verbal valence dictionary DicoValence specifies the allowed types of complements per verb sense in French. For each verb, we extract two binary flags:

* QUEV: one of the senses of the verb has one object that can be introduced by *que*.<fnref n="14" />

<footnote n="13">¹³FTB-train for AQ/DD and for nominal/general MWEs is the same corpus, but the number of tokens differs because all MWEs other than AQ and DD were represented using words-with-spaces in FTB-train for AQ/DD.</footnote>

<footnote n="14">¹⁴In DicoValence, an object P₁, P₂ or P₃ licenses a complementizer `QPIND`</footnote>
* DEV: one of the senses of the verb has a locative, temporal or prepositional paradigm that can be introduced by de.<fnref n="15" />

We also use a lexicon containing nominal MWEs that were automatically extracted from the frWaC. They were obtained with the help of the mwetoolkit by first extracting all lemma sequences that match the nominal MWE pattern described above. Then, for each sequence, we calculate its number of occurrences as well as the number of occurrences of its member words, which are then used to calculate the association measures listed in §3.1.

When integrating this lexicon in the CRF as features, special treatment was required for overlapping expressions. If a given token belonged to more than one overlapping MWE, we considered the maximum value of the association measures. Moreover, since CRFs cannot deal with real-valued features, we have quantized each association score through a uniform distribution that assigned an equal number of expressions to each bin.

## 4.4 Evaluation measures

For general and nominal MWEs, we analyse the performance on the FTB reported by the evaluation script of PARSEME shared task (see Savary et al. 2018 [this volume]).<fnref n="16" /> The script provides us with two different scores: one based on MWEs, and one based on MWE tokens. The MWE-based measure requires that all tokens in the MWE be predicted by the system, whereas the token-based measure is calculated based on each token individually, so that partially correct predictions are taken into account. Each variant (MWE-based and token-based) is reported in terms of precision, recall and F-measure. In this work, we will particularly focus on the F-measure.

For AQ and DD combinations, we evaluated on the MORPH dataset. We report precision (*P*), recall (*R*) and F-measure (*F*₁) of MWE tags. In other words, instead of calculating micro-averaged scores over all BIO tags, we only look at the proportion of correctly guessed B tags. Since all of our target expressions are composed of exactly 2 contiguous words, we can use this simplified score because all B tags are necessarily followed by exactly one I tag. As a consequence, the measured precision, recall and F-measure scores on B and I tags are identical.

<footnote n="15">¹⁵In DicoValence, the paradigm is PDL, PT or PP.</footnote>

<footnote n="16">¹⁶http://multiword.sf.net/sharedtask2017</footnote>
# 5 Results

We present our results for different categories of MWEs, performing feature selection on the *dev* datasets. §5.1 presents an evaluation of our approach on ambiguous AQ and DD expressions. §5.2 evaluates the broader category of nominal MWEs. §5.3 then extends the latter results to an evaluation of all MWEs. §5.4 compares the best results we obtained against the state of the art. Finally, §5.5 presents the results of a detailed analysis focusing on variable and unseen MWEs.

## 5.1 Experiments on AQ and DD expressions

Our first evaluation was performed on the *dev* part of the MORPH dataset. We consider a subset of around 1/4 sentences containing AQ constructions (DEV$_{AQ}$, 175 sentences) and DD constructions (DEV$_{DD}$, 100 sentences). We evaluate the results under different levels of feature selection, regarding both coarse groups and individual features.

In these experiments, the CRF is trained to predict BIO labels on the training part of the FTB, where only the target AQ and DD constructions have been annotated as MWEs, as described in §4.1. Feature selection is performed on development set of the MORPH dataset, in which each sentence contains exactly one occurrence to disambiguate (MWE or accidental co-occurrence).

### 5.1.1 First feature selection: coarse

As shown in the first row of Table 2, when we include all features described in §3 (ALL), we obtain an $F_1$ score of 75.47 for AQ and 69.70 for DD constructions. The following rows of the table show the results of a first ablation study, conducted to identify coarse groups of features that are not discriminant and may hurt performance.

When we ignore orthographic features (ALL - ORTH), all scores increase for DEV$_{AQ}$ and DEV$_{DD}$, suggesting that MWE occurrences are not correlated with orthographic characteristics. $F_1$ also increases when we remove all surface-level wordform features, including single words and *n*-grams (represented by W). We hypothesize that lemmas and POS are more adequate, as they can reduce sparsity by conflating variations of the same lexeme, while wordforms only seem to introduce noise.

We then evaluate the removal of lexicon features (ALL - LF). At a first intuition, one would say that this information is important to our system, as it allows assigning O tags to conjunctions and prepositions that introduce verbal
<table><caption>Table 2: Ablation study results on the dev portion of the MORPH dataset focusing on AQ and DD expressions - impact of the removal of coarse-grained feature sets.</caption><thead><tr><th rowspan="2">Feature set</th><th colspan="3">DEV<sub>AQ</sub></th><th colspan="3">DEV<sub>DD</sub></th></tr><tr><th>P</th><th>R</th><th>F<sub>1</sub></th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>ALL</td><td>89.55</td><td>65.22</td><td>75.47</td><td>92.00</td><td>56.10</td><td>69.70</td></tr><tr><td>ALL - ORTH</td><td>90.28</td><td>70.65</td><td>79.27</td><td>95.83</td><td>56.10</td><td>70.77</td></tr><tr><td>ALL - W</td><td>90.79</td><td>75.00</td><td>82.14</td><td>87.10</td><td>65.85</td><td>75.00</td></tr><tr><td>ALL - LF</td><td>91.18</td><td>67.39</td><td>77.50</td><td>88.89</td><td>58.54</td><td>70.59</td></tr><tr><td>ALL - t<sub>±2</sub></td><td>87.67</td><td>69.57</td><td>77.58</td><td>88.00</td><td>53.66</td><td>66.67</td></tr><tr><td>ALL - T<sub>i-1</sub>T<sub>i</sub>T<sub>i+1</sub></td><td>87.84</td><td>70.65</td><td>78.31</td><td>91.67</td><td>53.66</td><td>67.69</td></tr><tr><td>ALL - T<sub>i-1</sub>T<sub>i</sub></td><td><strong>93.55</strong></td><td>63.04</td><td>75.32</td><td>95.83</td><td>56.10</td><td>70.77</td></tr><tr><td>ALL - T<sub>i-1</sub>T<sub>i</sub> - T<sub>i-1</sub>T<sub>i</sub>T<sub>i+1</sub></td><td>88.57</td><td>67.39</td><td>76.54</td><td><strong>96.00</strong></td><td>58.54</td><td>72.73</td></tr><tr><td>ALL - ORTH - W</td><td>90.24</td><td><strong>80.43</strong></td><td><strong>85.06</strong></td><td>87.10</td><td>65.85</td><td>75.00</td></tr><tr><td>ALL - ORTH - W - t<sub>±2</sub> (REF<sub>1</sub>)</td><td>89.74</td><td>76.09</td><td>82.35</td><td>85.29</td><td><strong>70.73</strong></td><td><strong>77.33</strong></td></tr></tbody></table>

complements. Surprisingly, though, the system performs better without them. We presume that this is a consequence of the sparsity of these features: since there are many features overall in the system, the CRF will naturally forgo LF features when they are present, rendering them superfluous to the system. These features will be analyzed individually later (see Table 4).

One might expect that single tokens located 2 words apart from the target token do not provide much useful information, so we evaluate the removal of the corresponding features (ALL - t<sub>±2</sub>). While this intuition may be true for DEV<sub>AQ</sub>, it does not hold for DEV<sub>DD</sub>. Next, we try to remove all trigrams, and then all trigam & bigram features at once. When we remove trigrams, F<sub>1</sub> decreases by 2.01 absolute points in DEV<sub>DD</sub> and increases by 2.84 absolute points in DEV<sub>AQ</sub>. Bigrams are somehow included in trigrams, and their removal has little impact on the tagger's performance. When we remove bigram and trigram features altogether, scores are slightly better, even though a large amount of information is ignored. Since these results are inconclusive, we perform a more fine-grained selection considering specific n-grams in §5.1.2.

Finally, we try to remove several groups of features at the same time. When we remove both orthographic and wordform features, F<sub>1</sub> increases to 85.06 for DEV<sub>AQ</sub> and 75.00 for DEV<sub>DD</sub>. When we also remove tokens located far away from the current one, performance increases for DEV<sub>DD</sub>, but not for DEV<sub>AQ</sub>. Un-
reported experiments have shown, however, that further feature selection also yields better results for $DEV_{AQ}$ when we ignore $t_{\pm 2}$ features. Therefore, our reference ($REF_1$) for the fine-grained feature selections experiments will be this set of features, corresponding to the last row of Table 2.

### 5.1.2 Second feature selection: fine

Table 3 presents the results from fine-grained feature selection. In the first row of the table, we replicate the reference ($REF_1$) feature set defined above. In the second row, we try to remove the lexicon features (LF) once again. When they were removed in previous experiments, shown in Table 2, we had a gain in performance, suggesting that these features were superfluous. When we remove them from $REF_1$, however, the precision and recall observed for $DEV_{DD}$ decrease by about 10 points. That is, the removal of LF yields a performance drop with respect to a relatively good model ($REF_1$), suggesting that these features are valuable after all. We hypothesise that LF can be better taken into account now that there are less noisy features overall in the whole system.

<table><caption>Table 3: Ablation study results on the dev portion of the MORPH dataset focusing on AQ and DD expressions - impact of the removal of fine-grained feature sets.</caption><thead><tr><th rowspan="2">Feature set</th><th colspan="3">DEV<sub>AQ</sub></th><th colspan="3">DEV<sub>DD</sub></th></tr><tr><th>P</th><th>R</th><th>F<sub>1</sub></th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>REF<sub>1</sub></td><td>89.74</td><td>76.09</td><td>82.35</td><td><strong>85.29</strong></td><td>70.73</td><td>77.33</td></tr><tr><td>REF<sub>1</sub> - LF</td><td>90.00</td><td>78.26</td><td>83.72</td><td>75.76</td><td>60.98</td><td>67.57</td></tr><tr><td>REF<sub>1</sub> - T<sub>-1</sub>T<sub>0</sub></td><td><strong>90.54</strong></td><td>72.83</td><td>80.72</td><td><strong>85.29</strong></td><td>70.73</td><td>77.33</td></tr><tr><td>REF<sub>1</sub> - T<sub>0</sub>T<sub>+1</sub></td><td>89.87</td><td>77.17</td><td>83.04</td><td>84.85</td><td>68.29</td><td>75.68</td></tr><tr><td>REF<sub>1</sub> - T<sub>0</sub>T<sub>+1</sub>T<sub>+2</sub> (BEST<sub>1</sub>)</td><td>87.36</td><td><strong>82.61</strong></td><td><strong>84.92</strong></td><td>83.78</td><td><strong>75.61</strong></td><td><strong>79.49</strong></td></tr></tbody></table>

The last three rows of the table presents the results from attempts at removing individual *n*-gram features that we expected to be redundant or not highly informative. First, we consider the removal of two types of bigram features independently (towards the left and towards the right of the target word). We remove their wordforms, POS and lemmas. The results suggest that bigrams can be mildly useful, as their removal causes the most scores to drop.

In the last row of the table, we present the results from removing all trigram features of the form $T_0T_{+1}T_{+2}$. As a result, we can see that performance increases for both datasets. While trigram features could be potentially useful to recognise
longer expressions, we assume that the number of all possible trigrams is actually too large, making the feature values too sparse. In other words, a much larger annotated corpus would be required for trigram features to be effective. This is the best configuration obtained on the development datasets, and we will refer to it as BEST₁ in the next experiments.

Our last feature selection experiments consider the influence of lexicon features (LF) individually, as shown in Table 4. We observe that DEV is an important feature, because when we remove it, F₁ decreases by almost 7 absolute points on the DEVDD set. The feature QUEV, however, seems less important, and its absence only slightly decreases the F₁ score on the DEVAQ set. This is in line with what was observed by Nasr et al. (2015) for the whole dataset. In sum, these features seem to help, but we would expect the system to benefit more from them with a more sophisticated representation.

<table><caption>Table 4: Ablation study results on the dev portion of the MORPH dataset focusing on AQ and DD expressions - impact of the removal of lexicon features (LF).</caption><thead><tr><th>Dataset</th><th>Feature set</th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td rowspan="2">DEV<sub>AQ</sub></td><td>BEST<sub>1</sub></td><td>87.36</td><td>82.61</td><td>84.92</td></tr><tr><td>BEST<sub>1</sub>-QUEV</td><td>91.25</td><td>79.35</td><td>84.88</td></tr><tr><td rowspan="2">DEV<sub>DD</sub></td><td>BEST<sub>1</sub></td><td>83.78</td><td>75.61</td><td>79.49</td></tr><tr><td>BEST<sub>1</sub>-DEV</td><td>77.78</td><td>68.29</td><td>72.73</td></tr></tbody></table>

The results obtained in this section focus on a limited number of very frequent expressions. Since our evaluation focuses on a small sample of 11 such MWEs only, it would be tempting to train one CRF model per target expression. However, there are a few more expressions with the same characteristics in French, and many of them share similar syntactic behaviour (e.g. conjunctions formed by an adverb and a relative conjunction). An approach with a dedicated model per expression would miss such regular syntactic behaviour (e.g. the fact that the surrounding POS are similar).

The experiments reported up to here show how it is possible to identify highly ambiguous (and frequent) expressions with a CRF, but they are hard to generalise to other MWE categories. Therefore, in the next sections, we evaluate our model on broader MWE categories such as nominal MWEs and general continuous MWEs (as defined in the FTB).
## 5.2 Experiments on nominal MWEs

We now focus on the identification of nominal MWEs in the FTB. As above, we separate our experiments in coarse-grained and fine-grained feature selection. In these experiments, the CRF was trained on the training part of the FTB where only nominal MWEs were tagged as B-I and all other words and MWEs were tagged as O. The feature selection experiments are performed on the development set of the FTB, also transformed in the same way. For the comparison with the state of the art, we report results for the test portion of FTB.

### 5.2.1 First feature selection: coarse

Table 5 presents the results obtained on FTB for different levels of feature selection. In the first row (ALL), we present the evaluation of all the features described in §3, except DEV and QUEV (only relevant to the previous experiments). We obtain a baseline with MWE-based $F_1$ score of 71.57%, and token-based score $F_1$ score of 73.85%.

<table><caption>Table 5: Ablation study results on FTB-dev focusing on nominal MWEs - impact of the removal of coarse-grained feature sets.</caption><thead><tr><th rowspan="2">Feature set</th><th colspan="3">MWE-based</th><th colspan="3">Token-based</th></tr><tr><th>P</th><th>R</th><th>F<sub>1</sub></th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>ALL</td><td>80.86</td><td>64.19</td><td>71.57</td><td>81.23</td><td>67.70</td><td>73.85</td></tr><tr><td>ALL - ORTH</td><td>81.85</td><td>64.78</td><td>72.32</td><td>82.16</td><td>68.02</td><td>74.43</td></tr><tr><td>ALL - W</td><td>80.41</td><td>64.78</td><td>71.75</td><td>80.95</td><td>68.44</td><td>74.17</td></tr><tr><td>ALL - AM</td><td>81.37</td><td>61.72</td><td>70.19</td><td>81.48</td><td>65.16</td><td>72.41</td></tr><tr><td>ALL - t<sub>±2</sub></td><td>81.49</td><td>65.84</td><td>72.83</td><td>81.80</td><td>69.50</td><td>75.15</td></tr><tr><td>ALL - T<sub>+2</sub></td><td>80.96</td><td>65.51</td><td>72.48</td><td>81.18</td><td>69.08</td><td>74.64</td></tr><tr><td>ALL - T<sub>i-1</sub>T<sub>i</sub></td><td>80.41</td><td>64.31</td><td>71.47</td><td>80.99</td><td>67.84</td><td>73.83</td></tr><tr><td>ALL - T<sub>i-1</sub>T<sub>i</sub>T<sub>i+1</sub> (REF<sub>2</sub>)</td><td>81.61</td><td>65.84</td><td>72.88</td><td>82.05</td><td>69.40</td><td>75.20</td></tr><tr><td>ALL - T<sub>i-1</sub>T<sub>i</sub>T<sub>i+1</sub> - AM</td><td>81.69</td><td>63.60</td><td>71.52</td><td>82.09</td><td>67.28</td><td>73.95</td></tr><tr><td>ALL - ORTH - W - t<sub>±2</sub> - T<sub>i-1</sub>T<sub>i</sub>T<sub>i+1</sub></td><td>79.59</td><td>63.37</td><td>70.56</td><td>81.00</td><td>67.88</td><td>73.86</td></tr><tr><td>ALL - ORTH - T<sub>i-1</sub>T<sub>i</sub>T<sub>i+1</sub></td><td>82.51</td><td>65.05</td><td>72.73</td><td>82.74</td><td>67.93</td><td>74.61</td></tr></tbody></table>

We consider the removal of the same groups of features that we removed on the AQ and DD experiments. We evaluate the independent removal of orthographic features, wordforms, association measures, $t_{±2}$, $t_{i+2}$, bigrams and trigrams. We notice that all of these columns have better results than ALL, except
for the column where we removed the bigrams and the one in which we removed association measures. In particular, we notice that the absence of the AMs significantly hurts recall, which in turn has an impact on the F₁ score (–1.38% for the MWE-based measure and –1.41% for the token-based measure). This is the first clue that indicates the importance of these features.

We then evaluate the removal of different groups of features at the same time. We begin by deleting all of the previous groups, except for AMs and bigrams, which seemed to provide useful information above. Nevertheless, we did not obtain better results. We then tried to remove only the trigrams and the orthographic features. Results were slightly higher than ALL, but still remain worse than the results with only the trigrams removed. Finally, we decided to verify if the AM features are still relevant to obtain this performance. This was confirmed, as without the AM, the MWE-based F₁ score decreased by 1.36%, and the token-based F₁ score decreased by 1.25%. Overall, the highest results were obtained by removing only trigrams from ALL, and so we take this feature set as our new reference (REF₂).

### 5.2.2 Second feature selection: fine

Experiments above have shown that association measures (AM) are a vital component of our system. We proceed now to evaluate the importance of individual association measures towards the identification of nominal MWEs. The results are shown in Table 6. We consider the impact of the different AMs in two baseline configurations: all features (ALL), and the features of the reference only (REF₂). We then remove individual measures and evaluate the new feature set on FTB-dev.

We consider the removal of multiple combinations of features. In most cases, we notice a slight improvement in the results against ALL, but not when compared to the reference group. The removal of the DICE measure did improve the results in both cases, ALL and REF₂. Therefore, this configuration was chosen as the BEST₂ set of features. We then evaluated these BEST₂ features on the FTB-test dataset, obtaining a MWE-based F₁ score of 71.38%, and a token-based score of 73.43%. As a sanity check, we have also evaluated the system without AMs on FTB-test (ALL - AM). The BEST₂ system is significantly different from both ALL and ALL - AM on the test set. Moreover, the large margin between ALL - AM and the two other systems indicates that association measures do provide useful features for this task.
<table><caption>Table 6: Ablation study results on FTB-dev focusing on nominal MWEs - impact of the removal of fine-grained feature sets.</caption><thead><tr><th rowspan="2">Feature set</th><th colspan="3">MWE-based</th><th colspan="3">Token-based</th></tr><tr><th>P</th><th>R</th><th>F<sub>1</sub></th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>ALL</td><td>80.86</td><td>64.19</td><td>71.57</td><td>81.23</td><td>67.70</td><td>73.85</td></tr><tr><td>ALL - DICE</td><td>81.07</td><td>64.55</td><td>71.87</td><td>81.39</td><td>68.02</td><td>74.11</td></tr><tr><td>ALL - T-MEAS</td><td>81.07</td><td>64.55</td><td>71.87</td><td>81.40</td><td>68.07</td><td>74.14</td></tr><tr><td>ALL - PMI</td><td>81.26</td><td>63.84</td><td>71.50</td><td>81.51</td><td>67.33</td><td>73.74</td></tr><tr><td>ALL - MLE - LL</td><td>81.13</td><td>64.31</td><td>71.75</td><td>81.40</td><td>67.65</td><td>73.89</td></tr><tr><td>ALL - T-MEAS - DICE</td><td>80.76</td><td>64.78</td><td>71.90</td><td>81.23</td><td>68.30</td><td>74.20</td></tr><tr><td>ALL - MLE - LL - T-MEAS - DICE</td><td>81.72</td><td>63.72</td><td>71.61</td><td>81.58</td><td>67.05</td><td>73.61</td></tr><tr><td>REF<sub>2</sub></td><td>81.61</td><td>65.84</td><td>72.88</td><td>82.05</td><td>69.40</td><td>75.20</td></tr><tr><td>REF<sub>2</sub> - DICE (BEST<sub>2</sub>)</td><td><strong>81.84</strong></td><td>65.84</td><td>72.98</td><td>82.33</td><td><strong>69.45</strong></td><td><strong>75.34</strong></td></tr><tr><td>REF<sub>2</sub> - T-MEAS</td><td>81.61</td><td>65.84</td><td>72.88</td><td>82.01</td><td>69.22</td><td>75.08</td></tr><tr><td>REF<sub>2</sub> - PMI</td><td>81.80</td><td>65.14</td><td>72.52</td><td><strong>82.36</strong></td><td>68.71</td><td>74.92</td></tr><tr><td>REF<sub>2</sub> - MLE - LL</td><td>81.67</td><td>65.61</td><td>72.76</td><td>82.03</td><td>69.08</td><td>75.00</td></tr><tr><td>REF<sub>2</sub> - T-MEAS - DICE</td><td>81.75</td><td><strong>65.96</strong></td><td><strong>73.01</strong></td><td>82.18</td><td>69.36</td><td>75.23</td></tr><tr><td>REF<sub>2</sub> - MLE - LL - T-MEAS - DICE</td><td>81.41</td><td>65.49</td><td>72.58</td><td>81.51</td><td>68.94</td><td>74.70</td></tr><tr><td>ALL (on FTB-test)</td><td>77.06</td><td>65.66</td><td>70.90</td><td>79.10</td><td>68.23</td><td>73.27</td></tr><tr><td>ALL - AM (on FTB-test)</td><td>76.96</td><td>61.81</td><td>68.56</td><td>78.94</td><td>64.91</td><td>71.24</td></tr><tr><td>BEST<sub>2</sub> (on FTB-test)</td><td>76.00</td><td>67.28</td><td>71.38</td><td>77.74</td><td>69.58</td><td>73.43</td></tr></tbody></table>

## 5.3 Experiments on general MWEs

We extend the experiments above to evaluate the feature sets against the whole FTB corpus, keeping all annotated MWEs in the training, development and test parts of the FTB. We would like to verify if our system is able to take into account the different MWE categories at the same time. This time, we only present coarse-grained feature selection (Table 7), since unreported fine-grained feature selection resulted in similar findings as in experiments focusing on nominal MWEs.

The first row in the table (ALL) presents the evaluation of all features described in §3. The prediction of general MWEs with ALL features yields a MWE-based $F_1$ score of 78.89% and a token-based $F_1$ score of 81.61%. We then consider what happens when one removes the same groups of features as in the previous sections. This time the results are quite different: all of these tests have worse results than ALL, except when we remove $t_{+2}$ features. In some unreported experiments,
<table><caption>Table 7: Ablation study results on FTB-dev focusing on general MWEs - impact of the removal of feature sets.</caption><thead><tr><th rowspan="2">Feature set</th><th colspan="3">MWE-based</th><th colspan="3">Token-based</th></tr><tr><th>P</th><th>R</th><th>F<sub>1</sub></th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>ALL</td><td><strong>85.60</strong></td><td>73.16</td><td>78.89</td><td><strong>87.32</strong></td><td>76.60</td><td>81.61</td></tr><tr><td>ALL - ORTH</td><td>85.09</td><td>72.97</td><td>78.57</td><td>86.97</td><td>76.56</td><td>81.44</td></tr><tr><td>ALL - W</td><td>83.96</td><td>72.59</td><td>77.86</td><td>86.13</td><td>76.37</td><td>80.96</td></tr><tr><td>ALL - AM</td><td>85.11</td><td>72.78</td><td>78.46</td><td>86.89</td><td>76.33</td><td>81.27</td></tr><tr><td>ALL - t<sub>±2</sub></td><td>84.03</td><td>72.45</td><td>77.81</td><td>86.57</td><td>76.94</td><td>81.47</td></tr><tr><td>ALL - t<sub>+2</sub></td><td>85.50</td><td>73.68</td><td>79.15</td><td>87.19</td><td>77.21</td><td>81.90</td></tr><tr><td>ALL - T<sub>i-1</sub>T<sub>i</sub></td><td>84.36</td><td>71.75</td><td>77.54</td><td>86.61</td><td>75.47</td><td>80.66</td></tr><tr><td>ALL - T<sub>i-1</sub>T<sub>i</sub>T<sub>i+1</sub></td><td>84.78</td><td>73.07</td><td>78.49</td><td>86.39</td><td>76.31</td><td>81.04</td></tr><tr><td>ALL - T<sub>+2</sub> - ORTH (REF<sub>3</sub>)</td><td>85.52</td><td><strong>73.82</strong></td><td><strong>79.24</strong></td><td>87.30</td><td><strong>77.35</strong></td><td><strong>82.03</strong></td></tr><tr><td>REF<sub>3</sub> - AM</td><td>85.37</td><td>72.69</td><td>78.52</td><td>87.08</td><td>76.33</td><td>81.35</td></tr><tr><td>REF<sub>3</sub> - T-MEAS (BEST<sub>3</sub>)</td><td>85.62</td><td><strong>73.87</strong></td><td><strong>79.31</strong></td><td>87.40</td><td><strong>77.43</strong></td><td><strong>82.11</strong></td></tr><tr><td>ALL (on FTB-test)</td><td>83.80</td><td>74.51</td><td>78.88</td><td>86.58</td><td>78.23</td><td>82.19</td></tr><tr><td>ALL - AM (on FTB-test)</td><td>84.19</td><td>73.52</td><td>78.49</td><td>86.90</td><td>77.30</td><td>81.82</td></tr><tr><td>BEST<sub>3</sub> (on FTB-test)</td><td>84.03</td><td>74.71</td><td>79.10</td><td>86.72</td><td>78.47</td><td>82.39</td></tr></tbody></table>

we have tried to remove other groups of features along with $t_{+2}$. We found that removing orthographic features along with $t_{+2}$ increased the results more than only removing $t_{+2}$ features. This group of features will be our new reference from now on (REF<sub>3</sub>). Once again, we tried to remove AMs from the reference to verify their impact. Here again, we notice that the removal of these features decreases the overall performance scores, even if their impact is weaker than it was in the case of nominal MWEs. Unreported experiments have shown that we obtain better results when we ignore the T-MEAS feature (BEST<sub>3</sub>).

Then, we applied the feature group BEST<sub>3</sub> on the FTB-test dataset, and we obtained a MWE-based $F_1$ score of 79.10%, and a token-based score of 82.39%. For the feature selection experiments on the test part of the FTB (both nominal and general MWEs), we calculated the p-value of the difference between the configuration called BEST and the one called ALL, using approximate randomisation with stratified shuffling. None of the observed differences was considered statistically significant with $\alpha = 0.05$.
## 5.4 Comparison with state of the art

We now compare the highest-scoring reference results with the state of the art. We begin by evaluating the identification of *DD* and *AQ* constructions, and then proceed to evaluate more generally the quality of our reference system for general MWE identification. The comparisons presented here focus on MWE identification only, and our model takes gold POS and lemma information as input (except on the MORPH dataset). On the other hand, some of the works mentioned in our comparisons also predict POS and/or syntactic structure, which makes the task considerably harder. Therefore, results presented here should be taken as an indication of our position within the current landscape of MWE identification, rather than as a demonstration of our model's superiority.

### 5.4.1 AQ and DD constructions

We report the performance of MWE identification on the full MORPH dataset, split in two parts: sentences containing AQ constructions (FULL\_AQ) and sentences containing DD constructions (FULL\_DD). The use of the full datasets is not ideal, given that we performed feature selection on part of these sentences, but it allows a direct comparison with related work.

Table 8 presents a comparison between the best system score obtained after feature selection (BEST₁) and the results reported by Nasr et al. (2015). We include two versions of the latter system, since they also distinguish their results based on the presence of lexicon features (LF) coming from DicoValence.

<table><caption>Table 8: Comparison with baseline and state of the art of AQ and DD identification on the full MORPH dataset.</caption><thead><tr><th rowspan="2">System</th><th colspan="3">FULL<sub>AQ</sub></th><th colspan="3">FULL<sub>DD</sub></th></tr><tr><th>P</th><th>R</th><th>F<sub>1</sub></th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>Baseline</td><td>56.08</td><td>100.00</td><td>71.86</td><td>34.55</td><td>100.00</td><td>51.35</td></tr><tr><td>Nasr et al. (2015)-LF</td><td>88.71</td><td>82.03</td><td>85.24</td><td>77.00</td><td>73.09</td><td>75.00</td></tr><tr><td>Nasr et al. (2015)+LF</td><td>91.57</td><td>81.79</td><td>86.41</td><td>86.70</td><td>82.74</td><td>84.67</td></tr><tr><td>BEST<sub>1</sub></td><td>91.08</td><td>78.31</td><td>84.21</td><td>79.14</td><td>74.37</td><td>76.68</td></tr></tbody></table>
We additionally report results for a simple baseline:

1. We extract a list of all pairs of contiguous AQ and DD from the FTB-train.

2. We calculate the proportion of cases in which they were annotated as MWEs (B-I tags) with respect to all of their occurrences.

3. We keep in the list only those constructions which were annotated as MWE at least 50% of the time.

4. We systematically annotate these constructions as MWEs (B-I) in all sentences of the MORPH dataset, regardless of their context.

Table 8 shows that this baseline reaches 100% recall, covering all target constructions, but precision is very low, as contextual information is not taken into account during identification. Our BEST₁ system can identify the target ambiguous MWEs much better than the baseline for both FULL AQ and FULL DD.

For some constructions, we do obtain results that are close to those obtained by the parsers (see Table 9 for more details). For FULL AQ, our BEST₁ system obtains an F₁ score that is 1.2 absolute points lower than the best parser. For FULL DD, however, our best system, which includes lexicon features (LF), is comparable with a parser without lexicon features. When the parser has access to the lexicon, it beats our system by a significant margin of 7.99 points, indicating that the accurate disambiguation of DD constructions indeed requires syntax-based methods rather than sequence taggers. These results contradict our hypothesis that sequence models can deal with continuous constructions with a performance equivalent to parsing-based approaches. While this may be true for non-ambiguous expressions, parsing-based methods are superior for AQ and DD constructions, given that they were trained on a full treebank, have access to more sophisticated models of a sentence's syntax, and handle long-distance relations and grammatical information.

Despite the different results obtained depending on the nature of the target constructions, the results are encouraging, as they prove the feasibility of applying sequence taggers for the identification of highly ambiguous MWEs. Our method has mainly two advantages over parsing-based MWE identification: (a) it is fast and only requires a couple of minutes on a desktop computer to be trained; and (b) it does not require the existence of a treebank annotated with MWEs.

Table 9 shows the detailed scores for each expression in the MORPH dataset. We notice that some expressions seem to be particularly difficult to identify, especially if we look at precision, whereas for others we obtain scores well above 90%. When we compare our results to those reported by Nasr et al. (2015), we can see that they are similar to ours: *ainsi* 'likewise', *alors* 'then' and *bien* 'well'
have $F_1$ higher than 90%, while *autant* ‘as much’ and *tant* ‘while’ have a score lower than 80%. The AQ constructions with *encore* ‘still’ and *maintenant* ‘now’ are the only ones which behave differently: our system is better for *encore* ‘still’, but worse for *maintenant* ‘now’. Likewise, for DD expressions, our system obtains a performance that is close to their system without lexicon features (LF), but considerably worse than their system including LFs for three out of 4 expressions. Both approaches are more efficient in identifying the plural article *de les* ‘of the.PL’ than the partitive constructions.

<table><caption>Table 9: Performance of the BEST₁ configuration broken down by expression, along with the results for the best model of Nasr et al. (2015) (with LF).</caption><thead><tr><th rowspan="2">Expression</th><th colspan="3">BEST<sub>1</sub> system</th><th colspan="3">Nasr et al. (2015)</th></tr><tr><th>P</th><th>R</th><th>F<sub>1</sub></th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>ainsi que</td><td>94.44</td><td>93.15</td><td>93.79</td><td>95.94</td><td>89.87</td><td>92.81</td></tr><tr><td>alors que</td><td>84.00</td><td>97.67</td><td>90.32</td><td>93.81</td><td>93.81</td><td>93.81</td></tr><tr><td>autant que</td><td>93.48</td><td>51.81</td><td>66.67</td><td>86.66</td><td>70.65</td><td>77.84</td></tr><tr><td>bien que</td><td>100.00</td><td>91.43</td><td>95.52</td><td>91.66</td><td>99.18</td><td>90.41</td></tr><tr><td>encore que</td><td>76.19</td><td>94.12</td><td>84.21</td><td>92.85</td><td>65.00</td><td>76.47</td></tr><tr><td>maintenant que</td><td>97.62</td><td>64.06</td><td>77.36</td><td>90.91</td><td>74.62</td><td>81.96</td></tr><tr><td>tant que</td><td>100.00</td><td>60.00</td><td>75.00</td><td>82.35</td><td>70.00</td><td>75.67</td></tr><tr><td>de le</td><td>78.05</td><td>71.11</td><td>74.42</td><td>85.41</td><td>91.11</td><td>88.17</td></tr><tr><td>de la</td><td>67.74</td><td>72.41</td><td>70.00</td><td>81.25</td><td>89.65</td><td>85.24</td></tr><tr><td>de les</td><td>92.41</td><td>71.57</td><td>80.66</td><td>98.70</td><td>76.00</td><td>85.87</td></tr><tr><td>de l'</td><td>61.11</td><td>95.65</td><td>74.58</td><td>64.51</td><td>86.95</td><td>74.07</td></tr></tbody></table>

### 5.4.2 General MWEs

We now compare our system with two baselines and with the system proposed in Le Roux et al. (2014).<fnref n="17" /> *Baseline*₁ consists in identifying as MWE every continuous occurrence of tokens that has been seen as an MWE in the training corpus. For example, the MWE *bien sûr* (lit. *well sure*) ‘of course’ can be seen in the training corpus, and so every occurrence of this expression was predicted as an MWE

<footnote n="17">¹⁷The comparison with Le Roux et al. (2014) is not ideal, since we predict MWEs with the help of gold POS and lemmas, whereas they try to predict both POS and MWEs. However, we could not find a fully comparable evaluation in the literature.</footnote>
for the test corpus. $Baseline_2$ filters the list of MWEs seen in the training corpus, so that only the expressions which had been annotated more than 40% of the time are predicted as MWEs. For example, the expression *d'un côté* (lit. of a side) 'on the one hand' is not predicted as MWE, as it was only annotated in 38% of its occurrences in the training corpus. The baselines were directly inspired by a predictive model applied to the English language in a similar task, where a threshold of 40% was found to yield the best results (Cordeiro et al. 2016). The applied threshold in $Baseline_2$ only eliminates 6.46% of the MWEs from the list, but it contributes to an increase of 20–30 points in precision without impacting the recall.

Our system ($BEST_3$ configuration) is more accurate than the baselines, both in terms of precision and recall. It also has a higher precision than the approach proposed by Le Roux et al. (2014), but the recall is considerably worse (9.48% less than their system). This means that our system misses more expressions, even if its predictions have higher precision. This could be partly explained by the fact that they employed dictionaries, and have access to more expressions that our system has never seen and could not predict. Nonetheless, our results are sufficiently close and represent a decent alternative if high-quality external resources are not available.

<table><caption>Table 10: Comparison with baseline and state of the art of general MWE identification on FTB-test.</caption><thead><tr><th rowspan="2">System</th><th colspan="3">MWE-based</th><th colspan="3">Token-based</th></tr><tr><th>P</th><th>R</th><th>F<sub>1</sub></th><th>P</th><th>R</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>Baseline<sub>1</sub></td><td>52.93</td><td>66.20</td><td>58.82</td><td>62.70</td><td>69.73</td><td>66.03</td></tr><tr><td>Baseline<sub>2</sub></td><td>82.76</td><td>69.36</td><td>75.47</td><td>84.80</td><td>69.62</td><td>76.46</td></tr><tr><td>BEST<sub>3</sub> configuration</td><td>84.03</td><td>74.71</td><td>79.10</td><td>86.72</td><td>78.47</td><td>82.39</td></tr><tr><td>Le Roux et al. (2014)</td><td>80.76</td><td>84.19</td><td>82.44</td><td>—</td><td>—</td><td>—</td></tr></tbody></table>

## 5.5 Analysis of results

The performance of our CRF identification model depends on the characteristics of the identified MWEs and of the training and test corpora. Therefore, we have performed a detailed analysis of its performance focusing on a subset of the test corpus. We focus on two phenomena: variants and unseen MWEs. We define a **variant** as an MWE whose lemmatised form occurs both in the training and in
the test corpus, but whose surface form in the test corpus is different from all of its surface forms in the training corpus. We define an **unseen** MWE as one whose lemmatised form occurs in the test corpus but never (under any surface form) in the training corpus. MWEs which have identical occurrences in the training and test corpora will be referred to as **seen** MWEs.

Both variants and unseen MWEs are harder to identify than seen MWEs. Nonetheless, we hypothesise that our model is able to recognise variants correctly, since its features are based on lemmas. On the other hand, we expect that unseen MWEs cannot be easily predicted given that our system is absed on categorical features and does not have access to much information about an expression that has never been seen in the training corpus, except for its association measures in a large unannotated corpus. To verify these hypotheses, we create sub-corpora of FTB-test, where the density of variants and unseen MWEs is higher than in the full FTB-test corpus. In these experiments, the model is not newly trained, but the BEST₂ and BEST₃ models are applied to different sub-corpora with a high density of variant/unseen MWEs.

The evaluation measures reported in our experiments (§4.4) consider the best bijection between predicted and gold MWEs. Therefore, we cannot simply remove seen MWEs from the test set, since they will be predicted anyway, artificially penalising precision. Therefore, instead of completely removing seen MWEs, we remove sentences that contain only seen MWEs and keep sentences that contain (a) at least one variant MWE or (b) at least one unseen MWE.

<table><caption>Table 11: Results of BEST₂ (nominal MWEs) and BEST₃ (general MWEs) on FTB-test, on sub-corpus containing unseen variants of a seen MWEs, and on sub-corpus containing unseen MWEs. Columns %var and %unseen show the proportion of variants/unseen MWEs in each sub-corpus.</caption><thead><tr><th rowspan="2">Feature set</th><th rowspan="2">%var</th><th rowspan="2">%unseen</th><th colspan="3">MWE-based</th><th colspan="3">Token-based</th></tr><tr><th>P</th><th>R</th><th>F₁</th><th>P</th><th>R</th><th>F₁</th></tr></thead><tbody><tr><td>Nominal full</td><td>5%</td><td>28%</td><td>76.00</td><td>67.28</td><td>71.38</td><td>77.74</td><td>69.58</td><td>73.43</td></tr><tr><td>Nom. variants</td><td>65%</td><td>N/A</td><td>86.42</td><td>63.64</td><td>73.30</td><td>85.84</td><td>66.20</td><td>74.75</td></tr><tr><td>Nom. unseen</td><td>N/A</td><td>72%</td><td>82.01</td><td>42.70</td><td>56.16</td><td>87.78</td><td>46.75</td><td>61.01</td></tr><tr><td>General full</td><td>5%</td><td>23%</td><td>84.03</td><td>74.71</td><td>79.10</td><td>86.72</td><td>78.47</td><td>82.39</td></tr><tr><td>Gen. variants</td><td>32%</td><td>N/A</td><td>88.91</td><td>69.44</td><td>77.98</td><td>92.77</td><td>74.05</td><td>82.36</td></tr><tr><td>Gen. unseen</td><td>N/A</td><td>51%</td><td>86.94</td><td>65.22</td><td>74.25</td><td>90.40</td><td>69.14</td><td>78.35</td></tr></tbody></table>
Table 11 presents the performance of the BEST<sub>2</sub> configuration for nominal MWEs (first row) and BEST<sub>3</sub> configuration for general MWEs (fourth row) on the full FTB-test corpus. For each expression (nominal and general), we also present the results for the sub-corpus containing a higher density of variants and of unseen MWEs. The numbers in columns %var and %unseen indicate the proportion of variant/unseen MWEs in each sub-corpus. Notice that, in the case of general MWEs, these proportions are quite low (32% and 51%), indicating that sentences containing variant and unseen general MWEs often contain seen ones too. When focusing on variants (Nom. variants and Gen. variants sub-corpora), the proportion of unseen MWEs is very small and not relevant (N/A), and vice-versa.

If we focus on variants, we can observe relatively stable results with respect to the full FTB-test corpus. For nominal MWEs, precision increases by 8-10%, whereas recall decreases by about 3% for both MWE-based and token-based measures. Results for general MWEs follow a similar pattern: around 4-6% improvement in precision at the cost of around 4-5% decrease in recall. The precision of general MWE identification in the variants sub-corpus is particularly impressive, reaching 92.77%.

The variants sub-corpora contain less unseen MWEs than the full FTB-test corpus, so the predicted MWEs are more reliable (better precision), showing that our model is robust to morphological variability. On the other hand, the fact that recall drops indicates that it is indeed slightly harder to recognise variants of MWEs than those seen identically in training and test corpora. In short, we infer that variants can be correctly handled and identified by our model, provided that a good lemmatiser is available (results presented here are based on gold lemmas, their substitution by predicted lemmas should be studied in the future).

On the other hand, predicting unseen MWEs is considerably harder. Recall drops drastically by about 23-25% for nominal MWEs and by about 9% for general MWEs, and the improvements in precision do not compensate for this, yielding much lower F-measure values, specially for nominal MWEs where the concentration of unseen MWEs in the sub-corpus is higher (72%). The improvements in precision are probably due to the fact that some seen and variant MWEs are still present in the sub-corpora. AMs could also have some predictive power to identify unseen MWEs, and we intend to verify their contribution for unseen MWE identification in the future. These results show that our model is limited in the identification of unseen MWEs, and can probably only identify some of those that appear in the AM lexicons.
# 6 Conclusions and future work

We have described and evaluated a simple and fast CRF tagger that is able to identify several categories of continuous multiword expressions in French. We have reported feature selection studies and shown that, for AQ constructions and for general MWEs, our results are almost as good as those obtained by parsers, even though we do not rely on syntactic trees. This was not true for DD constructions, though, which seem to require parsing-based methods to be properly analysed. Based on these results, we conclude that, when treebanks are not available, sequence models such as CRFs can obtain reasonably good results in the identification of continuous MWEs. On the other hand, when MWE-annotated treebanks exist, parsing-based models seem to obtain better results, especially for expressions whose high ambiguity requires syntax to be resolved.

An interesting direction of research would be to study the interplay between automatic POS tagging and MWE identification. We recall that our results were obtained with an off-the-shelf POS tagger and lemmatizer. Potentially, performing both tasks jointly could help obtaining more precise results (Constant & Siggogne 2011). Moreover, we could explore CRFs' ability to work with lattices in order to pre-select the most plausible MWE identification (and POS tagging) solutions, and then feed them into a parser which would take the final decision.

Another idea for future work would be an investigation of the features themselves. For example, in this work, we were not fully satisfied with the quality of the representation of lexical features. We would like to investigate the reason why lexical features were not always useful for the task of MWE identification, which could be done by performing an error analysis on the current systems. Another interesting question is whether annotated corpora are at all necessary: could hand-crafted and/or automatically built lexicons be employed to identify MWEs in context in a fully unsupervised way?

While these experiments shed some light on the nature of MWEs in French, the feature selection methodology is highly empirical and cannot be easily adapted to other contexts. Therefore, we would like to experiment different techniques for generic automatic feature selection and classifier tuning (Ekbal & Saha 2012). This could be performed on a small development set, and would ease the adaptation of the tagger to other contexts.

Finally, we would like to experiment with other sequence tagging models such as recurrent neural networks. In theory, such models are very efficient to perform feature selection and can also deal with continuous word representations able to encode semantic information. Moreover, distributed word representations could
be helpful in building cross-lingual MWE identification systems.

## Acknowledgments

This work has been funded by projects PARSEME (Cost Action IC1207), PARSEME-FR (ANR-14-CERA-0001), and AIM-WEST (FAPERGS-INRIA 1706-2551/13-7). We also thank the anonymous reviewers for their valuable suggestions.

## Abbreviations

<table><tr><td>AQ</td><td>adverb+<em>que</em></td></tr><tr><td>AM</td><td>association measure</td></tr><tr><td>BIO</td><td>begin-inside-outside</td></tr><tr><td>CRF</td><td>conditional random field</td></tr></table>

<table><tr><td>DD</td><td><em>de</em>+determiner</td></tr><tr><td>FTB</td><td>French Treebank</td></tr><tr><td>LF</td><td>lexicon feature</td></tr><tr><td>MWE</td><td>multiword expression</td></tr></table>

## References

Abeillé, Anne, Lionel Clément & François Toussenel. 2003. Building a treebank for French. In Anne Abeillé (ed.), *Treebanks: Building and using parsed corpora*, 165–187. Dordrecht, The Netherlands: Kluwer Academic Publishers.

Baroni, Marco & Silvia Bernardini (eds.). 2006. *Wacky! Working papers on the web as corpus*. Bologna, Italy: GEDIT.

Boukobza, Ram & Ari Rappoport. 2009. Multi-word expression identification using sentence surface features. In *Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing*, 468–477. August 6-7, 2009.

Buchholz, Sabine & Erwin Marsi. 2006. CoNLL-x shared task on multilingual dependency parsing. In *Proceedings of the 10th Conference on Computational Natural Language Learning (CoNLL-X ’06)*, 149–164. http://dl.acm.org/citation.cfm?id=1596276.1596305.

Candito, Marie & Matthieu Constant. 2014. Strategies for contiguous multiword expression analysis and dependency parsing. In *Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers)*, 743–753. Association for Computational Linguistics. http://www.aclweb.org/anthology/P14-1070.
Carpuat, Marine & Mona Diab. 2010. Task-based evaluation of multiword expressions: A pilot study in Statistical Machine Translation. In *Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics*, 242–245. Association for Computational Linguistics. http://www.aclweb.org/anthology/N10-1029.

Constant, Matthieu & Joakim Nivre. 2016. A transition-based system for joint lexical and syntactic analysis. In *Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers)*, 161–171. Association for Computational Linguistics. http://www.aclweb.org/anthology/P16-1016.

Constant, Matthieu & Anthony Sigogne. 2011. MWU-aware part-of-speech tagging with a CRF model and lexical resources. In *Proceedings of the ALC Workshop on Multiword Expressions: From Parsing and Generation to the Real World (MWE 2011)*, 49–56. Association for Computational Linguistics. http://www.aclweb.org/anthology/W11-0809.

Cordeiro, Silvio, Carlos Ramisch & Aline Villavicencio. 2016. UFRGS & LIF at SemEval-2016 task 10: Rule-based MWE identification and predominant-super-sense tagging. In *Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)*, 910–917. Association for Computational Linguistics. http://www.aclweb.org/anthology/S16-1140.

Diab, Mona & Pravin Bhutada. 2009. Verb noun construction MWE token classification. In *Proceedings of the Workshop on Multiword Expressions: Identification, Interpretation, Disambiguation and Applications*, 17–22. Association for Computational Linguistics. http://www.aclweb.org/anthology/W/W09/W09-2903.

Ekbal, Asif & Sriparna Saha. 2012. Multiobjective optimization for classifier ensemble and feature selection: An application to named entity recognition. *International Journal on Document Analysis and Recognition (IJDAR)* 15(2). 143–166. DOI:10.1007/s10032-011-0155-7

Fazly, Afsaneh, Paul Cook & Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. *Computational Linguistics* 35(1). 61–103. http://aclweb.org/anthology/J09-1005.

Finlayson, Mark & Nidhi Kulkarni. 2011. Detecting multi-word expressions improves word sense disambiguation. In *Proceedings of the ACL Workshop on Multiword Expressions: From Parsing and Generation to the Real World (MWE '11)*, 20–24. Association for Computational Linguistics. http://www.aclweb.org/anthology/W11-0805.
Green, Spence, Marie-Catherine de Marneffe & Christopher D. Manning. 2013. Parsing models for identifying multiword expressions. *Computational Linguistics* 39(1). 195–227. DOI:10.1162/COLI_a_00139

Lafferty, John D., Andrew McCallum & Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In *Proceedings of the Eighteenth International Conference on Machine Learning* (ICML ’01), 282–289. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. http://dl.acm.org/citation.cfm?id=645530.655813.

Laporte, Éric, Takuya Nakamura & Stavroula Voyatzi. 2008. A French corpus annotated for multiword expressions with adverbial function. In *Proceedings of the 2nd Linguistic Annotation Workshop*, 48–51. https://halshs.archives-ouvertes.fr/halshs-00286541.

Le Roux, Joseph, Antoine Rozenknop & Matthieu Constant. 2014. Syntactic parsing and compound recognition via dual decomposition: Application to French. In *Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers* (COLING 2014), 1875–1885. Dublin, Ireland: Dublin City University & Association for Computational Linguistics. http://www.aclweb.org/anthology/C14-1177.

Nasr, Alexis, Carlos Ramisch, José Deulofeu & André Valli. 2015. Joint dependency parsing and multiword expression tokenization. In *Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, 1116–1126. Association for Computational Linguistics. http://www.aclweb.org/anthology/P15-1108.

Nivre, Joakim, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty & Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the Tenth International Conference on Language Resources and Evaluation* (LREC 2016), 1659–1666. European Language Resources Association (ELRA). 23-28 May, 2016.

Nivre, Joakim & Jens Nilsson. 2004. Multiword units in syntactic parsing. In *Workshop on methodologies and evaluation of multiword units in real-world applications* (MEMURA 2004), 39–46. http://stp.lingfil.uu.se/~nivre/docs/mwu.pdf.
Okazaki, Naoaki. 2007. *CRFsuite: A fast implementation of conditional random fields (CRFs)*. http://www.chokkan.org/software/crfsuite/.

Ramisch, Carlos. 2014. *Multiword expressions acquisition: A generic and open framework* (Theory and Applications of Natural Language Processing XIV). Cham, Switzerland: Springer. 230. http://link.springer.com/book/10.1007/2F978-3-319-09207-2.

Ramshaw, Lance & Mitchell P. Marcus. 1995. Text chunking using transformation-based learning. In *3rd Workshop on Very Large Corpora*, 82–94. http://aclweb.org/anthology/W95-0107.

Riedl, Martin & Chris Biemann. 2016. Impact of MWE resources on multi-word recognition. In *Proceedings of the 12th Workshop on Multiword Expressions* (MWE ’16), 107–111. Association for Computational Linguistics. http://anthology.aclweb.org/W16-1816.

Sag, Ivan A., Timothy Baldwin, Francis Bond, Ann A. Copestake & Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In *Proceedings of the 3rd International Conference on Computational Linguistics and Intelligent Text Processing*, vol. 2276/2010 (CICLing ’02), 1–15. Springer-Verlag.

Savary, Agata, Marie Candito, Verginica Barbu Mititelu, Eduard Bejček, Fabienne Cap, Slavomír Čéplö, Silvio Ricardo Cordeiro, Gülşen Eryiğit, Voula Giouli, Maarten van Gompel, Yaakov HaCohen-Kerner, Jolanta Kovalevskaitė, Simon Krek, Chaya Liebeskind, Johanna Monti, Carla Parra Escartín, Lonneke van der Plas, Behrang QasemiZadeh, Carlos Ramisch, Federico Sangati, Ivelina Stoyanova & Veronika Vincze. 2018. PARSEME multilingual corpus of verbal multiword expressions. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 87–147. Berlin: Language Science Press.
DOI:10.5281/zenodo.1471591

Savary, Agata, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova & Antoine Doucet. 2017. The PARSEME Shared Task on automatic identification of verbal multiword expressions. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 31–47. Association for Computational Linguistics. DOI:10.18653/v1/W17-1704

Schneider, Nathan, Emily Danchik, Chris Dyer & Noah A. Smith. 2014. Discriminative lexical semantic segmentation with gaps: Running the MWE gamut. *Transactions of the Association for Computational Linguistics* 2(1). 193–206. https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/281.
Schneider, Nathan, Spencer Onuffer, Nora Kazour, Nora Emily Danchik, Michael T. Mordowanec, Henrietta Conrad & Smith Noah A. 2014. Comprehensive annotation of multiword expressions in a social web corpus. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the 9th International Conference on Language Resources and Evaluation* (LREC 2014), 455–461. European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2014/pdf/521_Paper.pdf.

Scholivet, Manon & Carlos Ramisch. 2017. Identification of ambiguous multiword expressions using sequence models and lexical resources. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 167–175. Association for Computational Linguistics. http://www.aclweb.org/anthology/W17-1723.

Seddah, Djamé, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richard Farkas, Jennifer Foster, Lakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yanick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska & Eric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013 Shared Task: a cross-framework evaluation of parsing morphologically rich languages. In *Proceedings of the 4th Workshop on Statistical Parsing of Morphologically-Rich Languages*, 146–182. Association for Computational Linguistics. http://www.aclweb.org/anthology/W13-4917.

Shigeto, Yutaro, Ai Azuma, Sorami Hisamoto, Shuhei Kondo, Tomoya Kouse, Keisuke Sakaguchi, Akifumi Yoshimoto, Frances Yung & Yuji Matsumoto. 2013. Construction of English MWE dictionary and its application to POS tagging. In *Proceedings of the 9th Workshop on Multiword Expressions* (MWE ’13), 139–144. Association for Computational Linguistics. http://www.aclweb.org/anthology/W13-1021.

Silberstein, Max, Tamás Váradi & Marko Tadić. 2012. Open source multi-platform NooJ for NLP. In *Proceedings of the 24th International Conference on Computational Linguistics: Demonstration Papers* (COLING-12), 401–408. The Coling 2012 Organizing Committee. http://www.aclweb.org/anthology/C12-3050.

van den Eynde, Karel & Piet Mertens. 2003. La valence: L’approche pronominale et son application au lexique verbal. *Journal of French Language Studies* 13. 63–104.

Vincze, Veronika, István Nagy T. & Gábor Berend. 2011. Detecting noun compounds and light verb constructions: A contrastive study. In *Proceedings of the ALC Workshop on Multiword Expressions: From Parsing and Generation to the*
*Real World* (MWE ’11), 116–121. Portland, OR, USA: Association for Computational Linguistics. http://www.aclweb.org/anthology/W11-0817.

Zeman, Daniel, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacova, Václava Kettnerová, Zdenka Uresova, Jenna Kanerva, Stina Ojala, Anna Missilä, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria dePaiva, Kira Droganova, Héctor Martínez Alonso, ÇağrÄ± Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Mannurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonca, Tatiana Lando, Rattima Nitisaroj & Josie Li. 2017. CoNLL 2017 shared task: Multilingual parsing from raw text to universal dependencies. In *Proceedings of the CoNLL 2017 shared task: Multilingual parsing from raw text to universal dependencies*, 1–19. Vancouver, Canada: Association for Computational Linguistics. http://www.aclweb.org/anthology/K17-3001.
<figure><img src="image_55.png" /></figure>
Chapter 11

Identification of multiword expressions:
A fresh look at modelling and evaluation

Shiva Taslimipoor

University of Wolverhampton

Omid Rohanian

University of Wolverhampton

Ruslan Mitkov

University of Wolverhampton

Afsaneh Fazly

Thomson Reuters

Automatic identification of multiword expressions (MWEs) in running text has
recently received much attention among researchers in computational linguistics.
The wide range of reported results for the task in the literature prompted us to take
a closer look at the algorithms and evaluation methods. For supervised classifica-
tion of Verb+Noun expressions, we propose a context-based methodology in which
we find word embeddings to be appropriate features. We discuss the importance
of train and test corpus splitting in validating the results and present type-aware
train and test splitting. Given our specialised data, we also discuss the benefits of
framing the task as classification rather than tagging.

1 Introduction

Ambiguity is a pervasive phenomenon in natural language. It starts from single
words, and can propagate through larger linguistic constructs. Multiword expres-

<figure><img src="image_56.png" /></figure>
sions (MWEs) which are idiosyncratic combinations of two or more words, be-
have differently in their separate usages in running text. In natural language pro-
cessing (NLP) tasks such as part-of-speech tagging, parsing and machine transla-
tion, these expressions should be treated either before the task (Nivre & Nilsson
2004) or combined with the process (Constant & Tellier 2012; Kordoni et al. 2011;
Nasr et al. 2015).

Examples of such expressions are: *take action, make sense and set fire*.<fnref n="1" /> MWEs
are a recurring theme in any language with some sources estimating their num-
ber to be in the same range as single words (Jackendoff 1997) or even beyond (Sag
et al. 2002). Besides, new expressions come to languages on a regular basis. It is
therefore not practically feasible to simply list MWEs in dictionaries or thesauri.

More importantly, most idiomatic expressions can also have literal meaning
depending on context. For instance consider the expression *play games*. It is
opaque with regards to its status as an MWE and depending on context could
mean different things. For example in *He went to play games online* it has a literal
sense but is idiomatic in *Don't play games with me as I want an honest answer*. Re-
solving these cases is critically important in many NLP applications (Katz 2006).
Katz (2006) framed the task as sense disambiguation. Tagging corpora for MWEs
or token-based identification of MWEs is an example of a task where it is nec-
essary to differentiate between idiomatic and literal usages of each expression
type.

Studies on MWEs can be divided into two main categories. One includes works regarding the canonical forms of expressions, their lexical properties and their potential to be considered as MWEs, namely type-based extraction of MWEs or MWE discovery; the other regards studies on tagging texts for the idiomatic usages of expressions, namely MWE tagging or token-based identification of MWEs. The former is a traditional approach which is of use to lexicographers as pointed in Ramisch (2014); the latter though, is more practical for NLP applications (Schneider et al. 2016).

Although discovering canonical forms of multiword expressions is still an ac-
tive research area (Salehi & Cook 2013; Farahmand & Martins 2014), recently
there is a significant move towards automatic tagging of corpora for MWEs
(Schneider et al. 2014; Constant & Tellier 2012).

The focus of our study is token-based identification of MWEs, and we model
it as a classification, rather than a sequence labelling problem. To determine the
idiomaticity of each Verb+Noun occurrence, we experiment with using solely

<footnote n="1">¹MWEs combine words from many different parts of speech. The pattern in our datasets is Verb+Noun, so all the examples in this chapter are of this kind.</footnote>
context features without any sophisticated linguistic information. We do not exploit parsing, tagging or external lexicon-based information. To discriminate between idiomatic and literal Verb+Noun expression tokens, we have proposed a context-based classification approach expounded in detail in the previous publication (Taslimipoor et al. 2017). In this chapter, we build on this approach, experimenting with additional languages and several more sophisticated machine learning models. However, here we take a closer look at modelling and evaluation aiming at devising approaches that have more generalisation power and lead to less misleading results. We also conduct experiments to better demonstrate the suitability of framing our task as classification.

For token-based identification of MWEs, there is a wide range of results in the literature reported as the state-of-the art: from F-score of 64% with the DiM-SUM dataset (Schneider et al. 2016) to 90% (Al Saied et al. 2017) for a dataset in the last PARSEME shared task (Savary et al. 2017). We find that in order for the performance results not to be misleadingly high, the distribution of the tokens between train and test corpus, henceforth called *TRAIN AND TEST SPLITTING*, should be controlled. Failure to do so will result in a kind of overfitting which could be overlooked during evaluation. For instance, an expression like *take advantage* is idiomatic consistently in all its usages in text. When different occurrences of this expression exist in both train and test corpus, the model memorises it from training data and predicts it very well in the test. Since such expressions are highly frequent, this memorisation helps the model to achieve erroneously high performance scores.

In the process of supervised identification of MWEs, we make observations with regards to the following: (1) the effect of train and test splitting of the tokens on generalisability of a model; (2) comparison between sequence labelling (tagging) and sequence classification.

## 1.1 Literature review

Identification of MWEs was shown to be effective in different NLP tasks, such as machine translation (Pal et al. 2011) and automatic parsing (Constant et al. 2012). There exists a considerable body of work in the literature attempting to investigate lexical and syntactic properties of expressions to account for their potential for being MWEs (Ramisch 2014; Baldwin & Kim 2010). However, recently there has been a great attention given to identifying where exactly this potential takes effect by tagging a running text for each individual occurrence (token) of an expression (Schneider et al. 2014; Constant & Tellier 2012; Gharbieh et al. 2017). Token-based identification of MWEs is effective in disambiguating be-
tween different behaviours of expressions in their individual usages. Evaluating
all occurrences of expressions in the whole corpus of big size is not feasible. For
this reason we have gathered a specialised dataset of concordances of particular
expressions.

To the best of our knowledge, there are very few comprehensive tagged cor-
pora for MWEs available, among which DiMSUM by Schneider et al. (2016) is
very recent and well-cited. This corpus was used in the SemEval (2016) shared
task in Detecting Minimal Semantic Units and their Meanings (DiMSUM). It is
not particularly clear if the current methodologies applied to this corpus are ca-
pable of disambiguating between different usages of one specific canonical form.

Cook et al. (2008) prepared a dataset of English Verb+Noun constructions, categorising expressions based on their idiomaticity and how consistent they behave in their different usages. Fazly et al. (2009) have used that dataset for classifying Verb+Noun tokens into idiomatic or literal categories.

Katz (2006) used context features for identifying the idiomaticity/non-compo-
sitionality of MWEs in a different way. They represent different occurrences of
an expression using LSA vectors and show that the vectors of the expressions
in their idiomatic sense are very different from those of the same expressions
in literal sense. Based on this observation they classify a test expression token
depending on whether it is more similar to the idiomatic sense of the expression
in training data or to the literal sense.

Scholivet & Ramisch (2017) recently tried to disambiguate a number of opaque French expressions using their contexts. They proposed a tagging approach using unigram and bigram features of the word forms and their POS. Qu et al. (2015) found word embedding representation of the words in context very useful for tagging a text with MWEs. We also used word vector representations of the verb and noun components of the expression and the words in a window size of two on the right of the expression as features for classifying expressions as MWE or not.

While most of the previous work on token-based identification of MWEs ap-
plied sequence tagging approaches using some kind of IOB labeling, Legrand &
Collobert (2016) looked at the problem as classification. They proposed a neu-
ral network based approach that learns fixed-size representations for arbitrary
sized chunks which is able to classify these representations as MWE or not. They
showed better performance in MWE identification over the CRF-based approach
in Constant et al. (2013).
## 1.2 Outline of the proposal

In almost all of the previous work on supervised modelling of MWE tokens, data is randomly split into train and test sets. In a random splitting, it is possible for occurrences of the same expression type to occur in both train and test sets. There are many instances where the expression almost always behaves idiomatically (e.g. *take part, make progress*) or literally (e.g. *eat food, give money*). In such cases a model learns every feature related to the POS and lemma form of the expression, and naturally can predict the correct tag for the expression perfectly in the test set (regardless of the expression being idiomatic or literal).

Having observed this issue, for evaluation we propose and perform type-aware train and test splitting. To this end we divide expression types into train and test folds and gather all occurrences of each type into the same fold. This makes the predication rigorous, since the model performs cross-type learning. One interesting study that considers cross-type learning of MWEs is the one by Fothergill & Baldwin (2012). However, they did not clearly explain the general advantages and effects of cross-type classification in evaluation. They used the approach in order to learn better features from specialised MWE resources.

We propose type-aware splitting of the data as a supplementary benchmark for evaluating MWE identification. We design experiments to show the effectiveness of this kind of evaluation in assessing the generalisability of models.

Recent studies on token-based identification of MWEs are heading towards using structured sequence tagging models. The choice of the model based on the data is an important issue. Our data includes occurrences of specific Verb+Noun expressions with the context around them. This makes it possible to have sizeable datasets annotated for a specific type of MWE in order to have an extensive evaluation. We observe that our data cannot benefit from sequence tagging and a regular classification approach can more reasonably model the data. We show better results from classification over a tagging model. Other than traditional machine learning classification approaches, we also propose a neural-network model by combining convolutional neural network and long short term memory models for identifying MWEs. Although some deep learning models have already been investigated for tagging MWEs by Gharbieh et al. (2017), to the best of our knowledge this is the first time this approach has been applied for classifying MWE instances.

We extensively discuss the following: 1) the division of data into train and test sets for evaluation and 2) the choice of model (classification versus tagging) based on the data.
## 2 Context-based identification of MWEs

In this study we use context features in a supervised environment to identify the idiomaticity of Verb+Noun expression tokens. In order to construct context features, for our first set of experiments (§4.1), given each occurrence of a Verb+Noun combination, we concatenate four different word vectors corresponding to the verb, noun, and their two following adjacent words while preserving the original order (following the previous work by Taslimipoor et al. 2017). Concatenated word vectors are fed into different classification models to be evaluated in terms of their performance.

The classification algorithms that have been used are Logistic Regression (LR), Decision Trees (DT), Random Forest (RF), Multi Layer Perceptron (MLP) and Support Vector Machine (SVM). We also experimented with different neural network-based classification models. The best result is achieved with a combination of bidirectional Long Short-Term Memory network with a convolutional layer as a front-end (ConvNet+LSTM).

For the second set of experiments (§4.2), in which we compare Conditional Random Field (CRF) as a tagger with a simple Naive Bayes Classifier (NBC), we consider simple word forms of the verb, the noun, and the two words after as lexical context features.

We conducted our extensive experiments with Italian. The experiments are augmented by applying the approach also for smaller data in Spanish and English.

## 3 Experiments

### 3.1 Data

We first experiment with two similarly formatted datasets in Italian and Spanish and later also on a standard available dataset for English.

For Italian, our data includes a large set of concordances of Verb+Noun expressions.<fnref n="2" /> Each item in the dataset is one concordance of a Verb+Noun expression and the whole item is annotated with 1 if the Verb+Noun inside is an MWE and with 0 otherwise. The data as explained in Taslimipoor et al. (2016) was annotated by two native speakers with Kappa agreement measure of 0.65. We resolved the disagreements by employing a third annotator who decided on most (but not all)

<footnote n="2">²The data as described in Taslimipoor et al. (2016) was gathered for four light verbs *fare*, *dare*, *trovare* and *prendre*. For some examples of expression instances refer to the same work.</footnote>
cases of disagreements. This results in 20,030 concordances of 1,564 expression
types. The Italian data is very imbalanced and almost 70% of the data is marked
as MWE. To resolve this issue, we ignore the 15 most frequent expression types
which are exclusively marked as MWE and also the expressions with frequency
lower than 3. As a result we run the experiments on 18,540 concordances of 940
expression types.

For Spanish, we extracted concordances of Verb+Noun expressions in the same
way using SketchEngine (Kilgarriff et al. 2004).<fnref n="3" /> After ignoring the concordances
for five most frequent expressions, 3,918 usages were marked by two native
speakers. The Kappa inter-annotator agreement was 0.55. Having seen the ob-
served agreement of 0.79, we ignored all cases of disagreements and considered
only the concordances on which both annotators agreed. This results in 3,090
concordances of 747 expression types.

For English, we employ a standard dataset called VNC-tokens prepared by
Cook et al. (2008).<fnref n="4" /> The dataset is a benchmark for English verb-noun idiomatic
expressions and was used for identifying MWE tokens in a number of previous
studies such as Fazly et al. (2009) and Salton et al. (2016). The dataset includes
sentences from the BNC corpus including occurrences of Verb+Noun expressions
and is suitable for our task since it contains expressions with both skewed and
balanced behaviour in being literal or idiomatic. Rather than concordances, it in-
cludes sentences from BNC containing occurrences of Verb+Noun expressions.
Two English native speakers selected the expression types based on whether they
have the potential for occurring in both idiomatic or literal senses. Although this
dataset is slightly different from our Italian and Spanish data (which are extracted
randomly), it has the same favourable pattern of different occurrences of same
expression types that can be split into train and test. We find it interesting to
investigate our observations on a differently gathered but standard dataset. The
Verb+Nouns in this dataset are not necessarily continuous. We ignore the cases
where the Verb+Noun occurs in passive form and the ones that the annotators
were unsure of and this results in 2,499 sentences consisting of Verb+Noun ex-
pressions. The statistics of the data for all three languages are reported in Table 1.

For all the three datasets, we consider the same context words as features for
classification: we extract the vectors of the verb, noun and the two words after
the noun.

<footnote n="3">³For Spanish, we focused on four light verbs *tener*, *hacer*, *dar* and *tomar*, similar to the ones we use for Italian.</footnote>

<footnote n="4">⁴The dataset is available in https://sourceforge.net/projects/multiword/files/MWE_resources/20110627/</footnote>
<table><caption>Table 1: Distribution of the data</caption><thead><tr><th></th><th>Italian</th><th>Spanish</th><th>English</th></tr></thead><tbody><tr><td>Expression types</td><td>940</td><td>747</td><td>53</td></tr><tr><td>Expression tokens</td><td>18,540</td><td>3,090</td><td>2,499</td></tr><tr><td>MWE tokens</td><td>10,804 (58.27%)</td><td>2,094 (66.57%)</td><td>1,981 (79.27%)</td></tr></tbody></table>

## 3.2 Evaluation

In all cases classifier performance was measured using 10-fold cross-validation.

### 3.2.1 Standard splitting of data into train and test

In the standard method of performing cross-validation, the whole data is randomly divided into *k* folds and then the model is repeatedly trained on the data of *k* - 1 folds and tested on the data of the remaining fold. The result is averaged among *k* different iterations. In our task, we find the result of this evaluation misleading, since the repetition of the same expression in both train and test partitions helps the model to predict those specific types of expressions well, while the model might not work for new unseen expressions in test. Even stratified cross-validation suffers from the same kind of overfitting. In standard stratified cross-validation, imbalance is coped with by controlling the distribution of labels alone, so that all folds have the same distribution of labels. Similar to standard cross-validation, this method is not informed about the idiosyncratic distribution of types and tokens.

Therefore, these methods of evaluation cannot precisely reflect the effectiveness of the model or features and show better results for models that are more prone to overfitting. It is not particularly clear from this kind of evaluation if a good performing model could be generalised to unseen expressions and also to ambiguous expressions that have balanced distribution of occurrences as literal or idiomatic. We show the performance computed using this type of evaluation for different classifiers in Table 2.

### 3.2.2 Type-aware splitting and evaluation

We perform a custom cross-validation by splitting the expression occurrences into different folds considering their types/canonical forms. We split the expression types into *k* groups and all the occurrences of the expressions in the *k*<sup>th</sup>
group goes into the *k*^th^ fold. This method ensures that the model performs cross-type learning and generalises to tokens from unseen types in the test fold. In other words, the model is learning the features and general patterns and does not overfit on highly recurrent token occurrences. The results for all classifiers evaluated using this approach is reported in Table 3.

# 4 Results

In this section, first a comparison of several classifiers using different train and test splitting methods is reported; then we present experiments using sequence tagging for identifying MWEs; and finally, the effectiveness of neural network-based word embeddings compared with count-based representations was analysed using one of the best classifiers.

## 4.1 Regular and type-aware evaluation

Evaluation performances for all classifiers using two different kinds of train and test splitting, namely regular (random) and our proposed type-aware, are reported in Table 2 and Table 3. The columns of the tables represent the results for Italian (IT), Spanish (ES) and English (EN). All traditional classifiers in this experiment use the same vectorised context features. The word vectors used in this study are available online.<fnref n="5" /> The generated Italian and Spanish word embeddings applied Gensim's skipgram word2vec model with the window size of 10 to extract vectors of size 300. For English we use word embeddings of the same dimension trained using Glove (Pennington et al. 2014) algorithm available via spaCy.<fnref n="6" />

We also report the results from a more sophisticated neural network based architecture comprising of a BiLSTM with an additional convolutional layer as a front-end (ConvNet+LSTM). For this architecture the context window size is 2 (two words before and two words after the Verb+Noun expression).<fnref n="7" /> Implementation details of these experiments can be found at https://github.com/shivaat/VN-tokens-clf.

Different classifiers show high performance with not much difference using regular cross-validation in which tokens are distributed into separate folds re-

<footnote n="5">⁵http://hlt.isti.cnr.it/wordembeddings/ for Italian and https://github.com/Kyubyong/ wordvectors for Spanish</footnote>

<footnote n="6">⁶https://spacy.io/docs/usage/word-vectors-similarities</footnote>

<footnote n="7">⁷The difference in results were negligible when considering only the two context words on the right.</footnote>
<table><caption>Table 2: Regular evaluation results: accuracy (standard deviation)</caption><thead><tr><th>Classifiers</th><th>IT</th><th>ES</th><th>EN</th></tr></thead><tbody><tr><td>Majority Baseline</td><td>0.5827</td><td>0.6657</td><td>0.7927</td></tr><tr><td>LR</td><td>0.8869 (0.007)</td><td>0.9129 (0.011)</td><td>0.8651 (0.020)</td></tr><tr><td>DT</td><td>0.8905 (0.008)</td><td>0.9065 (0.017)</td><td>0.8799 (0.018)</td></tr><tr><td>RF</td><td>0.9218 (0.005)</td><td>0.9337 (0.019)</td><td>0.9024 (0.017)</td></tr><tr><td>MLP</td><td>0.9069 (0.006)</td><td>0.933 (0.009)</td><td>0.9056 (0.016)</td></tr><tr><td>SVM</td><td>0.9116 (0.005)</td><td>0.9207 (0.009)</td><td>0.7927 (0.021)</td></tr><tr><td>ConvNet+LSTM</td><td><strong>0.9220 (0.007)</strong></td><td><strong>0.9668 (0.01)</strong></td><td><strong>0.8860 (0.024)</strong></td></tr></tbody></table>

<table><caption>Table 3: Type-aware evaluation results: accuracy (standard deviation)</caption><thead><tr><th>Classifiers</th><th>IT</th><th>ES</th><th>EN</th></tr></thead><tbody><tr><td>Majority Baseline</td><td>0.5827</td><td>0.6657</td><td>0.7927</td></tr><tr><td>LR</td><td>0.6909 (0.06)</td><td>0.8178 (0.074)</td><td>0.8092 (0.149)</td></tr><tr><td>DT</td><td>0.6048 (0.03)</td><td>0.7483 (0.078)</td><td>0.6327 (0.128)</td></tr><tr><td>RF</td><td>0.6337 (0.08)</td><td>0.7604 (0.097)</td><td>0.7321 (0.19)</td></tr><tr><td>MLP</td><td>0.7053 (0.06)</td><td>0.8319 (0.086)</td><td>0.7294 (0.169)</td></tr><tr><td>SVM</td><td><strong>0.7369 (0.07)</strong></td><td>0.8460 (0.093)</td><td>0.8062 (0.152)</td></tr><tr><td>ConvNet+LSTM</td><td>0.6601 (0.053)</td><td><strong>0.8681 (0.072)</strong></td><td><strong>0.8112 (0.106)</strong></td></tr></tbody></table>

gardless of their types (Table 2). ConvNet+LSTM, in particular, performs the best, which we believe is the result of overfitting arising from this method of train and test splitting. However, we can see notable differences between classifiers in Table 3 where we cross validate in a way that no same expression type occurs in both train and test partitions.

In the case of cross-type learning (Table 3), the SVM classifier showed the best results in identifying MWEs using vectorised context features for Italian, and close to the second best for Spanish and English data for which ConvNet+LSTM is the best. The performance of this classifier is followed by that of MLP and LR for both Italian and Spanish. For English the results of SVM and LR are comparable. Computed performance for other classifiers like DT and RF dropped sharply when we use our type-aware cross-validation. This is also the case for ConvNet+LSTM for Italian data. This experiment determines how well a classifier can generalise among different expression types. SVM and LR in particular
are shown to be fairly suitable for cross-type identification of MWEs. MLP also
performs relatively well overall.

As for the English data it is worth noting that the VNC data is very imbalanced
with the majority baseline of 0.7927 which is difficult to beat by classifiers.

## 4.2 Sequence classification versus sequence tagging

The experimental data in this study can be perfectly processed with standard classification approach, since the goal is to predict idiomaticity of an expression in a given context. However, Scholivet & Ramisch (2017) modelled such a data with sequence tagging. We believe that since not all the words in a sequences are going to be tagged, MWE identification using such a data cannot benefit from sequence labelling. We applied sequence tagging on the data to properly investigate the effects. Specifically, simple Naive Bayes Classifier (NBC) was considered as a simple sequence classification methodology and Conditional Random Field (CRF) was used as the sequence tagging approach. Both of the models use simple nominal features: the verb, the noun, and the two words after the noun.<fnref n="8" /> The results are reported in Table 4 in terms of accuracy.

<table><caption>Table 4: Performance of sequence classification versus sequence tagging</caption><thead><tr><th rowspan="2"></th><th colspan="3">regular<br>cross-validation</th><th colspan="3">type-aware<br>cross-validation</th></tr><tr><th>IT</th><th>ES</th><th>EN</th><th>IT</th><th>ES</th><th>EN</th></tr></thead><tbody><tr><th>NBC</th><td>0.9504</td><td>0.9601</td><td>0.8560</td><td>0.7291</td><td>0.7298</td><td>0.6013</td></tr><tr><th>CRF</th><td>0.9165</td><td>0.9142</td><td>0.8176</td><td>0.6447</td><td>0.7199</td><td>0.6848</td></tr></tbody></table>

As can be seen in Table 4, CRF cannot even beat the simple naive bayes classifier except in the case of English data (when we apply cross-type learning). This is because our data is naturally suited for sequence classification and cannot benefit from sequence labelling models.

## 4.3 Effectiveness of word embedding representation

To specifically show the effectiveness of neural network-based embeddings for
the classifiers to identify Verb+Noun MWEs, we performed an experiment using

<footnote n="8">⁸The features are the surface text occurrences of these words.</footnote>
sparse bag-of-words count vectors with tf-idf weighting. In this case each sentence is considered as a collection of words, disregarding any word order. The sparse vector for each word is constructed based on its occurrence in different sentences. Each entry of the vector is weighted by tf (the word frequency) * idf (the inverse of frequency of the sentences containing the word). Similar to previous experiments, we feed the vectors to a Multi Layer Perceptron (MLP) which works reasonably well compared to other classifiers based on the previous experiment. Note that the execution time for the best performing model, SVM, is almost 5 times that of MLP which makes it inefficient in comparison. The results of this comparison can be seen at Table 5.

<table><caption>Table 5: The accuracy of MLP in identifying Verb+Noun MWEs using word2vec and count-based embedding</caption><thead><tr><th rowspan="2"></th><th colspan="3">Accuracy (std.)</th></tr><tr><th>IT</th><th>ES</th><th>EN</th></tr></thead><tbody><tr><td>MLP with count based embedding</td><td>0.6504 (0.0354)</td><td>0.7851 (0.042)</td><td>0.7002 (0.099)</td></tr><tr><td>MLP with word2vec</td><td>0.7053 (0.06)</td><td>0.8319 (0.086)</td><td>0.7294 (0.169)</td></tr></tbody></table>

The results in Table 5 show the improvement in performance when using word embeddings rather than the vanilla count-based vectors for all three languages (although less significant for English).

## 5 Discussion

In order to understand the argument behind type-aware evaluation and decide its applicability, we have to look at the distribution of data points. In the Italian data, for instance, the majority of data points belong to MWE types whose tokens occur invariably as idiomatic or literal only. In other words, if we plot the distribution of tokens with regards to the degree of idiomaticity of their corresponding types, we would see a skewed distribution (even after ignoring the 15th most frequent expressions), where only a smaller portion of tokens belong to MWE types whose usages can be fluid between literal and idiomatic. In such a scenario, a model easily overfits on the majority of the data, where labels were assigned invariably. However, this skewedness is not necessarily reflected in the distribution of MWE labels, as we might see a relatively balanced distribution of literal and idiomatic labels. This means there might be no severe class imbalance in the dataset, but within-class imbalance (Ali et al. 2015).
<figure><img src="image_57.png" /><figcaption>Figure 1: Distribution of expression types.</figcaption></figure>

To illustrate the point, we operationalise two categories for MWE types, namely Consistent (C) and Fluid (F). Those types whose tokens occur more than 70% or less than 30% of the time as only literal or idiomatic are tagged as C, and the rest are considered F. Accordingly, Figure 1 shows the distribution of the expression types with regards to the behavior of their corresponding tokens. As can be seen, the majority of expressions with higher token frequencies are from the sub-class C. For this reason, evaluation using a vanilla cross validation or even stratified cross validation would not provide us with reliable results, since splitting of train and test disregards the within-class imbalance inherent in the data.

Since this is the case with data in real world, we propose type-aware train and test splitting as a supplementary approach for modelling the data and evaluating the results. This way, we make sure that a model has the best ability for generalisation, learns general properties for MWEs and is not merely based on memorising the words that construct MWEs.

It is worth noting that we did not use any linguistic or lexical features and we expect vector representation of context to be generalisable enough. Even with these generalisable features we observe substantial differences between regular and type-aware cross-validation. A proper method for train and test splitting is even more essential to validate the evaluation when a model trains on more exact features such as lexical ones.

With regards to previous data and models for MWEs, DiMSUM is one of the most noteworthy shared tasks. DiMSUM includes a recent tagged corpus for MWEs with a fairly small size of 4,799 sentences in train and 1,000 in test, including all types of MWEs. With such limited data, we observed only a few number
of expressions of the form Verb+Noun occurring in both train and test. To give
an example, with a selection of 6 most frequent light verbs, all their combina-
tions with nouns are only 13 occurrences in the test data, out of which only 3
are MWEs. There are no repeated occurrences of these cases in both train and
test data. Therefore, we believe that this data inherently does not lead to mislead-
ing results. In other words, a model that works well on this data could be fairly
generalised.

Gharbieh et al. (2017) showed better performance when using deep neural net-
work models compared with traditional machine learning on DiMSUM. How-
ever, in our experiment of type-aware classification, SVM performed the best,
even outperforming LSTM and ConvNet and their combinations for Italian and
Spanish. Since neither DiMSUM nor our data is big enough for a proper analysis
with deep learning, more studies are required to find the most effective model to
identify MWEs.

Another data for token-based identification of MWEs in English that we also
used in this study is VNC-tokens (Cook et al. 2008). One advantage of this corpus
is that the data is particularly gathered for the task of disambiguation between
idiomatic and literal usages of expressions. Before the annotation, they selected
only the expressions that have the potential for occurring in both idiomatic and
literal senses. Although we did not follow the initial development splitting of
the data for this study (i.e. we followed our proposed way of splitting the data
into train and test), the development and test splitting of this data is type-aware.
Therefore, an experiment with this data, is able to truly measure generalisation.

In PARSEME shared task (Savary et al. 2017), which features the most recent
multi-lingual data for MWEs, Maldonado et al. (2017) presented statistics on the
percentage of previously seen data in test sets of all languages (i.e. proportion
of MWE instances in the test set that were seen also in the training set). The
correlation between these percentages and the results stress the need for proper
train and test splitting. Maldonado & QasemiZadeh (2018 [this volume]) further
discuss the characteristics of the shared task data and report the performance
results of the systems on seen and un-seen data separately. The experiments with
the data for the Parseme shared task, which is also discussed in Savary et al.
(2018 [this volume]), would definitely benefit from such type-aware train and
test splitting.
# 6 Conclusions

In this study, we explored a context-based classification method for identification of Verb+Noun expressions. We employed word embedding to represent context features for MWEs. We evaluated the methodology using type-aware cross-validation and discussed its effectiveness compared with standard evaluation. We argue that only this proposed method properly accounts for the generalisability of a model. We also showed that our data (and similar ones) for this task cannot benefit from structured sequence tagging models.

The effectiveness of word embeddings as context features for identifying MWEs should be examined in more detail with datasets of larger size and with more sophisticated embeddings that consider linguistic features. We would also like to analyse the effect of our proposed approach on unseen and less frequent data.

# Acknowledgements

The authors would like to thank Manuela Cherchi, Anna Desantis, Martina Cotella, Andrea Silvestre and Alondra Nava Zea for their help in annotations.

# Abbreviations

<table><tr><td>BILSTM</td><td>bi-directional long short-term memory</td><td>LSTM</td><td>long short-term memory</td></tr><tr><td>CRF</td><td>conditional random field</td><td>MLP</td><td>multi-layer perceptron</td></tr><tr><td>CONVNet</td><td>convolutional neural network</td><td>MWE</td><td>multiword expression</td></tr><tr><td>DT</td><td>decision tree</td><td>NLP</td><td>Naive Bayes classifier</td></tr><tr><td>IOB</td><td>inside-outside-beginning</td><td>POS</td><td>part of speech</td></tr><tr><td>LSA</td><td>Latent Semantic Analysis</td><td>RF</td><td>random forest</td></tr><tr><td>LR</td><td>logistic regression</td><td>SVM</td><td>support vector machine</td></tr></table>

# References

Al Saied, Hazem, Matthieu Constant & Marie Candito. 2017. The ATILF-LLF system for parseme shared task: A transition-based verbal multiword expression tagger. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 127–132. Association for Computational Linguistics. DOI:10.18653/v1/W17-1717
Ali, Aida, Siti Mariyam Shamsuddin & Anca L. Ralescu. 2015. Classification with class imbalance problem: A review. *International Journal of Advances in Soft Computing and its Application* 7(3). 687-719.

Baldwin, Timothy & Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya & Fred J. Damerau (eds.), *Handbook of Natural Language Processing, Second edition*, 267-292. Boca Raton: CRC Press.

Constant, Matthieu, Marie Candito & Djamé Seddah. 2013. The LIGM-alpage architecture for the SPMRL 2013 shared task: Multiword expression analysis and dependency parsing. In *Proceedings of the 4th Workshop on Statistical Parsing of Morphologically Rich Languages*, 46-52.

Constant, Matthieu, Anthony Sigogne & Patrick Watrin. 2012. Discriminative strategies to integrate multiword expression recognition and parsing. In *Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics*, 204-212. Association for Computational Linguistics.

Constant, Matthieu & Isabelle Tellier. 2012. Evaluating the impact of external lexical resources into a CRF-based multiword segmenter and part-of-speech tagger. In *Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC 2012)*, 646-650. European Language Resources Association (ELRA).

Cook, Paul, Afsaneh Fazly & Suzanne Stevenson. 2008. The VNC-tokens dataset. In *Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions (MWE ’08)*, 19-22. Association for Computational Linguistics.

Farahmand, Meghdad & Ronaldo Martins. 2014. A supervised model for extraction of multiword expressions, based on statistical context features. In *Proceedings of the 10th Workshop on Multiword Expressions (MWE ’14)*, 10-16. Association for Computational Linguistics.

Fazly, Afsaneh, Paul Cook & Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. *Computational Linguistics* 35(1). 61-103. http://aclweb.org/anthology/J09-1005.

Fothergill, Richard & Timothy Baldwin. 2012. Combining resources for MWE-token classification. In *Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval ’12)*, 100-104. Association for Computational Linguistics.

Gharbieh, Waseem, Virendra Bhavsar & Paul Cook. 2017. Deep learning models for multiword expression identification. In *Proceedings of the 6th Joint Confer-
ence on *Lexical and Computational Semantics* (*SEM 2017*), 54–64. Association for Computational Linguistics.

Jackendoff, Ray. 1997. *The architecture of the language faculty*. Cambridge, MA: MIT Press.

Katz, Graham. 2006. Automatic identification of non-compositional multi-word expressions using latent semantic analysis. In *Proceedings of the ACL/COLING-06 Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties* (MWE ’06), 12–19. Association for Computational Linguistics.

Kilgarriff, Adam, Pavel Rychlý, Pavel Smrz & David Tugwell. 2004. The Sketch Engine. In Geoffrey Williams & Sandra Vessier (eds.), *Proceedings of the 11th EU-RALEX international congress*, 105–116. Lorient, France: Université de Bretagne-Sud, Faculté des lettres et des sciences humaines.

Kordoni, Valia, Carlos Ramisch & Aline Villavicencio. 2011. Proceedings of the ACL Workshop on Multiword Expressions: From Parsing and Generation to the Real World. In (MWE ’11). Association for Computational Linguistics.

Legrand, Joël & Ronan Collobert. 2016. Phrase representations for multiword expressions. In *Proceedings of the 12th Workshop on Multiword Expressions* (MWE ’16), 67–71. Association for Computational Linguistics. http://anthology.aclweb.org/W16-1810.

Maldonado, Alfredo, Lifeng Han, Erwan Moreau, Ashjan Alsulaimani, Koel Dutta Chowdhury, Carl Vogel & Qun Liu. 2017. Detection of verbal multi-word expressions via conditional random fields with syntactic dependency features and semantic re-ranking. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 114–120. Association for Computational Linguistics. DOI:10.18653/v1/W17-1715

Maldonado, Alfredo & Behrang QasemiZadeh. 2018. Analysis and Insights from the PARSEME Shared Task dataset. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 149–175. Berlin: Language Science Press. DOI:10.5281/zenodo.1469557

Nasr, Alexis, Carlos Ramisch, José Deulofeu & André Valli. 2015. Joint dependency parsing and multiword expression tokenization. In *Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, 1116–1126. Association for Computational Linguistics. http://www.aclweb.org/anthology/P15-1108.

Nivre, Joakim & Jens Nilsson. 2004. Multiword units in syntactic parsing. In *Workshop on methodologies and evaluation of multiword units in real-world ap-
plications (MEMURA 2004), 39–46. http://stp.lingfil.uu.se/~nivre/docs/mwu.pdf.

Pal, Santanu, Tanmoy Chakraborty & Sivaji Bandyopadhyay. 2011. Handling multiword expressions in phrase-based Statistical Machine Translation. In *Proceedings of the 13th machine translation summit* (MT Summit 2011), 215–224. September 19-23, 2011.

Pennington, Jeffrey, Richard Socher & Christopher D. Manning. 2014. GloVe: global vectors for word representation. In *Proceedings of the Conference on Empirical Methods in Natural Language Processing* (EMNLP 2014), 1532–1543. http://www.aclweb.org/anthology/D14-1162. October 25–29, 2014.

Qu, Lizhen, Gabriela Ferraro, Liyuan Zhou, Weiwei Hou, Nathan Schneider & Timothy Baldwin. 2015. Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: the impact of word representations on sequence labelling tasks. In *Proceedings of the SIGNLL Conference on Computational Natural Language Learning* (CoNLL 2015), 83–93.

Ramisch, Carlos. 2014. *Multiword expressions acquisition: A generic and open framework* (Theory and Applications of Natural Language Processing XIV). Cham, Switzerland: Springer. 230. http://link.springer.com/book/10.1007/2F978-3-319-09207-2.

Sag, Ivan A., Timothy Baldwin, Francis Bond, Ann A. Copestake & Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In *Proceedings of the 3rd International Conference on Computational Linguistics and Intelligent Text Processing*, vol. 2276/2010 (CICLing ’02), 1–15. Springer-Verlag.

Salehi, Bahar & Paul Cook. 2013. Predicting the compositionality of multiword expressions using translations in multiple languages. In *Second Joint Conference on Lexical and Computational Semantics*, vol. 1 (*SEM 2013*), 266–275. June 13-14, 2013.

Salton, Giancarlo, Robert Ross & John Kelleher. 2016. Idiom token classification using sentential distributed semantics. In *Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers)*, 194–204. Berlin, Germany: Association for Computational Linguistics.

Savary, Agata, Marie Candito, Verginica Barbu, Mititelu, Eduard Bejček, Fabienne Cap, Slavomír Čéplö, Silvio Ricardo Cordeiro, Gülşen Eryiğit, Voula Giouli, Maarten van Gompel, Yaakov HaCohen-Kerner, Jolanta Kovalevskaitė, Simon Krek, Chaya Liebeskind, Johanna Monti, Carla Parra Escartín, Lonneke van der Plas, Behrang QasemiZadeh, Carlos Ramisch, Federico Sangati, Ivelina Stoyanova & Veronika Vincze. 2018. PARSEME multilingual corpus of verbal multiword expressions. In *Stella Markantonatou, Carlos Ramisch, Agata Savary &*
Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 87–147. Berlin: Language Science Press.
DOI:10.5281/zenodo.1471591

Savary, Agata, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova & Antoine Doucet. 2017. The PARSEME Shared Task on automatic identification of verbal multiword expressions. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 31–47. Association for Computational Linguistics. DOI:10.18653/v1/W17-1704

Schneider, Nathan, Emily Danchik, Chris Dyer & Noah A. Smith. 2014. Discriminative lexical semantic segmentation with gaps: Running the MWE gamut. *Transactions of the Association for Computational Linguistics* 2(1). 193–206. https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/281.

Schneider, Nathan, Dirk Hovy, Anders Johannsen & Marine Carpuat. 2016. SemEval-2016 task 10: Detecting minimal semantic units and their meanings (DiMSUM). In *Proceedings of the 10th International Workshop on Semantic Evaluation* (SemEval-2016), 546–559. Association for Computational Linguistics. http://www.aclweb.org/anthology/S16-1084.

Scholivet, Manon & Carlos Ramisch. 2017. Identification of ambiguous multiword expressions using sequence models and lexical resources. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 167–175. Association for Computational Linguistics. http://www.aclweb.org/anthology/W17-1723.

Taslimipoor, Shiva, Anna Desantis, Manuela Cherchi, Ruslan Mitkov & Johanna Monti. 2016. Language resources for Italian: Towards the development of a corpus of annotated Italian multiword expressions. In *Proceedings of 3rd Italian Conference on Computational Linguistics (CLiC-it 2016) & 5fth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian* (EVALITA 2016) (Collana dell'Associazione Italiana di Linguistica Computazionale), 5–6 December 2016. Torino: Accademia University Press. online.

Taslimipoor, Shiva, Omid Rohanian, Ruslan Mitkov & Afsaneh Fazly. 2017. Investigating the opacity of verb-noun multiword expression usages in context. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 133–138. Association for Computational Linguistics.
<figure><img src="image_58.png" /></figure>
Chapter 12

Comparing bilingual word embeddings
to translation dictionaries for extracting
multilingual collocation equivalents

Marcos Garcia

Universidade da Coruña

This chapter introduces a strategy for the automatic extraction of multilingual col-
location equivalents which takes advantage of parallel corpora to train bilingual
word embeddings. First, monolingual collocation candidates are retrieved using
syntactic dependencies and standard association measures. Then, the distributional
models are applied to search for equivalents of the elements of each collocation in
the target languages. The proposed method extracts not only collocation equiva-
lents with direct translations between languages, but also other cases where the
collocations in the two languages are not literal translations of each other. Several
experiments – evaluating collocations with five syntactic patterns – in English,
Spanish, and Portuguese show that this approach can effectively extract large sets
of bilingual equivalents with an average precision of about 85%. Moreover, prelim-
inary results on comparable corpora suggest that the distributional models can be
applied for identifying new bilingual collocations in different domains. This strat-
egy is compared to both hand-crafted bilingual dictionaries and to probabilistic
translation dictionaries learned from the same resources as the bilingual word em-
beddings, showing that it achieves much larger recall values while keeping high
precision results.

1 Introduction

MWEs have been repeatedly classified as an important problem for developing
Natural Language Processing (NLP) tools, as well as to automatically analyze

<figure><img src="image_59.png" /></figure>
linguistic utterances (Sag et al. 2002). Among the different types of MWEs, pro-
cessing collocations in an automatic way may pose various problems due to their
intrinsic properties such as compositionality or unpredictability (Mel'čuk 1998).

From a theoretical perspective, there are at least two main views on collo-
cations. On the one hand, there is a tendency to consider any frequent pair of
words to be a collocation (Smadja 1993; Evert & Kermes 2003; Kilgarriff 2006).
On the other hand, the phraseological tradition needs both a lexical restriction
and a syntactic relation to consider two lexical units as a collocation.<fnref n="1" /> From this
phraseological point of view, a collocation is a restricted binary co-occurrence
of lexical units between which a syntactic relation holds, and that one of the
lexical units (the *BASE*) is chosen according to its meaning as an isolated lexical
unit, while the other (the *COLLOCATE*) is selected depending on the base and the
intended meaning of the co-occurrence as a whole, rather than on its meaning as
an isolated lexical unit (Mel'čuk 1998). Thus, a noun in English such as *picture* (as
a direct object) requires the verb *to take* (and not *to do*, or *to make*) in the phrase
*take a picture*, while *statement* selects *to make* (*make a statement*).

In a bilingual (or multilingual) scenario, equivalent collocations are needed
to produce more natural utterances in the target language(s). In this regard, the
referred noun *fotografia* 'picture' would select the verb *tirar* 'to remove' in Por-
tuguese (*tirar uma fotografia*). Similarly the Spanish *vino* 'wine' would require
the adjective *tinto* (*vino tinto*), which is not the main translation of *red* (*red wine*).

The unpredictability of these structures poses problems for tasks such as ma-
chine translation, whose performance can benefit from lists of multilingual col-
locations or transfer rules for these units (Orliac & Dillinger 2003). In areas like
second language learning, it has been shown that even advanced learners need
to know which word combinations are allowed in a specific linguistic variety (Al-
tenberg & Granger 2001; Alonso-Ramos et al. 2010). Thus, obtaining resources of
multilingual equivalent collocations could be useful for a variety of applications
such as those mentioned above. However, this kind of resource is scarce, and
constructing them manually requires a large effort from expert lexicographers.

Since the 1990s, a number of approaches were implemented aimed at extract-
ing bilingual collocations, both from parallel corpora (Kupiec 1993; Smadja et al.
1996; Wu & Chang 2003), and from comparable or even from non-related mono-
lingual resources (Lü & Zhou 2004; Rivera et al. 2013), often combining statistical
approaches with the use of bilingual dictionaries to find equivalents of each base.

<footnote n="1">¹An overview of different views on collocations – both from theoretical and practical perspectives – can be found in Seretan (2011).</footnote>
This chapter explores the use of distributional semantics (by means of bilin-
gual word embeddings) for identifying bilingual equivalents of monolingual col-
locations: On the one hand, monolingual collocation candidates are extracted
using a harmonized syntactic annotation provided by Universal Dependencies
(UD),<fnref n="2" /> as well as standard measures for lexical association. On the other hand,
bilingual word embeddings are trained using lemmatized versions of noisy paral-
lel corpora. Finally, these bilingual models are employed to search for semantic
equivalents of both the base and the collocate of each collocation.

Several experiments using the OpenSubtitles2016 parallel corpora (Lison & Tiedemann 2016) in English, Portuguese, and Spanish show that the proposed method successfully identifies bilingual collocation equivalents with different patterns: *adjective-noun*, *noun-noun*, *verb-object*, *verb-subject*, and *verb-adverb*. Furthermore, preliminary results in comparable corpora suggest that the same strategy can be applied in this kind of resources to extract new pairs of bilingual collocations. In this regard, this chapter is an extended version of a previous work on bilingual collocation extraction (Garcia et al. 2017), including new collocation patterns and a larger evaluation which compares the proposed approach to probabilistic translation dictionaries (Hiemstra 1998; Simões & Almeida 2003).

Apart from this introduction, §2 includes a review of previous work on collo-
cation extraction, especially on papers dealing with bilingual resources. Then, §3
and §4 present and evaluate the method, respectively. Finally, some conclusions
and further work are discussed in §5.

## 2 Previous studies on collocation extraction

The extraction of monolingual collocation candidates (as well as other MWEs)
from corpora is a well-known topic in corpus and computational linguistics and
was the focus of a significant body of work in different languages.

In this respect, most strategies use statistical association measures on win-
dows of n-grams with different sizes (Church & Hanks 1990; Smadja 1993). Other
methods, such as the one presented by Lin (1999), started to apply dependency
parsing to better identify combinations of words which occur in actual syntactic
relations.

More recently, the availability of better parsers allowed researchers to combine
automatically obtained syntactic information with statistical methods to extract
collocations more accurately (Evert 2008; Seretan 2011).

<footnote n="2">²http://universaldependencies.org/</footnote>