polarity was also possible (e.g., *take in* vs. *take out*), (iv) if specific argument types (e.g., MANNER, RESULT, LOCATION, AFFECTED) were instantiated in the sentence, etc. In the rest of this section, we present the sense classes particles in compositional VPCs tend to fall into.

## 4.1 Sense classes for particles in VPCs

While particles may encode subtle nuances of meanings in each of their occurrences in (compositional) VPCs, they may also display some general senses across many VPCs. WordNet attempts to capture the nuances by storing each of the VPCs as a separate lexical item. However, this approach results in having as many sense categories as there are VPCs and we lose information about the common contributions made by the particles in VPC semantics which can be useful while producing semantic representation of sentences with new VPCs not stored in WordNet or another lexical resource. Hence, we focus on the general senses particles display across compositional VPCs.

We identified two general sense classes for the particles in compositional VPCs, namely DIRECTION and ASPECTUAL. The DIRECTION sense class has a number of subclasses, each instantiated by a specific directional particle, such as *away*, *down*, *in*, *off*, *on*, *out*, and *up* denoting a specific direction sense. For example, the directional particle *away* instantiates the subclass DIRECTION-AWAY in *I took one last look at the house and walked away*. Similarly, the particle *out* in *My mom never threw it out* and the particle *up* in *The magic ketchup should sink when you squeeze the bottle and float up when you release it* instantiate subclasses DIRECTION-OUT and DIRECTION-UP respectively. The DIRECTION sense class particles assume the RESULT role in relation to the verb. For example, in the DIRECTION-AWAY example, the particle *away* denotes the RESULT of a *walking* event in terms of the direction the AFFECTED entity is walking to.

The ASPECTUAL sense class has two subclasses, namely COMPLETELY and CONTINUING, where the particle modifies the verb by providing aspectual information. In terms of the TRIPS semantic roles, the aspectual sense particles assume the MANNER role in relation to the verb. The COMPLETELY sense is used to express that the activity denoted by the verb is performed to the full extent or with thoroughness. Particles with COMPLETELY sense may also emphasize the telicity of an action. For example, the particle *out* in *He sorted out every scrap of manuscript, every map, and the native letters* emphasizes that each of the items mentioned were thoroughly and completely sorted. Particles *up*, *out*, and *down* are used more often than other particles to convey this aspectual sense.
The CONTINUING sense is used to emphasize the durative nature of the event denoted by the verb. For example, in *Day after day she worked away* ... the particle *away* conveys that the activity continued for a duration. The particles in this class may also convey an iterative sense when they are used with semelfactive verbs, e.g., note the use of the particle *away* in *Start your explorations here, click away all you want*. Usually particles *away* and *on* are used to convey this sense in VPCs.

These sense classes correspond to the VPC classes based on their compositionality types mentioned in §3. For example, the DIRECTION sense class is generally instantiated by the symmetrically compositional or LV-compositional VPCs. The ASPECTUAL sense class, on the other hand, is instantiated by the LP-compositional VPCs. Besides these two sense classes, there are a few other senses that particles may express in VPCs corresponding to the symmetrically compositional and LV-compositional VPCs. For example, the senses IN-WORKING-ORDER-VAL and NOT-IN-WORKING-ORDER-VAL mentioned in Table 2 and Table 4 are expressed by particles in LV-compositional VPCs. These senses are usually conveyed by the particles *up*, *down*, and *out*. Similarly, the sense DISAPPEARANCE, conveyed by the particle *away*, may be taken to correspond to the particle in symmetrically compositional VPCs. Table 2 contains information about each of these senses and corresponding sense classes for the particles *up*, *down*, *away*, and *out*. Table 3 and Table 4 present generalizations corresponding to these senses and sense classes.

We did not get very high agreement on human annotations for the sense labels for the particles in VPCs from Test Set 1. The three annotators had full agreement in 20 cases (40% of the cases). 75% of these 20 cases had DIRECTION sense, 20% had COMPLETELY, and 5% IDIOMATIC. The disagreement cases involved labels such as DIRECTION vs. COMPLETELY, DIRECTION vs. IDIOMATIC, or COMPLETELY vs. IDIOMATIC. For a more reliable identification of senses for the particles in VPCs, one may adopt tests as in Figure 2.

We also explored the use of compositionality heuristics to determine the senses of particles in VPCs, the general sense labels being (DIRECTION, ASPECTUAL, and IDIOMATIC). For example, if heuristic 3 applied to a VPC, the particle was assumed to have an ASPECTUAL sense. If heuristic 7 applied, the VPC was taken to be noncompositional and hence with an IDIOMATIC sense. In 30% of the cases, a correct sense label was assigned unambiguously. In another 32% of the cases, multiple sense labels were identified and one of them was correct. In the rest of the 19 cases where the sense label based on the application of specific heuristics was incorrect, the DIRECTION sense was the most misclassified sense (57.89%), followed by the COMPLETELY sense (31.58%), followed by the IDIOMATIC sense (10.53%).
<table><caption>Table 2: Sense classes for particles *up*, *down*, *away*, and *out* in compositional VPCs.</caption><thead><tr><th>Sense class<br>Sense type</th><th>Particle</th><th>Example</th></tr></thead><tbody><tr><td><strong>DIRECTION:</strong></td><td></td><td></td></tr><tr><td>DIRECTION-UP</td><td>up</td><td>He took the carpet up.</td></tr><tr><td>DIRECTION-DOWN</td><td>down</td><td>He skied down.</td></tr><tr><td>DIRECTION-AWAY</td><td>away</td><td>He ran away.</td></tr><tr><td>DIRECTION-OUT</td><td>out</td><td>He cried out.</td></tr><tr><td><strong>ASPECTUAL:</strong></td><td></td><td></td></tr><tr><td>COMPLETELY</td><td>up</td><td>Clean up the room!</td></tr><tr><td></td><td>down</td><td>London nightclub closed down over fights with knives and bottles.</td></tr><tr><td></td><td>out</td><td>He sorted out every scrap of manuscript, every map, and the native letters.</td></tr><tr><td><strong>ASPECTUAL:</strong></td><td></td><td></td></tr><tr><td>CONTINUING</td><td>away</td><td>Night and day they hammered away, coming on like great waves.<br>He scrubbed away at the floor.</td></tr><tr><td>IN-WORKING-ORDER-VAL</td><td>up</td><td>Bring the browser up!</td></tr><tr><td>NOT-IN-WORKING-ORDER-VAL</td><td>down</td><td>The computer went down again.</td></tr><tr><td>DISAPPEARANCE</td><td>out</td><td>The national electric grid went out.</td></tr><tr><td></td><td>away</td><td>The echo died away.<br>The music faded away.</td></tr></tbody></table>
<table><caption>Table 3: Generalizations about sense classes DIRECTION and ASPECTUAL for particles in compositional VPCs.</caption><thead><tr><th>Sense class:<br>Sense type</th><th>Generalizations</th></tr></thead><tbody><tr><td><strong>DIRECTION:</strong><br>DIRECTION-UP<br>DIRECTION-DOWN<br>DIRECTION-AWAY<br>DIRECTION-OUT</td><td>verb trajectory = +<br>Particle relative to some scale/domain<br>PP alternative is possible for the particles <i>up</i>, <i>down</i>, and <i>out</i>.<br>The particle <i>away</i> can take a PP-location as its GROUND argument.<br>The senses tend to appear with motion and movement-related verbs in the TRIPS ontology.<br>Cases pass two tests: (1) Is there a change in physical location/on a scale? (2) Does the entity exist anywhere?<br>DIRECTION-UP is also possible with verbs in the TRIPS ontology classes apply-force and acquire.<br>DIRECTION-DOWN is also possible with verbs in the TRIPS ontology classes hitting.<br>DIRECTION-AWAY is also possible with event-of-change verbs where grammatical resultative construction is used.<br>DIRECTION-OUT is also possible with verbs of vocalization (and the verbs in the TRIPS ontology classes locution and manner-say) and some perception verbs.</td></tr><tr><td><strong>ASPECTUAL:</strong><br>COMPLETELY</td><td>verb trajectory = -<br>PP alternative not possible<br>Default sense with the event-of-change verbs in the TRIPS ontology<br>COMPLETELY for the particle <i>up</i> is also possible with verbs in the TRIPS ontology classes protecting, joining, acquire-by-action, herd, and arrange-text.<br>COMPLETELY for the particle <i>down</i> is also possible with verbs in the TRIPS ontology classes change, consume, protecting, put, and pursue.<br>COMPLETELY for the particle <i>out</i> is also possible with verbs in the TRIPS ontology classes evoke-tiredness, arranging, and put.</td></tr><tr><td><strong>ASPECTUAL:</strong><br>CONTINUING</td><td>verb trajectory = -<br>Usually with atelic verbs that have extended duration<br>Iterative usage with semelfactive verbs<br>PP alternative is not possible, i.e., particle cannot take NP object, but can take PP object in conative constructions.<br>CONTINUING is also possible with verbs in the TRIPS ontology class event-of-action.</td></tr></tbody></table>
<table><caption>Table 4: Generalizations about senses IN-WORKING-ORDER-VAL, NOT-IN-WORKING-ORDER-VAL, and DISAPPEARANCE for particles in compositional VPCs. Note that NP* refers to the object argument of a transitive verb, e.g., *the carpet in he took the carpet up*, or the subject of an intransitive verb, e.g., *he in he skied down*.</caption><thead><tr><th>Sense class:<br>Sense type</th><th>Generalizations</th></tr></thead><tbody><tr><td>IN-WORKING-ORDER-VAL</td><td>NP* is phys-obj<br>object-function = provides-service-up-down<br>i.e., the entity is a device with some functionality.<br>PP alternative not possible<br>The verbs tend to be light/causal verbs.</td></tr><tr><td>NOT-IN-WORKING-ORDER-VAL</td><td>NP* is phys-obj<br>object-function = provides-service-up-down<br>i.e., the entity is a device with some functionality.<br>PP alternative not possible<br>NOT-IN-WORKING-ORDER-VAL is also possible for<br>the particle <i>down</i> with verbs in the TRIPS ontology<br>class event-of-undergoing-action.<br>The verbs tend to be light/causal verbs.</td></tr><tr><td>DISAPPEARANCE</td><td>verb trajectory = -<br>Usually with verbs of disappearance<br>PP alternative not possible<br>Cases fail two tests: (1) Is there a change in physical<br>location/on a scale? (2) Does the entity exist anywhere?<br>DISAPPEARANCE is also possible with verbs in the<br>TRIPS ontology classes event-of-undergoing-action<br>and change.</td></tr></tbody></table>
<figure><img src="image_15.png" /><figcaption>Figure 2: Tests to identify particle senses</figcaption></figure>

## 5 Computing semantics of sentences with VPCs

For the task of interpreting sentences with VPCs, we first need to determine if the VPC is compositional or not. We use the heuristics mentioned in §3.2 to determine the compositionality of a VPC. For the compositional cases, we get the senses for the verb and the particle from the TRIPS ontology and/or Word-Net. The senses for the particles as well as relevant linguistic generalizations to identify these senses are encoded in the TRIPS lexicon and ontology. In this section, we briefly discuss some of the generalizations encoded in the ontology and demonstrate the process of computing the semantics of sentences containing compositional VPCs using three example sentences involving VPCs with different senses for the particle *up*: *She cleaned up her room*, *She pushed the ball up*, and *The network came up*. The logical forms (LFs) produced for these sentences using the TRIPS parser, a broad coverage deep semantic parser driven by the TRIPS ontology, are presented in Figure 3 to Figure 5.
<figure><img src="image_16.png" /><figcaption>Figure 3: LF for the sentence: *She pushed the ball up.*</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 4: LF for the sentence: *She cleaned up her room.*</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Figure 5: LF for the sentence: *The network came up.*</figcaption></figure>

Particles in compositional VPCs can express the senses mentioned in Tables 2–4, and in Figure 2. This information is encoded in the TRIPS ontology by adding the ontology types corresponding to these senses in the particle’s lexicon. For example, the lexical entry for the particle *up* lists sense ontology types ONT::DIRECTION-UP, ONT::COMPLETELY, and ONT::IN-WORKING-ORDER-VAL, used in Figure 3 to Figure 5 respectively, among other possible senses.<fnref n="10" />

<footnote n="10">¹⁰For a better idea of what information the lexical entries and semantic/ontology classes carry in the TRIPS lexicon and ontology, see http://www.cs.rochester.edu/research/cisd/projects/trips/lexicon/browse-ont-lex-axiom.html.</footnote>
WordNet sense keys corresponding to the particle may be added in the ontology entries for these sense ontology types. For example, WordNet sense keys up%4:02:00 and up%4:02:05 are added in the entry for the sense ontology type ONT::DIRECTION-UP.

The senses that particles express in a VPC may depend on the verb type they combine with in the VPC. That is, a particle may convey the same sense when it appears with any of the verbs in a specific verb ontology class.<fnref n="11" /> For example, particles tend to get DIRECTION sense with verbs of motion and CONTINUING sense with atelic verbs that have extended durations (e.g., activity type verbs). With verb ontology types corresponding to continuous-change verbs (which appear under change verbs in the TRIPS ontology), particles tend to get the COMPLETELY sense. Specific verb ontology classes are also identified for specific particles, e.g., particle *down* exhibits COMPLETELY sense with the verbs in the TRIPS ontology class ONT::PURSUE, as can be seen in *The internet tracked down this guy's stolen car (...) and A motorist chased down, slapped, and threatened a boy (...)*.

In addition, we observed an interesting fact that the particles *up* and *out* seem to be in complementary distribution with respect to various verb ontology classes for the COMPLETELY sense. That is, for the COMPLETELY sense, either *up* or *out* is used, but not both with verbs from a specific verb ontology class.<fnref n="12" /> For example, with the verb ontology class ONT::ACQUIRE, *up* is used with COMPLETELY sense, *out* cannot be used with the verbs in this ontology class with the same sense. Note the COMPLETELY sense in the VPC *acquire up* in *Techstars has acquired up Global*, but we do not observe a VPC *acquire out* with the same sense. Similarly, with the verb ontology class ONT::EVOKE-TIREDNESS, *out* is used with the COMPLETELY sense, but *up* cannot be used. Note that we can say, *Someone's a bit tuckered out*, but not *tuckered up*.

Similarly there are generalizations observed for specific senses of particles corresponding to the semantic relation labels. For example, the verb takes a particle with an ASPECTUAL sense as its MANNER argument and the ASPECTUAL sense particle takes the verb as a FIGURE, as is observed in the LF in Figure 4. For the DIRECTION sense class particles, the verb assigns a RESULT argument

<footnote n="11">¹¹We find that different verb ontology types that were distinguished for other reasons in the TRIPS ontology (Allen et al. 2007) also line up with the particle distinctions.</footnote>

<footnote n="12">¹²This observation about the complementary distribution of usage between *up* and *out* may not be accidental. The Law of Differentiation (Paul 1890; Bréal 1900), and the Avoid Synonymy principle (Kiparsky 1983; Clark 1987) have been proposed in the lexico-semantic sphere which suggest that languages prefer not to have a given semantic slot be filled by two distinct lexical items.</footnote>
role instead of a MANNER role to the particle, as illustrated in Figure 3. The par-
ticle, on the other hand, assigns the FIGURE role to the AFFECTED entity as
in Figure 3. For the IN-WORKING-ORDER-VAL sense, the particle takes the AF-
FECTED or NEUTRAL entity as its FIGURE argument as is illustrated in Figure 5.
Thus, we see that different semantic relations may be involved in sentences with
different senses for the particles.

In order to get the correct semantic relations in different cases, we encode this
information in the ontology. The sense ontology type ONT::COMPLETELY, used
in Figure 4, for example, specifies for its FIGURE argument all the verb ontology
types with which a particle gets this sense. This information is presented in the
Generalizations column in Table 3. Note that under ONT::COMPLETELY, we list
all the verb ontology types with which we get the COMPLETELY sense irrespec-
tive of the specific particles used.<fnref n="13" /> Hence, ONT::COMPLETELY would specify
for its FIGURE argument the ontology types ONT::PURSUE, ONT::ACQUIRE as
well as ONT::EVOKE-TIREDNESS, for example.

Each of the verb ontology types mentioned above specifies which particles
can take a specific semantic relation label. For example, the verb ontology type
ONT::PURSUE would specify for its MANNER argument particle *down*, verb on-
tology type ONT::ACQUIRE would specify particle *up*, and verb ontology type
ONT::EVOKE-TIREDNESS would specify particle *out*.

For the DIRECTION senses, the particle could be replaced with a RESULT-Pre-
positional Phrase (RESULT-PP) in the sentence. For example, the particle *off* in
*The officer got off when he spotted an illegally parked car* can be replaced with a
RESULT-PP *off his motorcycle* as in *The officer got off his motorcycle when he
spotted an illegally parked car*.

Depending on the compliance or violations of constraints such as the ones
described above, the parser assigns scores for various parse options involving
various senses of the particle in a VPC. The parse with the highest score is se-
lected as the semantic representation of the sentence involving the VPC.

In *She cleaned up her room* (Figure 4), the verb *clean* (ONT::CLEAN which ap-
pears under ONT::CHANGE-STATE in the ontology) is not among the list of rel-
evant verb ontology types with which a DIRECTION-UP sense is licensed for the
particle *up*. Additionally, a restriction on the verb argument for the DIRECTION-
UP sense is that the argument have a semantic feature [+moveable] which is
also violated in the given sentence, since *the room* is generally not a moveable
entity. Hence, the parser assigns a low score to the parse which involves the

<footnote n="13">¹³Note that since the ontology is hierarchical, there is no need to list all the children ontology types as well if the parent ontology types are included.</footnote>
DIRECTION-UP sense for the particle *up* in this sentence. The verb *push*, how-
ever, in the sentence in Figure 3, is a movement-related verb and the affected
entity *ball* is a moveable object. Hence, the parser assigns a higher score to the
parse that involves the DIRECTION-UP sense for the particle *up* in this sentence.

The IN-WORKING-ORDER-VAL sense requires restrictions on the verbs that they take cognitive entities, devices or processes as their AFFECTED arguments which provide service, i.e., they have some functionality associated with them. The AFFECTED argument for the verb *clean*, namely *the room*, in Figure 4 does not satisfy this restriction. Similarly, The AFFECTED argument for the verb *push*, namely *the ball*, in Figure 3 does not satisfy this restriction. Hence, the parser assigns a low score for the parses involving an IN-WORKING-ORDER-VAL sense for the particle *up* in the two sentences. On the other hand, the entity *The network* in the sentence in Figure 5 satisfies the restriction and hence the sentence in Figure 5 gets a higher score for the parse that contains IN-WORKING-ORDER-VAL sense for the particle *up*.

The constraints for the COMPLETELY sense of the particle *up* are satisfied for the sentence in Figure 4. The verb *clean* is among the set of verbs in the ontology type (ONT::CHANGE-STATE) with which the relevant particle has been identified in the ontology to get this sense. Hence, the parser assigns a higher score to the parse for the sentence with the COMPLETELY sense for the particle *up*. The other two sentences get a lower score for a parse with the COMPLETELY sense for the particle *up*.

Thus, as mentioned above, the parse with the highest score is selected as a
semantic representation for each of the sentences involving the VPC. Hence, the
sentence in Figure 3 has the DIRECTION-UP sense for the particle *up*, the sen-
tence in Figure 4 has the COMPLETELY sense, and the sentence in Figure 5 has
the IN-WORKING-ORDER-VAL sense for the particle *up*.

**6 Conclusion**

Identification of MWEs, such as verb-particle constructions, as well as the com-
putation of their meaning is necessary for a broad coverage understanding. It is
not plausible to hand enumerate all the possible combinations, although Word-
Net is an admirable start. We have described an approach where the meaning
of a wide range of VPCs is computed compositionally, with the advantage that
VPCs not explicitly found in the lexicon can be both identified and semanti-
cally interpreted. To accomplish this, we identified the core senses of particles
that have broad application across verb classes. This information is used while
building computational lexica. We encoded the generalizations corresponding to various senses of particles in the TRIPS ontology and used them to identify these senses. We found that the generalizations corresponding to grammatical/semantic/ontological information help us identify appropriate senses of the particle. The procedure adopted enables compositional parsing by helping differentiate between particle senses, which is then used to obtain full semantic representations of sentences.

While we have outlined our approach to compute the semantics of sentences with VPCs in English using resources such as the TRIPS lexicon and ontology and WordNet, a similar approach can be adopted to incorporate more lexical or other linguistic resources to further improve semantic parsing of sentences involving VPCs in English. Also, such an approach can be adopted for semantic parsing of sentences involving other MWEs, as well as for developing semantic parsing capabilities for other languages using the resources available for those languages.

## Abbreviations

<table><tr><td>LF</td><td>logical form</td></tr><tr><td>LP-compositional</td><td>light particle compositional</td></tr><tr><td>LV-compositional</td><td>light verb compositional</td></tr><tr><td>MWE</td><td>multiword expression</td></tr><tr><td>RESULT-PP</td><td>RESULT-prepositional phrase</td></tr><tr><td>VPC</td><td>verb-particle construction</td></tr></table>

## References

Allen, James F., Myroslava Dzikovska, Mehdi Manshadi & Mary Swift. 2007. Deep linguistic processing for spoken dialogue systems. In *Proceedings of the ACL 2007 Workshop on Deep Linguistic Processing*, 49–56. Association for Computational Linguistics. June 28, 2007.

Allen, James F., Mary Swift & Will de Beaumont. 2008. Deep semantic analysis of text. In *Symposium on semantics in systems for text processing (STEP)*, 343–354.

Allen, James F. & Choh Man Teng. 2017. Broad coverage, domain-generic deep semantic parsing. In *Proceedings of the AAAI Spring Symposium, Computational Construction Grammar and Natural Language Understanding 2017*, 108–115.
Baldwin, Timothy, Colin James Bannard, Takaaki Tanaka & Dominic Widdows. 2003. An empirical model of multiword expression decomposability. In *Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment - Volume 18* (MWE '03), 89–96. Stroudsburg, PA, USA: Association for Computational Linguistics. DOI:10.3115/1119282.1119294

Bannard, Colin James. 2002. *Statistical techniques for automatically inferring the semantics of verb-particle constructions*. Tech. rep. University of Liverpool.

Bannard, Colin James & Timothy Baldwin. 2003. Distributional models of preposition semantics. In *Proceedings of the ACL-SIGSEM Workshop on the Linguistic Dimensions of Prepositions and their Use in Computational Linguistics Formalisms and Applications*, 169–180. Association for Computational Linguistics. http://lingo.stanford.edu/pubs/tbaldwin/sigsemprep2003-prepsem.pdf.

Bannard, Colin James, Timothy Baldwin & Alex Lascarides. 2003. A statistical approach to the semantics of verb-particles. In *Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment - Volume 18* (MWE '03), 65–72. Stroudsburg, PA, USA: Association for Computational Linguistics. DOI:10.3115/1119282.1119291

Bhatia, Archna, Choh Man Teng & James F. Allen. 2017. Compositionality in verb-particle constructions. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE '17), 139–148. Association for Computational Linguistics. http://www.aclweb.org/anthology/W17-1719.

Bréal, Michel. 1900. *Semantics*. Trans. by Mrs. H. Cust. New York: Henry Holt.

Chen, Ping. 1986. Discourse and particle movement in English. *Studies in Language* 10. 79–95.

Clark, Eve V. 1987. The principle of contrast. In Brian MacWhinney (ed.), *Mechanisms of language acquisition*, 1–33. New York: Academic Press.

Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. *Educational and Psychological Measurement* 20. 37–46.

Cook, Paul & Suzanne Stevenson. 2006. Classifying particle semantics in English verb-particle constructions. In *Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties* (MWE '06), 45–53. Association for Computational Linguistics. July 23, 2006.

Dehé, Nicole. 2002. *Particle verbs in English: Syntax, information structure and intonation*. Amsterdam: John Benjamins.

Fellbaum, Christiane. 1998. *WordNet: An electronic lexical database*. Cambridge, MA: MIT Press.

Fleiss, Jacob L. 1971. Measuring nominal scale agreement among many raters. *Psychological Bulletin* 76(5). 378–382.
Fraser, Bruce. 1976. *The verb-particle combination in English*. New York: Academic Press.

Ishizaki, Yasuaki. 2010. Some notes on the grammaticalization of *away*. In Hirozo Nakano, Masayuki Ohkado, Tomoyuki Tanaka, Tomohiro Yanagi & Azusa Yokogishi (eds.), *Synchronic and diachronic approaches to the study of language: A collection of papers dedicated to the memory of Professor Masachiyo Amano*, 71–83. Tokyo: Eichosha.

Ishizaki, Yasuaki. 2012. A usage-based analysis of phrasal verbs in early and late modern English. *English Language and Linguistics* 16(2). 241–260.
DOI:10.1017/S1360674312000020

Jackendoff, Ray. 1997. *The architecture of the language faculty*. Cambridge, MA: MIT Press.

Jackendoff, Ray. 2002. English particle constructions, the lexicon, and the autonomy of syntax. In Nicole Dehé, Ray Jackendoff, Andrew McIntyre & Silke Urban (eds.), *Verb-Particle Explorations* (Interface Explorations [IE]), 67–94. New York: Mouton de Gruyter.

Johnson, Christopher & Charles J. Fillmore. 2000. The FrameNet tagset for frame-semantic and syntactic coding of predicate-argument structure. In *Proceedings of the First Conference on North American Chapter of the Association for Computational Linguistics*, 56–62. Morgan Kaufmann Publishers Inc.

Kiparsky, Paul. 1983. Word-formation and the lexicon. In Frances Ingemman (ed.), *Proceedings of the 1982 Mid-America Linguistics Conference*, 47–78. University of Kansas, Dept. of Linguistics.

McCarthy, Diana, Bill Keller & John Carroll. 2003. Detecting a continuum of compositionality in phrasal verbs. In *Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment - Volume 18* (MWE ’03), 73–80. Stroudsburg, PA, USA: Association for Computational Linguistics.
DOI:10.3115/1119282.1119292

Miller, George A. 1995. WordNet: A lexical database for English. *Communications of the ACM* 38(11). 39–41.

Moreno-Ortiz, Antonio, Chantal Pérez-Hernández & M. ángeles Del-Olmo. 2013. Managing multiword expressions in a lexicon-based sentiment analysis system for Spanish. In *Proceedings of the 9th Workshop on Multiword Expressions* (MWE ’13), 1–10. Association for Computational Linguistics.

Niinuma, Fumikazu. 2014. Grammaticalization of the particle *away* in English: A cartographic approach. *Interdisciplinary Information Sciences* 20(2). 163–188.

O'Dowd, Elizabeth M. 1998. *Prepositions and particles in English: A discourse-functional account*. Oxford: Oxford University Press.
Patrick, Jon & Jeremy Fletcher. 2004. Differentiating types of verb particle constructions. In *Proceedings of Australasian Language Technology Workshop 2004 (ALTW 2004)*. http://aclweb.org/anthology-new/U/U04/U04-1022.pdf. December 8, 2004.

Paul, Hermann. 1890. *Principles of the history of language*. Trans. by H. A. Strong from the 2nd German Edition. Reprinted College Park, MD. 1970: McGrat.

Tyler, Andrea & Vyvyan Evans. 2003. *The semantics of English prepositions: Spatial scenes, embodied meaning, and cognition*. New York: Cambridge University Press.

Villavicencio, Aline. 2003. Verb-particle constructions and lexical resources. In *Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment (MWE 2003)- Volume 18*, 57–64. Association for Computational Linguistics. DOI:10.3115/1119282.1119290

Villavicencio, Aline. 2006. Verb-particle constructions in the World Wide Web. In Patrick Saint-Dizier (ed.), *Syntax and semantics of prepositions*, 115–130. The Netherlands: Springer.

Vincze, Veronika, István Nagy T. & Gábor Berend. 2011. Multiword expressions and named entities in the Wiki50 corpus. In *Proceedings of the International Conference Recent Advances in Natural Language Processing 2011*, 289–295. RANLP 2011 Organising Committee. http://aclweb.org/anthology/R11-1040.
Chapter 4

PARSEME multilingual corpus of verbal
multiword expressions

Agata Savary¹, Marie Candito², Verginica Barbu Mititelu³, Ed-
uard Bejček⁴, Fabienne Cap⁵, Slavomír Čéplö⁶, Silvio Ricardo
Cordeiro⁷, Gülşen Eryiğit⁸, Voula Giouli⁹, Maarten van Gom-
pel¹⁰, Yaakov HaCohen-Kerner¹¹, Jolanta Kovalevskaitė¹², Si-
mon Krek¹³, Chaya Liebeskind¹¹, Johanna Monti¹⁴, Carla Parra
Escartín¹⁵, Lonneke van der Plas⁶, Behrang QasemiZadeh¹⁶,
Carlos Ramisch⁷, Federico Sangati¹⁷, Ivelina Stoyanova¹⁸ &
Veronika Vincze¹⁹

¹Université de Tours (France), ²Université Paris Diderot (France), ³Romanian Academy Research Institute for Artificial Intelligence (Romania), ⁴Charles University (Czech Republic), ⁵Uppsala University (Sweden), ⁶University of Malta (Malta), ⁷Aix Marseille University (France), ⁸Istanbul Technical University (Turkey), ⁹Athena Research Center in Athens (Greece), ¹⁰Radboud University in Nijmegen (Netherlands), ¹¹Jerusalem College of Technology (Israel), ¹²Vytautas Magnus University in Kaunas (Lithuania), ¹³Jožef Stefan Institute in Ljubljana (Slovenia), ¹⁴"L'Orientale" University of Naples (Italy), ¹⁵ADAPT Centre, Dublin City University (Ireland), ¹⁶University of Düsseldorf (Germany), ¹⁷independent researcher (Italy), ¹⁸Bulgarian Academy of Sciences in Sofia (Bulgaria), ¹⁹University of Szeged (Hungary)

Multiword expressions (MWEs) are known as a “pain in the neck” due to their idiosyncratic behaviour. While some categories of MWEs have been largely studied, verbal MWEs (VMWEs) such as to take a walk, to break one’s heart or to turn off have been relatively rarely modelled. We describe an initiative meant to bring about substantial progress in understanding, modelling and processing VMWEs. In this joint effort carried out within a European research network we elaborated

<figure><img src="image_19.png" /></figure>
a universal terminology and annotation methodology for VMWEs. Its main out-
comes, available under open licenses, are unified annotation guidelines, and a cor-
pus of over 5.4 million words and 62 thousand annotated VMWEs in 18 languages.

# 1 Introduction

One of the basic ideas underlying linguistic modelling is compositionality (Bag-
gio et al. 2012), seen as a property of language items (Janssen 2001; Partee et
al. 1990) or of linguistic analyses (Kracht 2007). Counterexamples which chal-
lenge the compositionality principles (Pagin & Westerståhl 2001) include multi-
word expressions (MWEs) (Sag et al. 2002; Kim 2008), and notably verbal MWEs
(VMWEs), such as (1-4).<fnref n="1" />

(1) *Ida skriva glavo v pesek.* (SL)
Ida hide.3.SG head in sand
Ida hides her head in the sand. ‘Ida pretends not to see a problem.’

(2) *Er legt die Prüfung ab.* (DE)
he lay.3.SG the exam PART
He lays the exam PART. ‘He takes the exam.’

(3) *Η Ζωή παίρνει μία απόφαση.* (EL)
i zoi perni mia apofasi
the Zoe take.3.SG a decision
Zoe takes a decision. ‘Zoe makes a decision.’

(4) *Alina se face doctor.* (RO)
Alina REFL.3.SG make.3.SG doctor
Alina REFL makes doctor. ‘Alina becomes a doctor.’

VMWEs pose special challenges in natural language processing (NLP):

1. **SEMANTIC NON-COMPOSITIONALITY:** The meaning of many VMWEs cannot be deduced in a way deemed grammatically regular on the basis of their syntactic structure and of the meanings of their components. For instance, the meaning of sentence (1) cannot be retrieved from the meanings of its component words (SL) *glava* ‘head’ and *pesek* ‘sand’, except when very specific interpretations of these words and of their combination are admitted.

<footnote n="1">¹See the preface for the description of the conventions used to present multilingual examples.</footnote>
2. **LEXICAL AND GRAMMATICAL INFLEXIBILITY:** VMWEs are frequently subject to unpredictable lexical or syntactic constraints. For instance, when the individual lexemes in (EN) *to throw somebody to the lions* are replaced by their synonyms or the noun is modified by an adjective, the expression loses its idiomatic meaning:<fnref n="2" /> (EN) \*to fling sb to the lions, \*to throw sb to the hungry lions. Similarly, the predicative noun in the light-verb construction (EN) *she took a glance at the headline* cannot take a modifier denoting an agent, especially if different from the verb's subject (\*she took Paul's glance at the headline).

3. **REGULAR VARIABILITY:** Despite this inflexibility the VMWEs can still exhibit some regular variability, e.g.: (i) inflection or passivisation, as in (EN) *he was thrown to the lions*, (ii) a restricted lexical replacement and an adjectival modification of the predicative noun, as in (EN) *he took/had a quick glance at the headline*, (iii) omission of components without change in meaning, as in (EL) *meno me ti glika (sto stoma)* 'I stayed with the sweetness (in the mouth)' ⇒ 'I was very close to enjoy something desired but I failed to'.

4. **DISCONTINUITY:** The components of a VMWE may not be adjacent, e.g. (EN) *a mistake was frequently made, never turn it off*.

5. **CATEGORICAL AMBIGUITY:** VMWEs of different categories may share the same syntactic structure and lexical choices. For instance, (EN) *to make a mistake* and (EN) *to make a meal of something* 'to treat something as more serious than it really is' are combinations of the same verb with a direct object but the former is a light-verb construction (since the verb is semantically void and the noun keeps its original predicative meaning), while the latter is an idiom (since the noun loses its original sense).

6. **SYNTACTIC AMBIGUITY:** Occurrences of VMWEs in text may be syntactically ambiguous, e.g. (EN) *on* is a particle in *to take on the task* 'to agree to be in charge of the task', while it is a preposition in (EN) *to sit on the fence* 'not to take sides in a dispute'.

7. **LITERAL-IDIOMATIC AMBIGUITY:** A VMWE may have both an idiomatic and a literal reading. For instance the VMWE (EN) *to take the cake* 'to be the

<footnote n="2">²Henceforth, an asterisk (*) preceding a sentence will mean that the sentence is ungrammatical, while a dash (#) will signal a substantial change in meaning with respect to the original expression.</footnote>
most remarkable of its kind' is understood literally in (EN) *to take the cake out of the fridge*.

8. **NON-LITERAL TRANSLATABILITY:** Word-for-word translation of VMWEs is usually incorrect, e.g. (EN) **to take the cake** 'to be the most remarkable of its kind' does not translate to (FR) **prendre le gâteau** 'to take the cake'.

9. **CROSS-LANGUAGE DIVERGENCE:** VMWEs behave differently in different languages and are modelled according to different linguistic traditions. For instance, functional tokens, such as (EN) *off*, have a status of stand-alone words and can form verb-particle constructions in Germanic languages, e.g. (EN) *to turn off*. In Slavic languages, conversely, they function as prefixes, as in (PL) *wylączyć* 'PART.connect' ⇒ 'turn off', and are seen as inherent parts of verbal lexemes. Therefore, they cannot trigger MWE-related considerations (cf. §8). Also, the scope of light (or support) verb constructions may greatly vary from one linguistic tradition to another, e.g. depending on whether the copula *to be* is considered a light verb or not (cf. §9.1).

10. **WORDPLAY PRONENESS:** In particular contexts, VMWEs can be a subject of ad hoc creativity or a playful usage, as in (EN) *they want us to put the cat back inside the bag* 'they want us to pretend that the revealed secret remains unrevealed'.

Due to these unpredictable properties, the description, identification, analysis and translation of VMWEs require dedicated procedures. For example, due to 2 and 3, the description of VMWEs can be constrained neither to the level of the lexicon nor to the one of the syntax only. Challenge 4 hinders VMWE identification with traditional sequence labelling approaches and calls for syntactic analysis. Challenges 5, 6 and 7, however, mean that their identification and categorisation cannot be based on solely syntactic patterns. Challenges 1, 2, 7 and 8 constitute central issues in machine translation. Challenge 9 affects cross-lingual VMWE modelling. Finally, challenge 10 goes far beyond the state of the art in semantic modelling and processing of VMWEs.

A consistent linguistic and NLP terminology is required in order to better understand the nature of VMWEs, compare their properties across languages, hypothesise linguistic generalisations, model VMWEs according to common principles, develop cross-language VMWE identifiers and compare results obtained by different authors on different datasets. Such a consistency is, however, largely missing: different authors assign different names to the same phenomena or call different phenomena by the same name, be it from a linguistic or an NLP point
of view. This situation is similar to other areas of linguistic modelling, where
universalism-driven efforts have been undertaken – such as the Universal De-
pendencies (UD) project dedicated to standardising morphological and syntactic
annotations for dozens of languages (Nivre et al. 2016), or the normalisation of
uncertainty cue annotation across languages, genres and domains (Szarvas et al.
2012).

This chapter describes an initiative taken by the European PARSEME network,<fnref n="3" />
towards bringing about substantial progress in modelling and processing MWEs.
Its main outcomes include unified definitions and annotation guidelines for sev-
eral types of VMWEs, as well as a large multilingual openly available VMWE-
annotated corpus. Eighteen languages are addressed (note that the last 4 are non-
Indo-European):

* *Balto-Slavic:* Bulgarian (BG), Czech (CS), Lithuanian (LT), Polish (PL) and Slovene (SL);

• *Germanic:* German (DE) and Swedish (SV);

* *Romance:* French (FR), Italian (IT), Romanian (RO), Spanish (ES) and Portuguese (PT);<fnref n="4" />

* *Others:* Farsi (FA), Greek (EL), Hebrew (HE), Hungarian (HU), Maltese (MT) and Turkish (TR).

The corpus gave rise to the PARSEME shared task on automatic identification
of VMWEs, whose organisation and results are described by Savary et al. (2017).
See also Taslimipoor et al. (2018 [this volume]) and Maldonado & QasemiZadeh
(2018 [this volume]) who address the use of the PARSEME corpus in VMWE
identification and its evaluation, as well as Moreau et al. (2018 [this volume]),
Al Saied et al. (2018 [this volume]) and Simkó et al. (2018 [this volume]) who
describe 3 of the 7 systems participating in the shared task.

This chapter builds upon those sections of the PARSEME shared task descrip-
tion paper (Savary et al. 2017), presented in the MWE 2017 workshop, which
describe the corpus construction. Each of these sections has been substantially
extended, except the descriptions of the corpus format and inter-annotator agree-
ment, which required few additions and updates. Many new analyses and exam-
ples have been added, conclusions drawn from the PARSEME annotation cam-
paign have been addressed and the state of the art has been thoroughly revised.
As a result, the chapter is organised as follows. We give the definitions underly-

<footnote n="3">³http://www.parseme.eu</footnote>

<footnote n="4">⁴In this chapter we address the Brazilian dialect of Portuguese. All examples cited here are taken from this dialect.</footnote>
ing the scope of our work (§2), and the VMWE typology (§3). We describe the an-
notation principles, including the VMWE identification and categorisation tests,
and the deviations from the unified guidelines applied in some languages (§4).
We discuss the annotation methodology and tools (§5). We present the resulting
corpus and a cross-language quantitative analysis of some phenomena relevant
to challenges 1–10 (§6). We describe some language-specific studies based on the
corpus (§7) and discuss interesting problems which occurred during the project
(§8). We analyse the state of the art in MWE modelling and annotation, and com-
pare it to our approach (§9). We finally conclude and discuss future work (§10).

## 2 Definitions and scope

While the definition of a MWE inherently relies on the notion of a **WORD** (i.e. a
linguistically motivated unit), identification of VMWEs is performed on pragmat-
ically defined **TOKENS**. The relation between tokens and words can be threefold:

(1) A token coincides with a word, e.g. (MT) *ferh* ‘happiness’, (SV) *förvån-ting* ‘surprise’.

(2) Several tokens build up one **MULTIToken WORD** (MTW), if punctuation marks are considered token boundaries, as in (EN) *Pandora’s*, (PL) *SMS-ować* ‘to write an SMS’. Note that the latter example is not a VMWE as it contains only one word.

(3) One **MULTIWORD TOKEN** (MWT) contains several words, as in contractions, e.g. (IT) *della* ‘of.the’, or detachable pre-verbal particles, e.g. (DE) *ausmachen* ‘part.make’ ⇒ ‘to turn off’. Note that the latter example is a (one-token) VMWE. A MWT is not always a simple concatenation of words, e.g. (IT) *della* is a contraction of *di* ‘of’ and *la* ‘the.fem’.

In this work, **MULTIWORD EXPRESSIONS** (MWEs) are understood as (continuous
or discontinuous) sequences of words which:

* contain at least two component words which are **lexicalised** , i.e. always realised by the same lexemes (see below for a more precise definition), including a head word and at least one other syntactically related word,

* display some degree of lexical, morphological, syntactic and/or semantic idiosyncrasy, formalised by the annotation procedures in §4.1–§4.2.

This definition relatively closely follows the one by Baldwin & Kim (2010). Two
notable exceptions are that we impose syntactic constraints on the lexicalised
components (one of them must be the head word), and that Baldwin & Kim (2010)
include pragmatic and statistical idiosyncrasy in the set of the MWE definition
criteria. For us, conversely, **COLLOCATIONS**, i.e. word co-occurrences whose id-
iosyncrasy is of pragmatic or statistical nature only (e.g. *all aboard, the graphic
shows, drastically drop*) are disregarded.

Note that there is no agreement on the understanding of the border between
the scopes of MWEs and collocations. For Sag et al. (2002), collocations are any
statistically significant word co-occurrences, i.e. they include all forms of MWEs.
For Baldwin & Kim (2010), collocations form a proper subset of MWEs. Accord-
ing to Mel'čuk (2010), collocations are binary, semantically compositional com-
binations of words subject to lexical selection constraints, i.e. they intersect with
what is here understood as MWEs. This chapter puts forward yet another point
of view: MWEs and collocations are seen as disjoint sets of linguistic objects.

Our definition of a MWE is also relatively close to the notion of non-composi-
tional semantic phrasemes in Mel'čuk (2010), but we include light-verb construc-
tions in our scope. It is compatible as well with the one by Sag et al. (2002), where
a MWE is seen as an "idiomatic interpretation that crosses word boundaries". The
major differences between our approach and these seminal works are its multi-
lingual context and the fact that, within the restricted scope of verbal MWEs (see
below), we delimit the MWE phenomenon by a relatively precise and complete
MWE identification and categorisation procedure, given in the form of decision
trees built upon linguistic tests (§4). Note that this approach does not focus on
another salient property of MWEs which is their variable degree of idiosyncrasy
(Gross 1988), that is, the fact that various MWEs exhibit more or less unexpected
lexical, syntactic and semantic properties. A scale-wise modelling of MWEs is
hard to implement in the task of MWE annotation, which is our major opera-
tional objective. Instead, we assume that decisions on MWE-hood are binary,
and the decision trees are designed so as to make them reproducible.

VERBAL MWEs (VMWEs) are multiword expressions whose canonical form
(see below) is such that: (i) its syntactic head is a verb V, (ii) its other lexicalised
components form phrases directly dependent on V. Boundary cases for condi-
tion (i) include at least two types of VMWEs. Firstly, those with irregular syn-
tactic structures may hinder the identification of the headword as in (EN) **short-**
**circuited**, where the verb is atypically prefixed by an adjective. Secondly, for
those with two coordinated lexicalised verbs there is no consensus as to which
component – the conjunction or the first verb – should be considered the head,
as in (5). Condition (ii) requires that the lexicalised components of a VMWE form
a connected dependency graph. For instance, in (EN) **to take on the task** 'to agree
to be in charge of the task' the particle **on** directly depends on the verb, thus **take**
on fulfils the syntactic requirements to be a VMWE. Conversely, if the lexicalist hypothesis in syntax is followed (de Marneffe et al. 2014),<fnref n="5" /> the preposition on in (EN) *to rely on someone* does not directly depend on the verb, thus, *rely on* cannot be considered a VMWE.

(5) wo *man lebt und leben lässt* (DE)

where one lives and live lets

where one lives and lets live ‘where one is tolerant’

Just like a regular verb, the head verb of a VMWE may have a varying number of arguments. For instance, the direct object and the prepositional complement are compulsory in (EN) *to take someone by surprise*. Some components of such compulsory arguments may be *LEXICALISED*, that is, always realized by the same lexemes. Here, *by surprise* is lexicalised while *someone* is not.

Note that lexicalisation is traditionally defined as a diachronic process by which a word or a phrase acquires the status of an autonomous lexical unit, that is, “a form which it could not have if it had arisen by the application of productive rules” (Bauer 1983 apud Lipka et al. 2004). In this sense all expressions considered VMWEs in this work are lexicalized. Our notion of lexicalisation extends this standard terminology, as it applies not only to VMWEs but to their components as well. The reason is that, in the context of the annotation task, we are in need of specifying the precise span of a VMWE, i.e. pointing at those words which are considered its inherent, lexically fixed components. Precisely these components are referred to as lexicalized within the given VMWE. Throughout this chapter, the lexicalised components of VMWEs are highlighted in bold.

A prominent feature of VMWEs is their rich morpho-syntactic variability. For instance, the VMWE (EN) *to take someone by surprise* can be inflected (*they took him by surprise*), negated (*they did not take him by surprise*), passivised (*he will be taken by surprise*), subject to extraction (*the surprise by which I was taken*), etc. Neutralizing this variation is needed when applying the linguistic tests defined in the annotation guidelines (§4), which are driven by the syntactic structure of the VMWE candidates. We define a *PROTOTYPICAL VERBAL PHRASE* as a minimal sentence in which the head verb *V* occurs in a finite non-negated form and all its arguments are in singular and realized with no extraction. For instance, (EN) *Paul made/makes a pie* is a prototypical verbal phrase while *Paul did not make a pie, the pie which Paul made and the pie was made by Paul* are

<footnote n="5">⁵The lexicalist hypothesis strongly inspired the PARSEME annotation guidelines, and is expected to be even more thoroughly followed in the future versions of the corpus.</footnote>
not. If a VMWE can occur as a prototypical verbal phrase while keeping its id-
iomatic meaning, then such a phrase is its **CANONICAL FORM**. Otherwise, its least
marked variation is considered canonical (a non-negated form is less marked
than a negated one, active voice is less marked then passive, and a form with an
extraction is more marked than one without it). For instance, a canonical form
of (EN) *a bunch of decisions* which were *made by him* is (EN) *he made a de-
cision*. But since (6) and (7) lose their idiomatic readings in active voice – (PL)
#wszyscy rzucili kości ‘everyone threw dies’ – and with no negation – (BG) #tya
iska i da chue ‘she wants to also hear’ – their canonical forms are passive and
negated, respectively. Whenever a VMWE candidate is identified in a sentence,
the linguistic tests are to be applied to one of its canonical forms (whether it is a
prototypical verbal phrase or not).

(6) **Kości zostały rzucone.** (PL)
dies were cast
The dies were cast. 'The point of no-return has been passed.'

(7) **Тя не иска и да чуе.** (BG)
Tya ne iska i da chue
she not want and to hear
She does not even want to hear. 'She opposes strongly.'

(8) **Пиле не може да прехвръкне.** (BG)
pile ne mozhe da prehvrakne
Bird not can to PART.fly
A bird cannot fly across something. 'Something is very strictly guarded.'

Throughout this chapter examples of VMWEs will always be given in their canonical forms, possibly accompanied by adjuncts, if the subject is lexicalised as in (8). Otherwise, their canonical forms may alternate – for brevity – with infinitive forms, or – rarely – with other variants when particular phenomena are to be illustrated.

MWEs containing verbs but not functioning as verbal phrases or sentences are excluded from the scope of annotation, e.g. (FR) *peut-être* ‘may-be’ ⇒ ‘maybe’, *porte-feuille* ‘carry-sheet’ ⇒ ‘wallet’.

Let us finally comment on the notion of universalism. Formally, this term
should only be used when a property or a phenomenon has been proven rele-
vant to all languages, which is practically out of range of any endeavour, however
multilingual and inclusive. Therefore, in this chapter we use the adjective ‘uni-
versal’ in the sense of a scientific hypothesis rather than of a proven fact. When
we speak about a universal category or property, it is to be understood that we deem them universal, based on the evidence from the languages currently in our scope. Since our framework is meant to continually evolve by including new languages and MWE types, we hope our definitions and findings to approximate the truly universal properties increasingly well.

## 3 VMWE typology

The typology of VMWEs, as well as linguistic tests enabling their classification, were designed so as to represent properties deemed universal in a homogeneous way, while rendering language-specific categories and features at the same time. The 3-level typology consists of:

1. Universal categories, valid for all languages participating in the task:

a) light-verb constructions (LVCs), as in (9):

(9) *Eles deram uma caminhada.* (PT)
they gave a walk
They gave a walk. 'They took a walk.'

b) idioms (ID), as in (10):

(10) ast **dide khab** man **baraye kafi** qadre be
is seen sleep me for enough quantity to
He had enough sleep for me. 'He has many plans for me.' (FA)

2. Quasi-universal categories, valid for some language groups or languages, but not all:

a) inherently reflexive verbs (IReflVs), as in (11):

(11) *Ils ne s'apercevront de rien.* (FR)
they not REFL.3.PL'perceive.3.PL.FUT of nothing
They will REFL-perceive nothing. 'They will not realise anything.'

b) verb-particle constructions (VPCs), as in (12):

(12) *Sie macht die Tür auf.* (DE)
she makes the door PART
She makes PART the door. 'She opens the door.'
3. Other verbal MWEs (OTH), not belonging to any of the categories above (due to not having a unique verbal head) e.g. (EN) *he never drinks and drives, she voice acted, the radio short-circuited*.

<table><caption>Table 1: Examples of various categories of VMWEs in four non-Indo-European languages.</caption><thead><tr><th>Lang.</th><th>ID</th><th>LVC</th><th>Quasi-universal / OTH</th></tr></thead><tbody><tr><td>HE</td><td><b>אבד</b><br>'Kelax is lost on him.'<br>'He is outdated.'</td><td><b>הגיע</b><br>'to come to a conclusion'<br>'to conclude'</td><td><b>לא</b><br>'the bashful does not learn'<br>'one should dare ask questions'</td></tr><tr><td>HU</td><td><b>kinyír</b><br>'to out-cut'<br>'to kill'</td><td><b>szabályozást ad</b><br>'to give control'<br>'to regulate'</td><td><b>feltüntet</b> (VPC)<br>'to PART-strike'<br>'to mark'</td></tr><tr><td>MT</td><td><b>Ghasfur żghir qalli.</b><br>'A small bird told me.'<br>'I learned it informally.'</td><td><b>ha deċizjoni</b><br>'to take a decision'<br>'to make a decision'</td><td><b>iqum u joqghod</b> (OTH)<br>'to jump and stay'<br>'to fidget'</td></tr><tr><td>TR</td><td><b>yüzüstü bırakmak</b><br>'to leave (sb) face down'<br>'to forsake'</td><td><b>engel olmak</b><br>'to become obstacle'<br>'to prevent'</td><td><b>karar vermek</b> (OTH)<br>'to give a decision'<br>'to make a decision'</td></tr></tbody></table>

While we allowed for language-specific categories, none emerged so far. Table 1 and Table 2 show examples of VMWEs of different categories in the 18 languages in our scope (4 non-Indo-European and 14 Indo-European). None of those languages seems to possess VMWEs of all 5 terminal categories (LVC, ID, IReflV, VPC and OTH).

We thoroughly considered introducing another universal category of inherently prepositional verbs (IPrepVs), such as (EN) *to rely on, to refer to, or to come across*. However, the IPrepV-related linguistic tests used in the pilot annotation proved not sufficiently reliable to distinguish such expressions from compositional verb-preposition combinations, such as (EN) *to give something to someone*. Therefore, we abandoned this category, considering that prepositions belong to the area of verb valency and should be handled by a regular grammar (combined with a valency lexicon). Reconsidering this category experimentally belongs to future work (§10).
<table><caption>Table 2: Examples of various categories of VMWEs in 14 Indo-European languages.</caption><thead><tr><th>Lang.</th><th>ID</th><th>LVC</th><th>Quasi-universal / OTH</th></tr></thead><tbody><tr><td>BG</td><td>бълвам змии и гущери<br>'to spew snakes and lizards'<br>'to shower abuse'</td><td>държа под контрол<br>'to keep under control'<br>'to keep under control'</td><td>усмихвам се (IReflV)<br>'to smile REFL'<br>'to smile'</td></tr><tr><td>CS</td><td>házet klacky pod nohy<br>'to throw sticks under feet'<br>'to put obstacles in one's way'</td><td>vyslovovat nesouhlas<br>'to voice disagreement'<br>'to disagree'</td><td>chovat se (IReflV)<br>'to keep REFL'<br>'to behave'</td></tr><tr><td>DE</td><td>schwarz fahren<br>'to drive black'<br>'to take a ride without a ticket'</td><td>eine Rede halten<br>'a hold a speech'<br>'to give a speech'</td><td>sich enthalten (IReflV)<br>'to contain REFL'<br>'to abstain'</td></tr><tr><td>EL</td><td>χάνω τα αυγά και τα καλάθια<br>'to lose the eggs and the baskets'<br>'to be at a complete and utter loss'</td><td>κάνω μία πρόταση<br>'to make a proposal'<br>'to propose'</td><td>μπαίνω μέσα (VPC)<br>'to get PART'<br>'to go bankrupt'</td></tr><tr><td>ES</td><td>hacer de tripas corazón<br>'to make heart of intestines'<br>'to pluck up the courage'</td><td>hacer una foto<br>'to make a picture'<br>'to take a picture'</td><td>coser y cantar (OTH)<br>'to sew and to sing'<br>'as easy as pie'</td></tr><tr><td>FA</td><td>دست گل به آن دادن<br>'to give a flower bouquet to water'<br>'to mess up, to do sth. wrong'</td><td>امتحان کردند<br>'to do an exam'<br>'to test'</td><td>به خود آمده<br>'to come to REFL'<br>'to gain focus'</td></tr><tr><td>FR</td><td>voir le jour<br>'to see the daylight'<br>'to be born'</td><td>avoir du courage<br>'to have courage'<br>'to have courage'</td><td>se suicider (IReflV)<br>'to suicide REFL'<br>'to commit suicide'</td></tr><tr><td>IT</td><td>entrare in vigore<br>'to enter into force'<br>'to come into effect'</td><td>fare un discorso<br>'to make a speech'<br>'to give a speech'</td><td>buttare giù (VPC)<br>'to throw PART'<br>'to swallow'</td></tr><tr><td>LT</td><td>pramušti dugną<br>'to break the bottom'<br>'to collapse'</td><td>priimti sprendimą<br>'to take on a decision'<br>'to make a decision'</td><td></td></tr><tr><td>PL</td><td>rzucać grochem o ścianę<br>'to throw peas against a wall'<br>'to try to convince somebody in vain'</td><td>odnieść sukces<br>'to carry-away a success'<br>'to be successful'</td><td>bać się (IReflV)<br>'to fear REFL'<br>'to be afraid'</td></tr><tr><td>PT</td><td>fazer das tripas coração<br>'make the tripes into heart'<br>'to try everything possible'</td><td>fazer uma promessa<br>'to make a promise'<br>'to make a promise'</td><td>se queixar (IReflV)<br>'to complain REFL'<br>'to complain'</td></tr><tr><td>RO</td><td>a trage pe sfoară<br>'to pull on rope'<br>'to fool'</td><td>a face o vizită<br>'to make a visit'<br>'to pay a visit'</td><td>a se gândi (IReflV)<br>'to think REFL'<br>'to think'</td></tr><tr><td>SL</td><td>spati kot ubit<br>'to sleep like killed'<br>'to sleep soundly'</td><td>postaviti vprašanje<br>'to put a question'<br>'to ask a question'</td><td>bati se (IReflV)<br>'to fear REFL'<br>'to be afraid'</td></tr><tr><td>SV</td><td>att plocka russinen ur kakan<br>'to pick raisins out of the cake'<br>'to choose only the best things'</td><td>ta ett beslut<br>'to take a decision'<br>'to make a decision'</td><td>det knallar och går (OTH)<br>'it trots and walks'<br>'it is OK/as usual'</td></tr></tbody></table>
# 4 Annotation guidelines

Given the definitions in §2 and a text to annotate, each iteration of the annotation process starts with: (i) selecting a candidate sequence, i.e. a combination of a verb with at least one other word which could form a VMWE, (ii) establishing the precise list of its lexicalised components and its canonical forms. These steps are largely based on the annotator's linguistic knowledge and intuition.

Once a candidate sequence has been selected, its status as a VMWE is tested in two steps: identification and categorisation. Each step is based on linguistic tests and examples in many languages, organised into decision trees, so as to maximise the determinism in decision making.

## 4.1 Identification tests

Five generic non-compositionality tests were defined in order to identify a VMWE (of any category):

**Test 1 [CRAN]**: Presence of a cranberry word, e.g. (EN) *it goes astray*;

**Test 2 [LEX]**: Lexical inflexibility, e.g. (EN) *they #allowed the feline out of the container (they let the cat out of the bag)*; *to give a stare (to give a look);

**Test 3 [MORPH]**: Morphological inflexibility, e.g. (EN) *to #take a turn (to take turns)*;

**Test 4 [MORPHOSYNT]**: Morpho-syntactic inflexibility, e.g. (EN) #I give you his word for that (*I give you my word for that*);

**Test 5 [SYNT]**: Syntactic inflexibility, e.g. (EN) #Bananas are gone (*he went bananas*).

If none of these tests apply, an additional hypothesis covers the LVC candidates, which usually fail Tests 1 and 3–5 and for which Test 2 is hard to apply due to their relatively high, although restricted, productivity.

[LVC hypothesis]: In a verb+(prep)+noun candidate the verb is a pure syntactic operator and the noun expresses an activity or a state, e.g. (EN) *makes a speech*.
Passing any of Tests 1–5 is sufficient for a candidate sequence to be identified as a VMWE, while the LVC hypothesis has to be confirmed by the LVC-specific tests.<fnref n="6" />

## 4.2 Decision tree for categorisation

Once a VMWE has been identified or hypothesised following the tests in the preceding section, its categorisation follows the decision tree shown in Figure 1. Tests 6–8 are structural, the others are category-specific.

<figure><img src="image_20.png" /><figcaption>Figure 1: Decision tree for VMWE categorisation.</figcaption></figure>

### 4.2.1 Structural tests

Categorisation of a VMWE depends on the syntactic structure of its canonical form determined by the following three tests:

**Test 6 [HEAD]:** Presence of a unique verb functioning as the syntactic head of the whole expression, like in (13) and unlike in (14).

<footnote n="6">⁶As explained in §10, feedback from the large-scale annotation of version 1.0 of the corpus led us to questioning the correctness of the two-stage VMWE annotation. In edition 1.1 we transformed the identification tests into ID-specific tests and performed VMWE identification simultaneously to their categorisation.</footnote>
(13) *Je laisse tomber.* (FR)
I let fall
I let fall. 'I let go, I abandon.'

(14) *wo man lebt und leben lässt* (DE)
where one lives and live lets
where one lives and lets live 'where one is tolerant'

Test 7 [IDEP]: Among the phrases dependent on the head verb exactly one contains lexicalised components, as in (EN) *made it up*, and unlike in (EN) *made up her mind*.

Test 8 [CATEG]: Morphosyntactic category of the verb's dependent. Contrary to most other tests, the result of this test is not binary but taken from a closed list of values: (i) reflexive clitic (REFL), as in (15), (ii) particle (PART), as in (16); (iii) nominal or prepositional phrase, as in (17); (iv) other (including a verb, an adverb, a non-reflexive pronoun, etc.), as in (18).

(15) *Тоў се страхува.* (BG)
toy se strahuva
he REFL fears
He fears REFL. 'He is afraid.'

(16) *Der Film fängt an.* (DE)
the film catches PART
The film catches PART. 'The film begins.'

(17) *Mój bratanek buja w oblokach.* (PL)
my nephew swings in clouds
My nephew swings in the clouds. 'My nephew fantasizes.'

(18) *Uma ajudinha cai muito bem.* (PT)
a help.DIM falls very well
A little help falls very well. 'A little help comes at the right moment.'

When a VMWE fails Test 6 or 7, it is automatically classified as OTH and ID, respectively. This means that we do not allow cumulative categories. For instance,
in (20) the reflexive clitic considerably changes the meaning of the base VPC from (19), which might qualify the whole as an IReflV. However, due to the presence of two lexicalised syntactic arguments of the verb, such cases are necessarily classified as IDs (here: with a nested VPC).

(19) *Er stellte mir seine Freundin vor.*
he put me his friend PART
He put his friend PART to me. ‘He presented his friend to me.’
(DE)

(20) *Er stellte sich die Reise vor.*
he put REFL.3.SG the travel PART
He put the travel PART to REFL. ‘He imagined the travel.’
(DE)

Test 8, with return values (i)-(iii), triggers the category-specific tests for IReflVs, VPCs and LVCs, respectively. For other categories the candidate automatically qualifies as an ID.

### 4.2.2 Light-verb constructions

Light-verb constructions (LVCs) gave rise to a vast literature since first introduced by Jespersen (1965), possibly because there is no consensus on their exact definition and scope. We consider a candidate sequence an LVC if it consists of a verb *V* and a nominal complement *N*, possibly introduced by a preposition, provided that it passes all of the following tests:

Test 9 [N-EVENT]: *N* denotes an event or a state, as in (21);

(21) Οι συσκευές έχουν τη δυνατότητα σύνδεσης.
I siskieves exun ti δινατοτιδα sinδesis
the devices have the ability connection.SG.GE.
The devices have the ability to connect. ‘The devices can connect.’
(EL)

Test 10 [N-SEM]: *N* has one of its original senses, as in (22) and unlike in (23);

(22) Steffi rend visite à Monica.
Steffi returns visit to Monica
Steffi returns a visit to Monica. ‘Steffi pays a visit to Monica.’
(FR)
(23) *Je jette l'éponge.* (FR)
I throw the'sponge
I throw the sponge. 'I give up.'

Test 11 [V-LIGHT]: *V* only contributes morphological features (tense, mod, person, number, etc.) but adds no semantics that is not already present in *N*, other than the semantic role of *V*'s subject with respect to *N*, as in (24);

(24) *Gydytojai padarė išvadą, kad gijimo procesas vyksta*
Doctors made conclusion, that recovery process happens
*sėkmingai.* (LT)
successfully.
The doctors made the conclusion that the recovery process is
successful. 'The doctors came to the conclusion that the recovery
process is successful.'

Test 12 [V-REDUC]: An NP headed by *N* can be formed containing all of *V*'s syntactic arguments, and denoting the same event or state as the LVC, e.g. (EN) *Paul had a nice walk* denotes the same event as (EN) *the nice walk of Paul*.

Test 13 [N-PROHIBIT-ARG]: A semantic argument of the same type can-
not be syntactically realised twice – both for *N* and for *V*, e.g. (EN) *Paul
made the decision of the committee* is meaningless, while (EN) *Paul leads
the discussion of the committee* is acceptable. Therefore, *to lead a discussion
is not an LVC.*

Tests 12 and 13 are syntactic tests approximating the property that one of *V*'s syntactic arguments (generally its subject) is *N*'s semantic argument.

Note that our definition of an LVC does not fully overlap with the state of the
art. On the one hand, we are more restrictive than some approaches in that we
do not include cases in which the verb does add some (even bleached) semantics
to the noun. For instance, inchoative verbs combined with non-inchoative nouns
such as (PL) *objąć patronat* ‘to embrace patronage’ ⇒ ‘to take on patronage’ fail
Test 11 and are therefore not classified as LVCs, although their fully bleached
counterparts are, as (PL) *sprawować patronat* ‘to perform patronage’ ⇒ ‘to
dispense patronage’. On the other hand, we include in LVCs those combinations
in which a semantically void verb selects a large class of action/state nouns so that its lexical non-compositionality is hard to establish, e.g. (FR) *commettre un crime/délit/meurtre/*... ‘to commit a crime/offence/murder/...’.

The latter reason makes LVCs belong to the grey area of (non-)compositionality. They are mostly morphologically and syntactically regular. They can also be seen as semantically compositional in the sense that the semantically void light verb is simply omitted in the semantic calculus. However, this omission may itself be seen as an irregular property. This confirms the observation of Kracht (2007) that compositionality is a property of linguistic analyses rather than of language items.

### 4.2.3 Idioms

A verbal idiomatic expression (ID) comprises a head verb *V* (possibly phrasal) and at least one of its arguments. Following the decision tree from Figure 1, a VMWE is classified as an ID in one of the 3 cases:

1. *V* has more than one lexicalised argument, as in (25) and (26)

(25) *Srce* *mu* *je padlo v hlače*. (SL)
heart him is fallen in pants
His heart fell into his pants. ‘He lost courage.’

(26) جانم به لیم رسید. resid labam be janam (FA)
arrived lips-my to soul-my
My soul arrived at my lips. ‘I am frustrated.’

2. *V*'s single lexicalised argument is of any category other than a reflexive clitic, a particle or a nominal phrase (possibly introduced by a preposition), as in (27), (28) and (29);

(27) Platforma *dopięła* *swego*. (PL)
Platform PART-buttoned OWN
The Platform buttoned PART her own. ‘The Platform fulfilled its plans.’
(28) *Es gibt kein Zurück.* (DE)
it gives no back
It gives no retreat. 'There is no retreat.'

(29) *Ele sabe onde pisar.* (PT)
he knows where step
He knows where to step. 'He knows how to succeed.'

3. V's single lexicalised argument is a nominal phrase (possibly introduced by a preposition), at least one of the LVC-specific Tests 9–13 fails but at least one of the identification Tests 1–5 applies, as in (30).

(30) *Artık kimsenin aklına gelmeyecek.* (TR)
anymore of-anyone to-his-mind it-will-not-come
It will not come to the mind of anyone anymore. 'No one will remember it anymore.'

Distinguishing an ID from an LVC in case 3 is one of the hardest and most frequent annotation challenges. In case 1, care must be taken to identify and also annotate nested VMWEs (if any), e.g. the VMWE in (31) contains a nested ID (RO) *dă pe față* 'gives on face' ⇒ 'reveals'.

(31) *El dă cărțile pe față.* (RO)
he gives cards on face
He gives the cards on the face. 'He reveals his intentions.'

Idioms whose head verb is the copula (*to be*) pose special challenges because their complements may be (nominal, adjectival, etc.) MWEs themselves. In this task, we consider constructions with a copula to be VMWEs only if the complement does not retain the idiomatic meaning when used without the verb. For instance, (PL) *on jest jedną nogą na tamtym świecie* 'he is with one leg in the other world' ⇒ 'he is close to death' is an ID because (PL) *jedna noga na tamtym świecie* 'one leg in the other world' loses the idiomatic meaning, while (PL) to stwierdzenie *jest do rzeczy* 'this statement is to the thing' ⇒ 'this statement is relevant' is not a VMWE since (PL) *do rzeczy* 'to the thing' ⇒ 'relevant' keeps the idiomatic reading.
### 4.2.4 Inherently reflexive verbs

Pronominal verbs, sometimes also called reflexive verbs, are formed by a verb combined with a reflexive clitic (REFL). They are very common in Romance and Slavic languages, and occur in some Germanic languages such as German and Swedish. Clitics can be highly polysemous and sometimes have an idiomatic rather than a reflexive meaning, in which case we call them inherently reflexive verbs (IReflVs). To distinguish regular from idiomatic uses of reflexive clitics, we rely on an IReflV-specific decision tree<fnref n="7" /> containing 8 tests, which are meant to capture an idiosyncratic relation between a verb with a reflexive clitic and the same verb alone. The first 3 of these tests are sufficient to identify most of the actual IReflVs:

**Test 14** [INHERENT]: V never occurs without C, as in (32);

(32) *Jonas har försovit sig* idag. (SV)
Jonas has overslept REFL.3.SG today
Jonas overslept REFL today. ‘Jonas overslept today.’

**Test 15** [DIFF-SENSE]: C markedly changes the meaning of V, as in (33);

(33) *kar se tiče* Kosova (SL)
what REFL touches Kosovo
what REFL touches Kosovo ‘as far as Kosovo is concerned’

**Test 16** [DIFF-SUBCAT]: C changes the subcategorisation frame of V, as in (34) vs. (PT) *você me esqueceu* ‘you forgot me’.

(34) *Você se esqueceu de mim.* (PT)
you REFL.3.SG forgot of me
You forgot REFL about me. ‘You forgot about me.’

IReflVs are hard to annotate because pronominal clitics have several different uses. For example, (IT) *si ‘REFL’* can occur not only in IReflVs such as (IT) *riferirsi* ‘to report.REFL’ ⇒ ‘to refer’, but also in the following non-idiomatic cases: reflexive (IT) *lavarsi* ‘to wash.REFL’, possessive reflexive (IT) *grattarsi la testa* ‘to scratch.REFL head’ ⇒ ‘to scratch one’s head’, reciprocal (IT) *baciarsi* ‘to

<footnote n="7">⁷http://parsemefr.lif.univ-mrs.fr/parseme-st-guidelines/1.0/?page=ireflv</footnote>
kiss.REFL' ⇒ 'to kiss each other', impersonal (IT) *si dorme molto* 'REFL sleeps much' ⇒ 'people sleep a lot', middle alternation (IT) *si affittano case* 'REFL rent houses' ⇒ 'houses are rented' or inchoative (IT) *la porta si apre* 'the door REFL opens' ⇒ 'the door opens'. The IRefV category was reported as the most challenging to annotate by some teams, notably the Spanish and the Romanian ones.

### 4.2.5 Verb-particle constructions

Verb-particle constructions (VPCs) are pervasive notably in Germanic languages and Hungarian, but virtually non-existent in Romance or Slavic languages. They are formed by a lexicalised head verb *V* and a lexicalised particle *P* dependent on *V*, whose joint meaning is non-compositional. The latter property is approximated by a unique syntactic test:

> Test 22 [V+PART-DIFF-SENSE] A sentence without *P* does not refer to the same event/state as the sentence with *P*. For example, the sentence in (35) does not imply (HU) *nekem jött ez a koktél* 'this cocktail bumped into me', while (DE) *er legt das Buch auf dem Tisch ab* 'he puts the book on the table *PART*' implies (DE) *er legt das Buch auf dem Tisch* 'he puts the book on the table'.

(35) **Be-jött** ez a koktél nekem. (HU)
PART-bumped this the cocktail for.me
This cocktail bumped PART into me. 'I like this cocktail.'

The first challenge in identifying a VPC is to distinguish a particle, as in (EN) *to get up a party*, from a homographic preposition, as in (EN) *to get up the hill*. Language-specific tests were designed for German and English to this aim.

In some Germanic languages and also in Hungarian, verb-particle constructions can be spelled either as one (multiword) token, as in (36), or separated, as in (37). Both types of occurrences are to be annotated.

(36) ő **be-rúgott**. (HU)
he PART-kicked
He kicked PART. 'He got drunk.'

(37) Nem ő **rúgott be**. (HU)
not he kicked PART
He did not kick PART. 'He did not get drunk.'
Special care must be taken with polysemous constructions having both a com-
positional and a non-compositional reading, as in (DE) *ein Schild aufstellen* ‘to
put up a sign’ vs. (DE) *einen Plan aufstellen* ‘to put up a plan’ ⇒ ‘to draw up a
plan’.

4.2.6 Other VMWEs

This category gathers the VMWEs which do not have a single verbal head (cf.
Test 6 in Figure 1 and §4.2.1). Those include:

• Coordinations like in example (14) p. 101, or (38)

(38) בריטניה נשאה ונתנה עם מצרים. micrayim 'im ve-natna nas'a britanya Egypt with and-gave carried Britain Britain carried and gave with Egypt. 'Britain negotiated with Egypt.' (HE)

(39) On court-circuite le réseau terrestre. one short-circuits the network terrestrial One short-circuits the terrestrial network. 'One bypasses the terrestrial network.' (FR)

• Compound verbs, resulting usually from conversion of nominal compo-
unds, and therefore having no regular verbal structure, as in (39) or in
(EN) *to pretty-print*.

(39) *On court-circuite le réseau terrestre.* (FR)
    one short-circuits the network terrestrial

    One short-circuits the terrestrial network. 'One bypasses the
    terrestrial network.'

4.3 Language-specific interpretation of the guidelines

Despite huge efforts put into setting up generic terminologies and methodolo-
gies, as well as into the pilot annotations and the project coordination, language-
specific interpretation of the final guidelines could not be avoided. This was
mainly due to different linguistic sensitivities and traditions, language-specific
challenges and incompleteness or imprecision of the guidelines.

The most notable deviation occurred in Farsi, where no categorisation was per-
formed, and the OTH label was used for all identified VMWEs instead. The main
reason is the particularly challenging nature of the VMWE phenomenon in this
language. There are less than 200 actively used simple (single-word) verbs, and
a large majority of events and processes are expressed by multiword combina-
tions, many of which are potential VMWEs. The implications on our annotation
process are at least threefold. Firstly, verbs are extremely polysemous, so Test 11
(§4.2.2) is very difficult to apply. In particular, the highly frequent light verb
*کرد /kardan/ ‘to do/make’ is ambiguous in its passive form شد/šodan/ ‘done/made’
with the semi-copula equivalent roughly to ‘become’. Only the former interpre-
tation should yield a VMWE annotation but the difference is hard to capture.
Secondly, rephrasing an LVC by a single verb, often used to approximate Test
9 in other languages (*to make a decision* = *to decide*), is rarely feasible in Farsi.
Thirdly, VMWEs are extremely pervasive, which is easily visible in Table 3: the
number of annotated VMWEs is roughly the same as the number of sentences,
i.e. almost every main verb is the head of a VMWE. As a result, the VMWE phe-
nomenon is particularly hard to capture in Farsi since it can rarely be contrasted
with verbal constructions deemed compositional.

Another notable deviation occurred in Slovene, where the VPC category, as
defined by the generic guidelines, hardly or never occurs, however it was used
instead to annotate idiomatic verb-preposition combinations, such as (SL) *prišlo
je do nesreče* ‘it came to an accident’ ⇒ ‘an accident occurred’.

The status of VPCs in Italian is interesting. As a Romance language, Italian
was expected not to exhibit VPCs, but several dozens of VPC annotations do oc-
cur in the Italian corpus, e.g. (IT) *volata via* ‘flew PART’ ⇒ ‘slipped away’, *tira
fuori* ‘pulls PART’ ⇒ ‘shows’, or *va avanti* ‘goes PART’ ⇒ ‘goes on’. This shows
the possibly ambiguous status of *via* ‘by/away’, *avanti* ‘on/forward’, *fuori* ‘out/out-
side’, etc. as either adverbs or particles, triggering the ID or the VPC category, re-
spectively. The semantic compositionality of some of these constructions might
also be examined more closely.

In Bulgarian and Czech, the auxiliaries accompanying the head verbs were
annotated as VMWE components, e.g. in (CS) *on se bude bavit* ‘he REFL will
play’ ⇒ ‘he will play’, in (BG) *te ne sa dali saglasie* ‘they not *are* given con-
sent’ ⇒ ‘they have not given consent’. This is in contrast with the guidelines,
which stipulate that only the lexicalised components should be annotated. The
motivation for this deviation was to always include a finite verb in the annotated
expression, so as to e.g. easily study the tense and mood restrictions in VMWEs.
Since such studies are enabled by the accompanying morpho-syntactic data (current-
ly existent in Czech and to be provided in Bulgarian in the future), these
divergences should be eliminated in new editions of the corpus.

In German, a deviation was observed with respect to VMWEs containing both
a reflexive clitic and a particle such as (DE) sie bringen sich ein ‘they bring REFL
PART' ⇒ 'they contribute'. Such cases were annotated as IReflVs with nested VPCs, which does not conform to Test 7 (§4.2.1) stipulating that, whenever the VMWE has more than one lexicalised dependent of the head verb, it should be classified as an ID (here: with a nested VPC). Good reasons exist for each of these strategies and more discussion is needed to arbitrate for future releases of the guidelines.

Lithuanian seems to have a surprisingly low number of LVCs, despite the large size of the annotated corpus. It would be worthwhile to study in more detail if this phenomenon is inherent to the language or results from a more restrictive understanding of the LVC scope.

In Hebrew, a relatively large number of VMWEs of type OTH was observed (cf. Table 3), and a necessity of defining a new category (specific to non-Indo-European languages) was hypothesised. A more detailed study revealed that most OTH annotations were spurious: they concerned statistical collocations or VMWEs of the ID or LVC types. Some idiomatic verb-preposition combinations were also annotated in Hebrew, despite the fact that we had abandoned the IPrepV category in the earlier stages of the project (§3). There, the annotators faced a particular challenge from prepositions which often attach to the governed noun and annotating them as separate lexicalised tokens was mostly impossible. Thus, in the following sequence: (HE) *sovel me.achuz avtala* 'supfers from.a.percentage of.unemployment' the free complement *achuz* 'percentage' had to be annotated as lexicalised together with its governing preposition *me* 'from'. This problem will be dealt with in the future, when inherently adpositional verbs will be addressed (§10).

In Turkish, the LVC and OTH types also had their language-specific interpretation. Namely, the Turkish PARSEME corpus resulted from adapting a pre-existing MWE typology and dataset (Adalı et al. 2016). There, the definition of a light verb, based on Turkish linguistic works (Siemieniec-Gołaś 2010), was context-independent, i.e. restricted to a closed list of 6 verbs: *olmak* 'to be', *etmek* 'to do', *yapmak* 'to make', *kılmak* 'to render', *eylemek* 'to make' and *buyurmak* 'to order'. Verb-noun combinations with other operator verbs, such as **söz vermek** 'promise to give' ⇒ 'to promise', were then classified as OTH. A closer look at the existing OTH annotations reveals, indeed, that most of them can be re-classified as LVC in future releases of the corpus.

Czech is another language in which a pre-existing MWE-annotated corpus (Hajič et al. 2017) was adapted to the needs of the PARSEME initiative. There, complex identification and conversion procedures had to be designed (Bejček et al. 2017). The resulting mapping procedure could be fully automatic, which suggests
that the understanding of the VMWE phenomenon is similar in both annotation projects. It would still be interesting to compare both annotation guidelines more thoroughly and look for possible divergences.

**5 Annotation methodology and tools**

Mathet et al. (2015) mention several challenging features of linguistic annotation, some of which are relevant to the VMWE annotation task:

• *Unitising*, i.e. identifying the boundaries of a VMWE in the text;

• *Categorisation*, i.e. assigning each identified VMWE to one of the pre-defined categories (§3);

• *Sporadicity*, i.e. the fact that not all text tokens are subject to annotation (unlike in part-of-speech annotation, for instance);

• *Free overlap*, e.g. in (CS) **ukládal různé sankce a penále** ‘put various sanctions and penalties’, where two LVCs share a light verb;

• *Nesting*,

– at the syntactic level, as in (40), where an IReflV (PL) **skarżyć się** ‘to complain REFL ⇒ ‘to complain’ occurs in a relative clause modifying the predicative noun of the LVC (PL) **popełnić oszustwo** ‘to commit a fraud’.

(40) *Oszustwa*, na jakie *skarża się* Cyganie, popełniły frauds, on which complain REFL Gypsies, committed grupy zorganizowane. (PL) groups organised Organised groups committed frauds about which the Gypsies REFL complain. ‘Frauds which Gypsies complain about were committed by organised groups.’

– at the level of lexicalised components, as in (41), where the ID (PT) *fazer justiça* ‘to make justice’ ⇒ ‘to do justice’ is nested within a larger ID.

(41) *Ales fizeram justiça com as próprias mãos.* (PT) they made justice with their own hands They made justice with their own hands. ‘They took the law into their own hands.’
Two other specific challenges are:

* *Discontinuities*, e.g. (CS) *on ukládal různé sankce* 'he put various sanctions';

* *Multiword token* VMWEs, e.g. separable IReflVs or VPCs:<fnref n="8" />
  (ES) *abstener.se* 'to abstain.REFL' ⇒ 'to abstain',
  (HU) *át.ruház* 'to PART.dress' ⇒ 'to transfer'.

This complexity is largely increased by the multilingual nature of the task, and
calls for efficient project management and powerful annotation tools.

**5.1 Project management**

The list of language teams having initially expressed their interest in this initia-
tive included those mentioned in p. 91, as well as English, Croatian and Yiddish,
for which no corpus release could be achieved due to the lack of sufficiently avail-
able native annotators. All languages were divided into four language groups
(LGs) - Balto-Slavic, Germanic, Romance and others - as also described in p. 91.
The coordination of this large project included the definition of roles – project
leaders, technical experts, language group leaders (LGLs), language leaders (LLs)
and annotators – and their tasks.

The biggest challenge in the initial phase of the project was the development
of the annotation guidelines<fnref n="9" /> which would be as unified as possible but which
would still allow for language-specific categories and tests. To this end, a two-
phase pilot annotation in most of the participating languages was carried out.
Some corpora were annotated at this stage not only by native but also by near-
native speakers, so as to promote cross-language convergences. Each pilot an-
notation phase provided feedback from annotators, triggered discussions among
language (group) leaders and organisers, and led to enhancements of the guide-
lines, corpus format and tools.

We also defined strategies for selecting the final corpora. They should: (i) be
written in the original, in order to avoid MWE-related translationese issues; (ii)

<footnote n="8">⁸Note that annotating separate syntactic words within such tokens would be linguistically more appropriate, and would avoid bias in inter-annotator agreement and evaluation measures – cf. §6.2 and (Savary et al. 2017). However, we preferred to avoid token-to-word homogenising mainly for the reasons of compatibility. Namely, for many languages, pre-existing corpora were used, and we would like VMWE annotations to rely on the same tokenisation as the other annotation layers.</footnote>

<footnote n="9">⁹Their final version, with examples in many participating languages, is available under the CC BY 4.0 license at http://parsemefr.lif.univ-mrs.fr/parseme-st-guidelines/1.0/.</footnote>
correspond to the same genre: newspaper texts or Wikipedia articles;<fnref n="10" /> (iii) consist of longer text fragments (rather than isolated sentences), so as to enable disambiguation and coreference resolution; (iv) not be automatically pre-selected in view of a higher density of VMWEs (so as to provide both positive and negative examples); (v) be free from copyright issues, i.e. compatible with open licenses.

## 5.2 Annotation platform

For this large-scale corpus construction, we needed a centralised web-based annotation tool. Its choice was based on the following criteria: (i) handling different alphabets; (ii) accounting for right-to-left scripts; and (iii) allowing for discontinuous, nested and overlapping annotations. We chose FLAT,<fnref n="11" /> a web platform which, in addition to the required criteria, enables token-based selection of text spans, including cases in which adjacent tokens are not separated by spaces. It is possible to authenticate and manage annotators, define roles and fine-grained access rights, as well as customise specific settings for different languages.

FLAT is implemented as a web-based frontend with support for multiple users, user groups, and with configurable access rights. The frontend communicates with the FoLiA document server backend,<fnref n="12" /> which loads and holds documents in memory as they are being edited, writes them to disk again at convenient times, and unloads them when they are not used anymore. The document server has Git version control support,<fnref n="13" /> allowing changes to be tracked. In addition, for each individual FoLiA annotation, e.g. each VMWE, information such as who made the annotation, and when, is automatically registered.

FLAT is document-centric, i.e. it supports annotation of full documents together with their structure (headers, bulleted lists, figures, etc.). This contrasts with tools which take a more corpus-based approach with keyword-in-context visualisation. FLAT does allow for various other *perspectives* on the document; for the PARSEME annotation task a sentence-based perspective was chosen, presenting users with one or more pages of clearly delimited sentences to annotate. An example is shown in Figure 2.

FLAT is based on FoLiA,<fnref n="14" /> a rich XML-based format for linguistic annotation (van Gompel & Reynaert 2013), and is compatible with a wide variety of linguistic

<footnote n="10">¹⁰Deviations from this rule occurred in some languages due to the choice of pre-existing corpora, e.g. in Hungarian legal texts were used.</footnote>

<footnote n="11">¹¹https://github.com/proycon/flat</footnote>

<footnote n="12">¹²https://github.com/foliadocserve</footnote>

<footnote n="13">¹³https://git-scm.com/</footnote>

<footnote n="14">¹⁴https://proycon.github.io/folia</footnote>
<figure><img src="image_21.png" /><figcaption>Figure 2: FLAT annotation interface with a Polish text. The VMWEs are coloured according to their categories. POS tags (fin, ger, ppas, ppas, and praet) are displayed above all verbal tokens. Some attributes (VMWE category, confidence level and a comment) of the highlighted VMWE (PL) wymierzyć karę ‘to PART measure a punishment’ ⇒ ‘to mete out a punishment’ are edited in the annotation editor.</figcaption></figure>

tic annotation types. VMWEs, or entities as they are called more generically in FoLiA, constitute the most important annotation type for PARSEME. Still, certain language teams worked on documents enriched with more linguistic annotations, such as part-of-speech tags, to aid the annotation process, as shown in Figure 2. The underlying aspiration of both FoLiA and FLAT is to provide a single unified solution for multiple annotation needs, with respect to the encoding format and the annotation environment, respectively.

While the FoLiA format specifies possible linguistic annotation types and struc-
tural types, it does not commit to any particular tagset/vocabulary nor language.
Instead, tagsets are defined externally in *FoLiA set definitions*, which can be pub-
lished anywhere online by anyone and are deliberately separate from the annota-
tion format itself. A dozen of set definitions for PARSEME, based on the VMWE
categories relevant to different languages or language groups (§3) are likewise
published in a public repository.<fnref n="15" /> All FoLiA documents declare which particular
set definitions to use for which annotation types. FLAT uses these set definitions
to populate various selection boxes, as shown in Figure 2.

<footnote n="15">¹⁵https://github.com/proycon/parseme-support</footnote>
All software discussed here is available under an open-source license.<fnref n="16" /> It is part of a wider and growing infrastructure of FoLiA-capable NLP tools (van Gompel et al. 2017), developed and funded in the scope of the CLARIAH<fnref n="17" /> project and its predecessor CLARIN-NL.

Although FLAT has been in use for various other annotation projects, the PARSEME initiative, currently with over 80 active FLAT users, is the biggest use case to date, and as such has had a very positive influence in terms of the maturity of the software, fixing bugs, attaining improved performance and scalability, and compiling appropriate documentation. Various features were added to accommodate PARSEME specifically: (i) uploading documents in non-FoLiA formats, needed for the parseme-tsv format (6.1); (ii) right-to-left support necessary for Farsi and Hebrew; (iii) a metadata editor; (iv) enhanced file and user management; (v) confidence level and free-text comments as part of the editable attributes (Figure 2).

Out of 18 language teams which achieved a corpus release, 13 used FLAT as their main annotation environment. The 5 remaining teams either used other (generic or in-house) annotation tools, or converted existing VMWE-annotated corpora.

## 5.3 Automatic VMWE pre-annotation

Automatic pre-annotation of corpora is a current practice in many annotation tasks. In the PARSEME corpus project, it was applied by the Bulgarian and Hungarian teams, on the basis of manually compiled lists of VMWEs. All texts were then manually checked and corrected.

More precisely, pre-annotation in Bulgarian included automatic annotation of: (a) verb forms (triggers for VMWEs), (b) IReflV candidates consisting of a verb and a reflexive particle, and (c) VMWEs from a large dictionary of Bulgarian MWEs (Koeva et al. 2016). Cases of false positives included: (i) literal uses of existing VMWEs, (ii) false IReflVs which are true reflexive or passive constructions instead (§4.2.4), or (iii) coincidental co-occurrence of VMWE components. All annotations were manually verified and such cases were eliminated. False negatives could also be efficiently tracked thanks to the highlighted verb forms.

Automatic pre-annotation is known to introduce a task-dependent bias (Marcus et al. 1993; Fort & Sagot 2010) which may be both positive (simple repetitive tasks are handled uniformly and speeded up) and negative (annotators may tend

<footnote n="16">¹⁶GNU Public License v3</footnote>

<footnote n="17">¹⁷https://www.clariah.nl</footnote>
to rely too much on the automatic pre-annotation and fail to detect false negatives). We are not aware of any studies about biases related to VMWE annotation. We expect a minor risk of bias to stem from a possibly unbalanced VMWE dictionary: if one category (e.g. LVCs) is better represented than others, annotators may become more attentive to it. A bias might also be introduced by relatively productive constructions, when a large majority, but not all, of their occurrences belong to a unique category. For instance, the verb (BG) *davam* ‘to give’ occurs often and in many different LVCs, e.g. with *saglasie* ‘consent’, *razreshenie* ‘permission’ *obyasnenie* ‘explanation’, etc. The annotators could, therefore, tend to wrongly assign the LVC category to other expressions containing the same verb, such as *davam duma* ‘to give word’ (ID), or *davam prizovka* ‘to give subpoena’ (non-VMWE or borderline case).

## 5.4 Consistency checks and homogenisation

Even though the guidelines heavily evolved during the two-stage pilot annotation, there were still questions from annotators at the beginning of the final annotation phase. We used an issue tracker (on Gitlab)<fnref n="18" /> in which language leaders and annotators could discuss issues with other language teams.

High-quality annotation standards require independent double annotation of a corpus followed by adjudication, which we could not systematically apply due to time and resource constraints. For most languages, each text was handled by one annotator only (except for a small corpus subset used to compute inter-annotator agreement, see §6.2). This practice is known to yield inattention errors and inconsistencies between annotators, and since the number of annotators per language varies from 1 to 10, we used consistency support tools.

Firstly, some language teams (Bulgarian, French, Hungarian, Italian, Polish, and Portuguese) kept a list of VMWEs and their classification, agreed upon by all annotators and updated collaboratively over time.<fnref n="19" /> Secondly, for some languages (German, French, Hebrew, Italian, Polish, Portuguese, Romanian and Spanish) the annotation was followed by homogenisation. An in-house tool extracted the annotated VMWEs from a given corpus and rescanned the corpus to find all potential occurrences of the same VMWEs, whether already annotated or not. It then generated an HTML page where all positive and negative examples of a given VMWE were grouped, and could be accepted or rejected manually. En-

<footnote n="18">¹⁸https://gitlab.com/parseme/sharedtask-guidelines/issues</footnote>

<footnote n="19">¹⁹Like automatic pre-annotation, this practice increases the consistency and speed of the annotator’s work, but it also introduces a risk of bias, since collective decisions may override linguistic intuition. Therefore, such instruments should always be used with special care.</footnote>
tries were sorted so that similar VMWEs, such as (EN) *payed a visit* and *re-ceived a visit*, appeared next to each other. In this way, noise and silence errors could easily be spotted and manually corrected. The tool was mostly used by language leaders and/or highly committed annotators. The resulting gain in precision and recall was substantial. For instance, in Spanish the number of the annotated MWEs increased by 40% (from 742 to 1248), most notably in the IReflV category. Figure 3 shows the interface used to correct consistency problems.

<figure><img src="image_22.png" /><figcaption>Figure 3: Consistency-check tool at work. Here, (ES) *poner en marcha* ‘to put in march’ ⇒ ‘to start’ was annotated once as LVC, twice as ID and once skipped. The clickable icon next to each example allows the user to add, correct or delete an annotation. VMWEs with the same noun, e.g. (ES) *poner fin* ‘to put end’ ⇒ ‘to terminate’ and *tocar a su fin* ‘to touch to its end’ ⇒ ‘to come to its end’ on the top of the screen, are gathered so as to enhance annotation consistency, especially for LVCs.</figcaption></figure>

# 6 Properties of the annotated corpus

Table 3 provides overall statistics of the corpus annotated for the shared task.<fnref n="20" /> In total, it contains almost 5,5 million tokens, 274 thousand sentences and 62 thousand VMWE annotations. The amount and distribution of VMWEs over categories varies considerably across languages.

No category was used in all languages, but the two universal categories, ID and LVC, were used in almost all languages. In Hungarian, no ID was annotated

<footnote n="20">²⁰The split into training and test corpora is indicated in Savary et al. (2017).</footnote>
<table><caption>Table 3: Overview of the annotated corpora in terms of the number of sentences, of tokens (whether belonging to the annotated VMWEs or not), and of the annotated VMWEs occurrences (overall and per category).</caption><thead><tr><th rowspan="2">Language</th><th rowspan="2">Sentences</th><th rowspan="2">Tokens</th><th colspan="6">VMWE occurrences</th></tr><tr><th>All</th><th>IDs</th><th>IReflVs</th><th>LVCs</th><th>OTHs</th><th>VPCs</th></tr></thead><tbody><tr><td>BG</td><td>8,860</td><td>200,128</td><td>2,406</td><td>517</td><td>1,376</td><td>511</td><td>2</td><td>0</td></tr><tr><td>CS</td><td>49,431</td><td>833,193</td><td>14,536</td><td>1,611</td><td>10,000</td><td>2,923</td><td>2</td><td>0</td></tr><tr><td>DE</td><td>7,500</td><td>144,856</td><td>2,947</td><td>1,219</td><td>131</td><td>218</td><td>10</td><td>1,369</td></tr><tr><td>EL</td><td>8,811</td><td>226,265</td><td>2,018</td><td>642</td><td>0</td><td>1,291</td><td>37</td><td>48</td></tr><tr><td>ES</td><td>4,634</td><td>159,807</td><td>1,248</td><td>362</td><td>556</td><td>320</td><td>10</td><td>0</td></tr><tr><td>FA</td><td>3,226</td><td>55,207</td><td>3,207</td><td>0</td><td>0</td><td>0</td><td>3,207</td><td>0</td></tr><tr><td>FR</td><td>19,547</td><td>486,005</td><td>4,962</td><td>1,905</td><td>1,418</td><td>1,633</td><td>6</td><td>0</td></tr><tr><td>HE</td><td>7,000</td><td>147,361</td><td>1,782</td><td>116</td><td>0</td><td>380</td><td>693</td><td>593</td></tr><tr><td>HU</td><td>4,311</td><td>108,175</td><td>3,499</td><td>0</td><td>0</td><td>730</td><td>0</td><td>2,769</td></tr><tr><td>IT</td><td>17,000</td><td>427,848</td><td>2,454</td><td>1,163</td><td>730</td><td>482</td><td>6</td><td>73</td></tr><tr><td>LT</td><td>14,863</td><td>256,235</td><td>502</td><td>287</td><td>0</td><td>215</td><td>0</td><td>0</td></tr><tr><td>MT</td><td>10,600</td><td>152,285</td><td>1,272</td><td>446</td><td>0</td><td>693</td><td>133</td><td>0</td></tr><tr><td>PL</td><td>13,606</td><td>220,934</td><td>3,649</td><td>383</td><td>1,813</td><td>1,453</td><td>0</td><td>0</td></tr><tr><td>PT</td><td>22,240</td><td>414,020</td><td>3,947</td><td>910</td><td>596</td><td>2,439</td><td>2</td><td>0</td></tr><tr><td>RO</td><td>51,500</td><td>879,427</td><td>4,540</td><td>599</td><td>2,786</td><td>1,154</td><td>1</td><td>0</td></tr><tr><td>SL</td><td>11,411</td><td>235,864</td><td>2,287</td><td>375</td><td>1,198</td><td>231</td><td>4</td><td>479</td></tr><tr><td>SV</td><td>1,800</td><td>29,517</td><td>292</td><td>60</td><td>17</td><td>27</td><td>2</td><td>186</td></tr><tr><td>TR</td><td>18,036</td><td>362,077</td><td>6,670</td><td>3,160</td><td>0</td><td>2,823</td><td>687</td><td>0</td></tr></tbody><tfoot><tr><td>Total</td><td>274,376</td><td>5,439,204</td><td>62,218</td><td>13,755</td><td>20,621</td><td>17,523</td><td>4,802</td><td>5,517</td></tr></tfoot></table>

due to the genre of the corpus, mainly composed of legal texts. In Farsi, no categorisation was performed (§4.3), and all annotated VMWEs are marked as OTH instead.

The most frequent category is IReflV, in spite of it being quasi-universal, mainly due to its prevalence in Czech. IReflVs were annotated in all Romance and Slavic languages, and in German and Swedish. VPCs were annotated in German, Swedish, Greek, Hungarian, Hebrew, Italian, and Slovene. In the three last languages this category had a language-specific interpretation, as was the case of OTH in Hebrew and Turkish (§4.3). No language-specific categories have been defined.
All the corpora are freely available on the LINDAT/CLARIN platform.<fnref n="21" /> The VMWE annotations are released under Creative Commons licenses, with constraints on commercial use and sharing for some languages. Some languages use data from other corpora (notably from the UD project), including additional annotations. These are released under the terms of the original licenses.

## 6.1 Format

The official format of the annotated data is the parseme-tsv format,<fnref n="22" /> exemplified in Figure 4. It is adapted from the CoNLL format, with one token per line and an empty line indicating the end of a sentence. Each token is represented by 4 tab-separated columns featuring (i) the position of the token in the sentence, or a range of positions (e.g. 1–2) in case of MWTs such as contractions; (ii) the token surface form; (iii) an optional nsp (no space) flag indicating that the current token is adjacent to the next one; and (iv) an optional VMWE code composed of the VMWE’s consecutive number in the sentence and – for the initial token in a VMWE – its category, for example, 2:ID if a token is the first one in an idiom which is the second VMWE in the current sentence. In case of nested, coordinated or overlapping VMWEs, multiple codes are separated with a semicolon.

Formatting of the final corpus required a language-specific tokenisation procedure, which can be particularly tedious in languages presenting contractions. For instance, (FR) *du* ‘of-the’ is a contraction of the preposition (FR) *de* ‘of’ and the article (FR) *le* ‘the.MASC’.

Some language teams resorted to previously annotated corpora which have been converted to the parseme-tsv format automatically (or semi-automatically if some tokenisation rules were revisited). Finally, scripts for converting the parseme-tsv format into the FoLiA format and back were developed to ensure corpus compatibility with FLAT (5.2).

## 6.2 Inter-annotator agreement

Inter-annotator agreement (IAA) measures are meant to assess the hardness of the annotation task, as well as the quality of the annotation guidelines, of the annotation methodology, and of the resulting annotations. Defining such measures is not always straightforward due to the challenges listed in §5.

To assess unitising, two annotators double-annotated an extract of the corpus in each language. We then calculated the MWE-based F-score ($F1_{unit}$) of one

<footnote n="21">²¹http://hdl.handle.net/11372/LRT-2282</footnote>

<footnote n="22">²²http://typo.uni-konstanz.de/parseme/index.php/2-general/184-parseme-shared-task-format-of-the-final-annotation</footnote>
<table><caption>Figure 4: Annotation of two sample sentences containing a contraction (wouldn't), a verbal idiom, and two overlapping VPCs.</caption><tr><td>1-2</td><td>Wouldn't</td><td></td><td></td></tr><tr><td>1</td><td>Would</td><td></td><td></td></tr><tr><td>2</td><td>not</td><td></td><td></td></tr><tr><td>3</td><td>questioning</td><td></td><td></td></tr><tr><td>4</td><td>colonial</td><td></td><td></td></tr><tr><td>5</td><td>boundaries</td><td></td><td></td></tr><tr><td>6</td><td>open</td><td>1:ID</td><td></td></tr><tr><td>7</td><td>a</td><td></td><td></td></tr><tr><td>8</td><td>dangerous</td><td></td><td></td></tr><tr><td>9</td><td>Pandora</td><td>nsp</td><td>1</td></tr><tr><td>10</td><td>'</td><td>nsp</td><td>1</td></tr><tr><td>11</td><td>s</td><td></td><td>1</td></tr><tr><td>12</td><td>box</td><td>nsp</td><td>1</td></tr><tr><td>13</td><td>?</td><td></td><td></td></tr><tr><td>1</td><td>They</td><td></td><td></td></tr><tr><td>2</td><td>were</td><td></td><td></td></tr><tr><td>3</td><td>letting</td><td></td><td>1:VPC;2:VPC</td></tr><tr><td>4</td><td>him</td><td></td><td></td></tr><tr><td>5</td><td>in</td><td></td><td>1</td></tr><tr><td>6</td><td>and</td><td></td><td></td></tr><tr><td>7</td><td>out</td><td></td><td>2</td></tr><tr><td>8</td><td>.</td><td>nsp</td><td></td></tr></table>

annotator with respect to the other.<fnref n="23" /> MWE-based F-score is defined in Savary et al. (2017) and was used to evaluate the systems submitted to the shared task.

We also report an estimated Cohen's κ (κ<sub>unit</sub>). Measuring IAA, particularly κ, for unitising is not straightforward due to the absence of negative examples, that is, spans for which both annotators agreed that they are not VMWEs. From an extreme perspective, any combination of a verb with other tokens (of any length) in a sentence is a potential VMWE.<fnref n="24" /> Consequently, as the density of VMWEs in most languages is rather low, one can argue that the probability of chance agreement approaches 0, and IAA can be measured simply using the observed agreement F₁<sub>unit</sub>. However, in order to provide a possibly less biased measure

<footnote n="23">²³That is, we suppose that one annotator represents the system, and the other one represents the gold standard. Note that F-score is symmetrical (depending on the order, recall and precision are inverted), so none of the two annotators is prioritised.</footnote>

<footnote n="24">²⁴Also note that annotated segments can overlap.</footnote>
to the reported F-scores, we assume that the total number of stimuli in the annotated corpora is approximately equivalent to the number of verbs, which is slightly higher than the number of sentences. We roughly estimate this quantity as the number of sentences plus the number of VMWEs annotated by at least one annotator.<fnref n="25" /> Finally, to assess categorisation, we apply the standard $κ (κ_{cat})$ to the VMWEs for which annotators agree on the span.

Due to time and resource constraints, the majority of the corpus for most languages was annotated by a single annotator. Only small fractions were double-annotated for the purpose of the IAA calculation. All available IAA results are presented in Table 4. For some languages the IAA in unitising is rather low. We believe that this results from particular annotation conditions. In Spanish, the annotated corpus is small (Table 3), so the annotators did not become sufficiently accustomed to the task. A similar effect occurs in Polish and Farsi, where the first annotator performed the whole annotation of the train and test corpora, while the second annotator only worked on the IAA-dedicated corpus. The cases of Hebrew, and especially of Italian, should be studied more thoroughly in the future. Note also that in some languages the numbers from Table 4 are a lower bound for the quality of the final corpus, due to post-annotation homogenisation (§5.4).

A novel proposal of the holistic $γ$ measure (Mathet et al. 2015) combines unitising and categorisation agreement in one IAA score, because both annotation subtasks are interdependent. In our case, however, separate IAA measures seem preferable both due to the nature of VMWEs and to our annotation methodology. Firstly, VMWEs are known for their variable degree of non-compositionality. In other words, their idiomaticity is a matter of scale. But this fact is not accounted for in current corpus annotation standards and identification tools, which usually rely on binary decisions, i.e. a candidate is seen as a VMWE or a non-VMWE, with no gradation of this status. Such a binary model is largely sub-optimal for a large number of grey-zone VMWE candidates. However, once a VMWE has been considered valid, its categorisation appears to be significantly simpler, as shown in the last 2 columns of Table 4 (except for Romanian and Hebrew). Secondly, as described in §4.1 – §4.2, our annotation guidelines are structured in two main decision trees – an identification and a categorisation tree – to be applied mostly sequentially. Therefore, separate evaluation of these two stages may be helpful in enhancing the guidelines.

<footnote n="25">²⁵In other words, the number of items on which both annotators agree as being no VMWEs is estimated as the number of sentences. This assumption ignores the fact that some verbs may be part of more than one VMWE, since this is rare.</footnote>
<table><caption>Table 4: IAA scores: #S, and #T show the number of sentences and tokens in the double-annotated sample used to measure IAA, respectively. #A₁ and #A₂ refer to the number of VMWE instances annotated by each of the annotators.</caption><thead><tr><th></th><th>#S</th><th>#T</th><th>#A<sub>1</sub></th><th>#A<sub>2</sub></th><th>F1<sub>unit</sub></th><th>κ<sub>unit</sub></th><th>κ<sub>cat</sub></th></tr></thead><tbody><tr><td>BG</td><td>608</td><td>27491</td><td>298</td><td>261</td><td>81.6</td><td>0.738</td><td>0.925</td></tr><tr><td>EL</td><td>1383</td><td>33964</td><td>217</td><td>299</td><td>68.6</td><td>0.632</td><td>0.745</td></tr><tr><td>ES</td><td>524</td><td>10059</td><td>54</td><td>61</td><td>38.3</td><td>0.319</td><td>0.672</td></tr><tr><td>FA</td><td>200</td><td>5076</td><td>302</td><td>251</td><td>73.9</td><td>0.479</td><td>n/a</td></tr><tr><td>FR</td><td>1000</td><td>24666</td><td>220</td><td>205</td><td>81.9</td><td>0.782</td><td>0.93</td></tr><tr><td>HE</td><td>1000</td><td>20938</td><td>196</td><td>206</td><td>52.2</td><td>0.435</td><td>0.587</td></tr><tr><td>HU</td><td>308</td><td>8359</td><td>229</td><td>248</td><td>89.9</td><td>0.827</td><td>1.0</td></tr><tr><td>IT</td><td>2000</td><td>52639</td><td>336</td><td>316</td><td>41.7</td><td>0.331</td><td>0.78</td></tr><tr><td>PL</td><td>1175</td><td>19533</td><td>336</td><td>220</td><td>52.9</td><td>0.434</td><td>0.939</td></tr><tr><td>PT</td><td>2000</td><td>41636</td><td>411</td><td>448</td><td>77.1</td><td>0.724</td><td>0.964</td></tr><tr><td>RO</td><td>2500</td><td>43728</td><td>183</td><td>243</td><td>70.9</td><td>0.685</td><td>0.592</td></tr><tr><td>TR</td><td>6000</td><td>107734</td><td>3093</td><td>3241</td><td>71.1</td><td>0.578</td><td>0.871</td></tr></tbody></table>

## 6.3 Cross-language analysis

The common terminology and annotation methodology achieved in this endeavour enable cross-language observations. In this section we offer a comparative quantitative analysis of several phenomena relevant to the challenges VMWEs pose in NLP, as discussed in §1. Namely, we analyse the lengths, discontinuities, coverage, overlapping and nesting of VMWEs across languages and VMWE types.

Table 5 provides statistics about the length and discontinuities of annotated VMWEs in terms of the number of tokens.<fnref n="26" /> The average lengths range between 1.27 (in Hungarian) and 2.71 (in Hebrew) tokens, but the dispersion varies across languages: the mean absolute deviation (MAD) is 0.75 for Hebrew, while it is 0.11 for Turkish. Single-token VMWEs (length=1) are frequent in Hungarian and German (63% and 24% of all VMWEs, respectively) but rare or non-existent in other languages. The right part of Table 5 shows the lengths of discontinuities (gaps). This factor is measured in terms of the total number of tokens not belonging to

<footnote n="26">²⁶Since the version published in Savary et al. (2017), we corrected a bug in the length average and MAD calculation, which impacted the results for languages containing VMWEs with one token only (especially DE and HU).</footnote>
<table><caption>Table 5: Length and discontinuities of VMWE occurrences in number of tokens in the training corpora. Col. 2–3: average and mean absolute deviation (MAD) for length. Col. 4: number of single-token VMWEs. Col. 5–6: average and MAD for the length of discontinuities. Col. 7–8: number and percentage of continuous VMWEs. Col. 9–11: number of VMWEs with discontinuities of length 1, 2 and 3. Col. 12–13: number and percentage of VMWEs discontinuities of length > 3.</caption><thead><tr><th rowspan="2">Lang.</th><th colspan="3">Length of VMWE</th><th colspan="10">Length of discontinuities (excl. VMWEs of length 1)</th></tr><tr><th>Avg</th><th>MAD</th><th>=1</th><th>Avg</th><th>MAD</th><th>0</th><th>%0</th><th>1</th><th>2</th><th>3</th><th>&gt;3</th><th>%&gt;3</th></tr></thead><tbody><tr><td>BG</td><td>2.45</td><td>0.63</td><td>1</td><td>0.64</td><td>1.05</td><td>1586</td><td>82.1</td><td>206</td><td>33</td><td>25</td><td>82</td><td>(4.2%)</td></tr><tr><td>CS</td><td>2.30</td><td>0.46</td><td>0</td><td>1.35</td><td>1.53</td><td>6625</td><td>51.5</td><td>2357</td><td>1465</td><td>944</td><td>1461</td><td>(11.4%)</td></tr><tr><td>DE</td><td>2.02</td><td>0.61</td><td>715</td><td>2.96</td><td>2.94</td><td>619</td><td>35.7</td><td>283</td><td>159</td><td>142</td><td>529</td><td>(30.5%)</td></tr><tr><td>EL</td><td>2.45</td><td>0.61</td><td>3</td><td>0.94</td><td>1.08</td><td>870</td><td>57.4</td><td>389</td><td>124</td><td>50</td><td>82</td><td>(5.4%)</td></tr><tr><td>ES</td><td>2.24</td><td>0.39</td><td>0</td><td>0.47</td><td>0.66</td><td>523</td><td>69.9</td><td>162</td><td>33</td><td>14</td><td>16</td><td>(2.1%)</td></tr><tr><td>FA</td><td>2.16</td><td>0.27</td><td>0</td><td>0.42</td><td>0.70</td><td>2243</td><td>82.9</td><td>202</td><td>103</td><td>60</td><td>99</td><td>(3.7%)</td></tr><tr><td>FR</td><td>2.29</td><td>0.44</td><td>1</td><td>0.65</td><td>0.80</td><td>2761</td><td>61.9</td><td>1116</td><td>336</td><td>125</td><td>123</td><td>(2.8%)</td></tr><tr><td>HE</td><td>2.71</td><td>0.75</td><td>0</td><td>0.47</td><td>0.74</td><td>1011</td><td>78.9</td><td>129</td><td>54</td><td>43</td><td>45</td><td>(3.5%)</td></tr><tr><td>HU</td><td>1.27</td><td>0.39</td><td>2205</td><td>1.01</td><td>1.29</td><td>506</td><td>63.7</td><td>178</td><td>34</td><td>15</td><td>61</td><td>(7.7%)</td></tr><tr><td>IT</td><td>2.58</td><td>0.64</td><td>2</td><td>0.28</td><td>0.46</td><td>1580</td><td>80.9</td><td>278</td><td>56</td><td>22</td><td>16</td><td>(0.8%)</td></tr><tr><td>LT</td><td>2.35</td><td>0.53</td><td>0</td><td>0.72</td><td>0.94</td><td>261</td><td>64.9</td><td>79</td><td>36</td><td>9</td><td>17</td><td>(4.2%)</td></tr><tr><td>MT</td><td>2.64</td><td>0.68</td><td>7</td><td>0.34</td><td>0.53</td><td>589</td><td>77.0</td><td>123</td><td>33</td><td>12</td><td>8</td><td>(1.0%)</td></tr><tr><td>PL</td><td>2.11</td><td>0.20</td><td>0</td><td>0.53</td><td>0.77</td><td>2307</td><td>73.3</td><td>470</td><td>195</td><td>90</td><td>87</td><td>(2.8%)</td></tr><tr><td>PT</td><td>2.19</td><td>0.37</td><td>76</td><td>0.67</td><td>0.78</td><td>1964</td><td>58.3</td><td>1016</td><td>223</td><td>82</td><td>86</td><td>(2.6%)</td></tr><tr><td>RO</td><td>2.15</td><td>0.25</td><td>1</td><td>0.55</td><td>0.72</td><td>2612</td><td>64.7</td><td>689</td><td>693</td><td>32</td><td>13</td><td>(0.3%)</td></tr><tr><td>SL</td><td>2.27</td><td>0.43</td><td>14</td><td>1.47</td><td>1.54</td><td>787</td><td>44.4</td><td>445</td><td>221</td><td>118</td><td>202</td><td>(11.4%)</td></tr><tr><td>SV</td><td>2.14</td><td>0.25</td><td>0</td><td>0.38</td><td>0.59</td><td>44</td><td>78.6</td><td>7</td><td>3</td><td>1</td><td>1</td><td>(1.8%)</td></tr><tr><td>TR</td><td>2.06</td><td>0.11</td><td>3</td><td>0.57</td><td>0.57</td><td>3043</td><td>49.4</td><td>2900</td><td>162</td><td>33</td><td>28</td><td>(0.5%)</td></tr></tbody></table>

a VMWE but appearing between its left- and right-most lexicalised components. For instance, a gap of length 3 is counted in (DE) *jetzt bin ich bestimmt aus dem Alter heraus* ‘now am I certainly out-of the age PART’ ⇒ ‘now I am too old’. The discontinuities vary greatly across languages. While for Bulgarian, Farsi and Italian more than 80% of VMWEs are continuous, only 35.7% of German VMWEs do not have any gaps, and 30.5% of them contain discontinuities of 4 or more tokens.

Figure 5 and Figure 6 show a breakdown of the length and discontinuity scores per VMWE category (Farsi, where categorisation was not performed, is not included). Not surprisingly, IDs are longer on average than all other categories (OTHs are omitted due to their rarity), and the average ID length ranges roughly between 2.5 and 3 components. The average lengths for the other categories are closer to 2, which is expected given their definitions. Note though that VPCs are
<figure><img src="image_23.png" /><figcaption>Figure 5: Average lengths of VMWE occurrences per category, in number of components. Single-token VMWEs (frequent for Hungarian and German) are included.</figcaption></figure>

more contrasted across languages, with a low average length for German and Hungarian, due to the massive presence of single-token VMWEs. As far as IReflVs are concerned, a similar effect can be observed for some languages depending on morphological and tokenisation rules, due to the presence of IReflVs of length 1, for instance (ES) *referir.se* 'to refer.REFL' ⇒ 'to refer'. IReflVs of length greater than 2 in Czech, Bulgarian and German result from language-specific interpretations of the guidelines (§4.3).

When comparing the lengths of discontinuities across languages (Figure 6), German stands clearly out in all categories and so does Slovene to a smaller extent (probably due to the language-specific interpretation of the VPC category, §4.3), whereas Italian, Hebrew or Maltese show very few discontinuities. Note the difference for LVCs within Romance languages, which should be studied in more detail. LVCs are clearly the category showing the longest discontinuities overall, mainly due to the presence of non-lexicalised determiners and pre-modifiers of the predicative nouns, although extraction of the nouns also comes into play.

While regularities do exist in the formation of MWEs, it essentially remains an idiosyncratic and lexical phenomenon. Hence, it is very likely that the annotated
<figure><img src="image_24.png" /><figcaption>Figure 6: Size of discontinuities in VMWEs. The gap size is the total number of tokens not belonging to a VMWE but appearing between its left- and right-most lexicalised components. VMWEs of length 1 are not considered. For German the VPC average gap size is 5.25.</figcaption></figure>

datasets cover only a small fraction of all the VMWEs existing in each of the 18 languages. In order to evaluate this coverage, we propose to measure the ratio of unknown VMWEs considering a corpus split into training and test sets, similar to the split used in the shared task (Savary et al. 2017). In other words, we arbitrarily split the corpus into a training and a test set, and study the proportion of VMWEs present in the test but absent in the training set.<fnref n="27" />

Ideally, we should perform this estimation on an intra- and inter-domain basis. Unfortunately, we do not know the domain of the source text for each annotated sentence.<fnref n="28" /> To circumvent this limitation, we can still provide a lower bound of the unknown VMWE ratios by considering different splits that use continuous portions of the corpus, as shown in Figure 7. For each language for which the morphological companion files were provided, we show the average rate of un-

<footnote n="27">²⁷See also Maldonado & QasemiZadeh (2018 [this volume]) and Taslimipoor et al. (2018 [this volume]) for in-depth considerations on how the training vs. test corpus split influences the results of automatic VMWE identification.</footnote>

<footnote n="28">²⁸For instance the French dataset contains the UD corpus, whose sentences come from various untraced sources and are mixed.</footnote>
known VMWEs<fnref n="29" /> computed over 5 cross-validation splits, plotted against the total number of VMWE occurrences. For instance for Italian we get an average unknown rate of 66.2%, with roughly 2,000 annotated VMWE tokens, which means that, on average, in a fraction of 400 VMWEs, two thirds are not present in the remaining 1,600 VMWEs. The ratios are rather high, except for Hungarian and Romanian. Although we would expect these scores to have negative correlation with the size of the annotated data, the plot shows great differences even among languages with comparable numbers of annotated VMWEs. We can hypothesise that other factors come into play, such as cross-language variability of domains, text genres and annotation quality.

<figure><img src="image_25.png" /><figcaption>Figure 7: Ratios of unknown VMWEs in the different language datasets. X-axis: the total number of VMWEs tokens in the train+test corpus. Y-axis: average proportion of unknown VMWEs (present in the test but not in the train set) when performing cross-validation with 5 different train/test splits.</figcaption></figure>

We also investigated two other challenging phenomena: overlapping and nesting of VMWEs. The former was measured in terms of the frequency of tokens belonging to at least 2 VMWEs. It occurs – most often due to ellipsis in coordinated VMWEs – in most of the languages but rarely concerns more than two VMWEs at a time, as shown in Table 6. The highest number of overlapping VMWEs was

<footnote n="29">²⁹Matching of VMWEs in train and test sets is performed on lemmatised forms, and with limited normalisation of the order of components (in particular verb-noun for LVCs, and clitic-verb for IReflVs). Note that better normalisation should be performed in order to match multitoken VMWEs against their single-token variants.</footnote>
five, as seen in (42), where the light verb (PL) *wykonywać* ‘perform’ is shared by five LVCs.

(42) *Piloci wykonywali podstawowe manewry i serie wznoszeń*,
pilots performed basic maneuvers and series climbs.GEN,
*nurkować, pętli i zwrotów*. (PL)
dives.GEN, rolls.GEN and turns.GEN

‘The pilots performed basic maneuvers and series of climbs, dives, rolls and turns.’

As far as nesting is concerned, measuring this phenomenon precisely, as defined in §5, would require the availability of syntactic annotations for all languages. Since this is not the case, we approximated nesting at the syntactic level by pairs of VMWEs $E_1$ and $E_2$ such that all lexicalised components of $E_2$ are placed between the left- and right-most lexicalised components of $E_1$. Single-token VMWEs were disregarded. As the last line of Table 6 shows, such configurations occur very rarely in the data. This might be due to the fact that large gaps introduced within the outer-most VMWEs by the nested structure are harder to process for the human mind.

<table><caption>Table 6: Overlapping and nested VMWEs. Overlap >=2 and >2: the token belongs to at least 2 or more than 2 VMWEs, respectively. Only percentages above 0.49% are indicated. They are counted wrt. all tokens belonging to VMWEs.</caption><thead><tr><th></th><th>BG</th><th>CS</th><th>DE</th><th>EL</th><th>ES</th><th>FA</th><th>FR</th><th>HE</th><th>HU</th><th>IT</th><th>LT</th><th>MT</th><th>PL</th><th>PT</th><th>RO</th><th>SL</th><th>SV</th><th>TR</th></tr></thead><tbody><tr><td>Overlap >= 2</td><td>0</td><td>520</td><td>122</td><td>5</td><td>22</td><td>1</td><td>60</td><td>235</td><td>30</td><td>73</td><td>0</td><td>1</td><td>44</td><td>65</td><td>53</td><td>0</td><td>1</td><td>19</td></tr><tr><td></td><td></td><td></td><td>(1.6%)</td><td>(2%)</td><td></td><td></td><td>(5%)</td><td></td><td>(1.2%)</td><td></td><td></td><td></td><td>(0.6%)</td><td>(0.5%)</td><td>(0.5%)</td><td></td><td></td><td></td></tr><tr><td>Overlap > 2</td><td>0</td><td>11</td><td>0</td><td>1</td><td>0</td><td>0</td><td>5</td><td>9</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>6</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Nested VMWEs</td><td>4</td><td>29</td><td>1</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>1</td><td>3</td><td>0</td><td>0</td><td>4</td><td>1</td><td>0</td><td>2</td><td>0</td><td>0</td></tr></tbody></table>

## 7 Language-specific studies based on the corpus

Since its publication in January 2017, the PARSEME VMWE-annotated corpus has enabled studies in corpus linguistics in several languages.

The French corpus was addressed by Pasquer (2017), who focuses on the variability of the most frequent VMWEs. Three aspects are studied: (i) morphological
variability of VMWE components, (ii) length and nature of discontinuities be-
tween the VMWE components, (iii) syntactic dependencies between the VMWE
components and their dependents/governors. The results show a distinctly higher
variability in LVCs than in IDs. Namely, nouns inflect and govern external mod-
ifiers, respectively, 8 and 1.7 times more often in LVCs (**il rend les derniers hom-
mages** 'he pays the last tributes') than in IDs. IDs include a lexicalised determiner
(**elle tourne la page** 'she turns the page' vs. **elle joue un role** 'she plays a role')
and a compulsory negation (ça ne paye pas de mine 'it does not pay a face' ⇒ 'it
is not much to look at'), 20 and 10 times more often than LVC, respectively. LVCs
exhibit discontinuities and passivise 1.5 and 29 times more often than IDs, respec-
tively. Additionally, types of syntactic variants are listed and quantified for the 3
most variable VMWEs. Interesting types of morphological variants, such as pre-
fixations (**redonner raison** 'to re-give reason' ⇒ 'to admit again that someone
is right'), are also revealed.

In Maltese, investigations on LVCs were also carried out in the PARSEME cor-
pus extended with the Maltese UD corpus. The annotated LVCs were extracted
and proofread, and the 20 most frequent light verbs (LVs) were listed. Those
were used to find other candidate LVCs in a larger raw corpus (not annotated
for VMWEs). For each LV the number of unique predicative nouns they combine
with could be established. The results show that some LVs are inherently light
(e.g. *ta* ‘to give’, *ħa* ‘to take’ and *għamel* ‘to make/do’) and combine with large
numbers of nouns (here: 60, 48, and 46, respectively), while others are light only
when combined with a few nouns (e.g. *ġarr* ‘to carry’, *laħaq* ‘to reach/achieve’, *ta-
lab* ‘to request/ask’). An analogous experiment, performed for nouns, shows that
most of them occur with two LVs (*ta* ‘to give’ and *ħa* ‘to take’), while only few
(*appoġġ* ‘support’, *kura* ‘care/treatment’ and *kenn* ‘shelter’) combine with many
LVs. Other interesting findings are of etymological nature. Maltese is a language
with influences from Semitic and Romance languages, as well as English. The in-
spected LVCs were mostly of Romance origin (70%), some of Semitic (25%) and
some of English (5%). Interestingly, some LVCs accommodate borrowings and
Semitic elements that are no longer productive, for example, *ħa nifs* ‘to take a
breath’ is ten times more frequent than the Semitic *niffes* ‘to breathe’.

LVC-specific analyses were also performed in Lithuanian. Two groups of verbs
were identified based on their frequencies in LVCs: (i) 4 high-connectivity verbs
i.e. those that combine with large numbers of nouns: *vykdyti* ‘to carry out’ con-
nects with 19 nouns, *atlikti* ‘to perform’ – 14, *turėti* ‘to have’ – 12, *daryti* ‘to do/to
make’ – 10; (ii) 17 low-connectivity verbs i.e. those combining with less than 10
nouns, e.g. *teikti* ‘to deliver’ – 6, *surengti* ‘to arrange’ – 4, *imtis* ‘to undertake’ – 3,
priimti 'to accept' – 3, patirti 'to experience' – 3, duoti 'to give' – 3, sudaryti 'to make' – 3, etc. The numbers of the LVCs containing the verbs from (i) and (ii) are comparable – 55 and 38, respectively – but the diversity of the verbs is significantly higher in (ii) than in (i). The LVCs containing the verbs from group (i) seem to be the most prototypical ones, e.g. vykdyti patikrinimus 'to carry out inspections', atlikti analizę 'to perform an analysis', daryti spaudimą 'to put pressure', etc. These findings pave the way towards developing a comprehensive list of light verbs for Lithuanian.

## 8 Interesting problems

The considerable collective PARSEME corpus effort led us to confront various phenomena across different language families, various linguistic traditions, and annotation practices. As a result, some interesting findings allow us to view the VMWE phenomenon more globally, which should enable further cross-language generalisations.

Since semantic non-compositionality is the most pervasive property of MWEs, it should possibly be captured by generic definitions and tests in a multilingual endeavour like ours. However, semantic properties show up in different languages via different morphological, syntactic and semantic means. As a result, some semantic non-compositionality phenomena cross word boundaries in some languages, and are therefore relevant to MWEs, and others do not. This distinction can also vary from language to language for the same phenomenon.

For instance, particles in Germanic and Finno-Ugric VPCs, like (EN) *to turn off*, have similar roles as prefixes in Slavic verbs, like (PL) wyłączyć 'to part.connect' ⇒ 'to turn off'. The former are traditionally considered separate lexemes, and can therefore form VMWEs with their governing verbs. The latter, conversely, are considered inherent components of verbs, and therefore cannot trigger MWE-related considerations.

Similarly, aspect can be realised by various lexical, morphological and syntactic means, and can therefore be seen as either a semantic or a morphological feature (or both). For instance, perfective or continuous aspect can be introduced by inflection and analytical tenses: (EN) *is doing, has done*. Starting, continuation, completion and perfective aspect can also be expressed by specific verbs modifying other verbs: (EN) *to start/continue/stop/complete the action*. Finally, in Slavic languages each verbal lexeme (i.e. independently of its inflected form), has inherent aspect, either perfective or imperfective, and is marked as a morphological feature (recognisable either by a prefix or by an ending): (PL) robić 'to do.imperf'
vs. z.robić ‘to PART.do.PERF’; wy.łączać ‘to PART.connect.IMPERF’ ⇒ ‘to turn off’ vs. wy.łączyć ‘to PART.connect.PERF’ ⇒ ‘to turn off’. Therefore, in Slavic languages the verb in an LVC necessarily adds aspect to the predicate, so its status in Test 11 (§4.2.2) should be examined along slightly different lines than in Romance and Germanic languages. Additionally, if adding any aspectual semantics to the predicate should necessarily block the LVC classification in Test 11, then (EN) *to take a decision* should be annotated as an LVC, while (EN) *taking a decision* might not. These observations led us to revise the LVC tests for future editions of the guidelines.

Another finding concerns productivity. Some verbs admit arguments from large semantic classes, and, conversely, some nouns select various verbal operators. More precisely, we observed the hardness of delimiting productive from non-productive cases in VMWE categories: (i) whose semantic non-compositionality is weak, or (ii) whose components are not content words. The former mainly concerns LVCs. We found no effective and reproducible way to distinguish lexical selection from selection of large semantic classes. For instance, (EN) *to deliver* is often used with the class of nouns expressing formal speech acts such as *speech*, *lecture*, *verdict*, etc. However, we can also use the verb *to give* instead of *to deliver* with the same class of nouns, which likely shows a productive rather than a strict lexical selection. Problem (ii) concerns VPCs, IReflVs and prepositional verbs. Namely, as the semantics of particles is hard to establish, we could come up with only one VPC-related test (§4.2.5), which should clearly evolve in future work. Also, the ambiguity of various uses of the reflexive clitic, and the resulting hardness of the IReflV annotation, was stressed by many language teams. Finally, the non-compositionality of prepositional verbs was so hard to establish in the pilot annotation that we abandoned them in the final annotation.

We also underestimated the importance of modelling not only the semantic non-compositionality of idioms but their conventionalisation as well. As a result, we currently have no efficient way to distinguish MWEs from metaphors. The resemblance is strong since many idioms are metaphors, e.g. (PT) *ele abre mão* ‘he opens hand’ ⇒ ‘he gives up’, but non-idiomatic metaphors, created for the need of a particular text, do occur, e.g. (PL) *podpisanie tej umowy to stryczek zalożony na szyję Polski* ‘signing this treaty is a noose put around Poland’s neck’. The difference is hard to tackle, and especially to test, since it seems to lie precisely in the fact that MWEs are conventionalised while metaphors are not necessarily so. A partial solution to this problem may probably stem from statistical estimations, although the “long tail” of conventionalised and still infrequent MWEs may largely resemble non-conventionalised metaphors. We put forward the MWE vs. metaphor distinction as a future research issue.
# 9 Related work

In this section we contextualise our work with respect to existing MWE typologies, annotation methodologies and annotated corpora.

## 9.1 MWE typologies

In previous approaches to modelling MWEs, various classifications of MWEs were put forward. Here, we focus on several proposals, summarised in Table 7, which seem relevant to our work in that they: (i) have been particularly influential in the NLP community (Sag et al. 2002; Baldwin & Kim 2010; Mel'čuk 2010) (ii) were tested against a representative data set (Mel'čuk 2010), notably in corpus annotation (Schneider et al. 2014), (iii) use MWE flexibility, which is a pervasive feature of verbal MWEs, as a major classification criterion (Sag et al. 2002), (iv) focus exclusively on verbal MWEs (Sheinfux et al. forthcoming), (v) put a verbal component in the heart of the classification criterion (Laporte 2018).

Sag et al. (2002) is a highly influential seminal work whose MWE classification implements the hypothesis put forward by Nunberg et al. (1994) about the correlation between the semantic decomposability of an idiom and its syntactic flexibility. According to this theory, it is because *pull* can be rephrased as *use* and *strings* as *one's influence* that the idiom *to pull strings* admits variations like *to pull all the (political) strings*, *the strings he pulled*, etc. The hypothesis has been criticised, e.g. by Sheinfux et al. (forthcoming) and Laporte (2018), notably by demonstrating non-decomposable MWEs which still exhibit flexibility. The Sag et al. (2002) classification also calls for adjustments in inflectionally rich and free-word-order languages. Still, it remains widely used, notably due to its usefulness for NLP applications. Namely, MWE flexibility is a major obstacle in MWE identification since it prohibits seeing a MWE as a “word with spaces” and using sequence labelling approaches.

Baldwin & Kim (2010) assume the flexibility-driven classification by Sag et al. (2002) and they additionally introduce an orthogonal typology based on purely syntactic criteria, that is, on the syntactic structure of the MWE. There, verbal subcategories are both English-specific and non-exhaustive since verb-noun idioms are considered, but not, for example, verb-adjective ones.

The typology of Mel'čuk (2010) is based, conversely, on mainly semantic criteria. Different types of semantic compositionality are defined, and non-compositional subtypes are those where the semantic head is missing. The latter further subdivide into: (i) *quasi-locutions* in which the meanings of the components are combined, as in (FR) *donner le sein* 'to give the breast' ⇒ 'to breastfeed', (ii)
<table><caption>Table 7: Various MWE classifications compared.</caption><thead><tr><th>Reference</th><th>Language</th><th>Scope</th><th>Classes</th><th># classified expressions</th><th>Defining criteria</th></tr></thead><tbody><tr><td>Sag et al. (2002)</td><td>EN</td><td>MWEs and collocations</td><td>I. Lexicalised: 1. Fixed (by and large); 2. Semi-fixed: non-decomposable idioms (shoot the breeze 'chat'), compound nominals (part of speech), proper names (San Francisco 49ers); 3. Syntactically-flexible: VPCs (break up), decomposable idioms (spill the beans), LVCs (make a decision); II. Institutionalised (traffic lights)</td><td>unknown</td><td>lexicalisation, morphological and syntactic flexibility, semantic decomposability</td></tr><tr><td>Baldwin & Kim (2010)</td><td>EN</td><td>MWEs and collocations</td><td>I. Nominal (golf club, connecting flight); II. Verbal: 1. VPCs (take off, cut short, let go); 2. Prepositional verbs (come across); 3. LVCs (take a walk); 4. Verb-noun idioms (shoot the breeze); III. Prepositional: 1. Determinerless prepositional phrases (on top, by car); 2. Complex prepositions (on top of, in addition to)</td><td>unknown</td><td>syntactic structure</td></tr><tr><td>Melčuk (2010)</td><td>FR</td><td>MWEs and collocations</td><td>I. Pragmatic (emphasis mine); II. Semantic: 1. Semantically compositional: 4.400 clichés (in other words), collocations (busy as a bee, award a prize); 2. Semantically non-compositional: quasi-locations (FR, donner le sein 'give the breast' ⇒ 'breastfeed'), 2. Semi-locations (FR, fruits de mer 'sea fruit' ⇒ 'seafood'), 3. Complete locations (FR, en tenue d'Adam et Eve 'in Adam's and Eve's dress' ⇒ 'naked')</td><td>4.400 collocations, 3.200 locutions (Pausé 2017)</td><td>selection constraints, semantic non-compositionality</td></tr><tr><td>Schneider et al. EN (2014)</td><td></td><td>all MWEs</td><td>I. Strong (close call); II. Weak (narrow escape)</td><td>3,500 occurrences</td><td>strength of association between words</td></tr><tr><td>Sheinfux et al. HE (forthcoming)</td><td></td><td>verbal idioms</td><td>I. Transparent figurative (saw logs 'snore'); II. Opaque figurative (shoot the breeze 'chat'); III. Opaque non-figurative (take umbrage 'feel offended')</td><td>15 VMWEs, 400 occurrences</td><td>transparency, figuration</td></tr><tr><td>Laporte (2018)</td><td>FR</td><td>MWEs and collocations</td><td>I. Lexicalised: 1. MWEs without support verbs: verbal (take stock), nominal (traffic lights), adverbial (for instance); 2. Support-verb constr.: a. Vsup is not copula (have an aim, get loose), b. Vsup in copula (be a genius, be angry, be on time); II. Non-lexicalised (salt and pepper)</td><td>dozens of thousands of (lexicalised) MWEs</td><td>lexicalisation, presence of a support verb</td></tr><tr><td>This chapter</td><td>BG, CS, DE, EL, verbal ES, FA, FR, HE, MWEs HU, IT, LT, MT, PL, PT, RO, SL, SV, TR</td><td></td><td>I. Universal: LVCs (make a decision), IDs (spill the beans); II. Quasi-universal: ReFlVs (FR) s'averer 'reFL 'reveal' ⇒ 'prove (to be)'); III. occurrences OTH (drink and drive, to voice act)</td><td>62,000</td><td>universalism, syntactic structure, lexical, syntactic and semantic idiosyncrasy</td></tr></tbody></table>
*semi-locutions* which include the meaning of only a part of their components, as in (FR) *fruits de mer* ‘sea fruit’ ⇒ ‘seafood’, (iii) *complete locutions*, which include the meaning of none of their components, as in (FR) *en tenue d’Adam et Eve* ‘in Adam’s and Eve’s dress’ ⇒ ‘naked’.

Schneider et al. (2014) propose a rather shallow typology with only two types based on the strength of association between component words. Strong MWEs are those whose meaning is not readily predictable from component words, as in (EN) *close call* ‘a situation in which something bad almost happened but could be avoided’. Weak MWEs are those with more transparent semantics and more flexibility, like (EN) *narrow escape* ‘a situation in which something bad almost happened but could be avoided’. This typology was applied to annotate a large publicly available corpus, underlying the DiMSUM<fnref n="30" /> shared task on identification of minimal semantic units and their supersenses.

In Sheinfux et al. (forthcoming) the hypothesis of Nunberg et al. (1994) is questioned on a sample of verbal Hebrew idioms, and a novel classification is put forward which relies on figuration (the degree to which the idiom can be assigned a literal meaning) and transparency (the relationship between the literal and idiomatic reading). In *transparent figurative* idioms the relationship between the literal and the idiomatic reading is easy to recover (*to saw logs* ‘snore’). In *opaque figurative* idioms the literal picture is easy to imagine but its relationship to the idiomatic reading is unclear (*to shoot the breeze* ‘chat’). Finally, in *opaque non-figurative* idioms no comprehensible literal meaning is available, notably due to cranberry words which have no status as individual lexical units (*to take umbrage* ‘to feel offended’). The study further tests VMWEs of the 3 categories against 4 types of lexical and syntactic flexibility, and stresses the fact that flexibility is a matter of scale rather than a binary property.

Laporte (2018) formalises a MWE classification emerging from the lexicon-grammar theory and encoding practice (Gross 1986; 1994). Its specificity is to put the notion of support verb (roughly equivalent to light verb) in the heart of the classification, and push the MWE frontier far beyond what is admitted in other approaches. Namely, with the copula support verb *to be*, large classes of nouns, adjectives and PPs are seen as predicates of support-verb constructions, which should, thus, be lexically described.

Comparing our classification (§3) to the above ones (Table 7), several facts are striking: (i) we restrict ourselves to verbal MWEs only, (ii) we perform a large-scale multilingual evaluation and enhancement of the classification via corpus annotation in 18 languages, (iii) we assess semantic non-compositionality via

<footnote n="30">³⁰https://dimsum16.github.io/</footnote>
mostly syntactic tests, (iv) we define a novel VMWE category of IReflVs and linguistic tests delimiting its borders, we also display the quantitative importance of this category, mainly in Romance and Slavic languages, (v) we give access to detailed annotation guidelines organised as decision trees, with linguistic tests illustrated in many languages. As far as the scope of the MWE-related phenomena are concerned, recall that we exclude statistical collocations and retain only lexically, syntactically or semantically idiosyncratic expressions. This fact seemingly contrasts with other approaches shown in Table 7. Note, however, that some of these authors understand collocations differently, as discussed in §2.

## 9.2 MWE annotation practices

Modelling the behaviour of MWEs in annotated corpora, and prominently in treebanks, has been undertaken in various languages and linguistic frameworks. Rosén et al. (2015) offer a survey of MWE annotation in 17 treebanks for 15 languages, collaboratively documented according to common guidelines.<fnref n="31" /> According to this survey, multiword named entities constitute by far the most frequently annotated category (Erjavec et al. 2010), sometimes with elaborate annotation schemes accounting for nesting and coordination (Savary et al. 2010). Continuous MWEs such as compound nouns, adverbs, prepositions and conjunctions are also covered in some corpora (Abeillé et al. 2003; Laporte et al. 2008; Branco et al. 2010). Verbal MWEs have been addressed for fewer languages. The survey also shows the heterogeneity of MWE annotation practices. For instance, VPCs are represented in dependency treebanks by dedicated relations between head verbs and particles. In constituency treebanks, particles constitute separate daughter nodes of sentential or verbal phrases and are assigned categories explicitly indicating their status of selected particles. Additionally, in an LFG (Lexical Functional Grammar) treebank, verbs and their particles are merged into single predicates appearing in functional structures.

Similar conclusions about the heterogeneity of MWE annotation were drawn concerning UD (McDonald et al. 2013), an initiative towards developing syntactically full-fledged and cross-linguistically consistent treebank annotation for many languages. Nivre & Vincze (2015) show that LVCs annotation in UD treebanks is threefold: (i) some treebanks lack or do not distinguish LVCs from regular verb-object pairs, (ii) some distinguish them by their structure (the direct object is dependent on the light verb rather than on the predicative noun), (iii) some account for them explicitly by the dependency labels between the noun

<footnote n="31">³¹http://clarino.uib.no/iness/page?page-id=MWEs_in_Parseme</footnote>
and the verb. Furthermore, De Smedt et al. (2015) point out that 3 different dependency relations in UD<fnref n="32" /> can be used to describe MWEs - *compound*, *mwe* and *name* (with possible sub-relations, e.g. *compound:prt* for verb-particle constructions) - and that these are used across different UD treebanks in a largely inconsistent way. More recent efforts (Adalı et al. 2016), while addressing VMWEs in a comprehensive way, still suffer from missing annotation standards.

As compared to this state of the art, the PARSEME effort aims at developing annotation guidelines and practices which would be universal but would leave room for language-dependent specificities. Our scope covers all types of VMWEs.

## 9.3 Corpora and datasets with VMWEs

As seen in the previous section, most efforts towards anotating MWEs were either language- or MWE category-specific. The same holds for verbal MWEs in particular. In this section we mention some outcomes of the previous VMWE annotation initiatives.

The Wiki50 (Vincze et al. 2011) corpus contains 50 English Wikipedia articles annotated for MWEs, including several VMWEs types. The dataset of Tu & Roth (2011) consists of 2,162 sentences from the British National Corpus in which verb-object pairs formed with *do*, *get*, *give*, *have*, *make*, and *take* are marked as positive and negative examples of LVCs. Tu & Roth (2012) built a crowdsourced corpus in which VPCs are manually distinguished from compositional verb-preposition combinations, again for six selected verbs. Baldwin (2005) presents another dataset of English VPCs. Finally, SZPFX (Vincze 2012) is an English-Hungarian parallel corpus with LVC annotations in both languages. For German, idiomatic combinations of verbs and prepositional phrases were described in a database by Krenn (2008) and annotated in the TIGER corpus by Brants et al. (2005).

In Slavic languages, a notable effort was made with the Prague Dependency Treebank of Czech (Hajič et al. 2017), annotated at 3 layers: morphological, analytical (accounting for syntax) and tectogrammatical (accounting for functional relations). MWEs, including some VMWEs, are annotated by identifying monosemic subtrees in the 3rd layer and replacing them by single nodes (Bejček & Straňák 2010), which unifies different morphosyntactic variants of the same MWE (Bejček et al. 2011). Each MWE occurrence is linked to its entry in an associated MWE lexicon. It is also argued that elements elided in MWEs (e.g. due to coordination) should be restored in deep syntactic trees. The Czech PARSEME corpus results from a mostly automatic (although challenging) transformation of the PDT annotations into the parseme-tsv format (Bejček et al. 2017).

<footnote n="32">³²This analysis concerns UD v1 - these labels evolved in UD v2.</footnote>
Kaalep & Muischnek (2006; 2008) and Vincze & Csirik (2010) present databases and corpora of VMWEs for Estonian particle verbs and Hungarian LVCs, respectively. VMWE annotations are available in several Turkish treebanks. In Eryiğit et al. (2015) various MWEs are labeled with a unique dependency label independently of their category, while in Adalı et al. (2016) they are classified as either strong or weak, similarly to Schneider et al. (2014). Finally, QasemiZadeh & Rahimi (2006) provide annotations for Farsi LVCs in the framework of the MULTEXT-East initiative, and in the Uppsala Persian Dependency Treebank (Seraji et al. 2014) the *lvc* dependency relationship is used for annotating non-verbal component of Farsi LVCs that are not in any other type of syntactic relationship.

The PARSEME corpus initiative builds upon these previous efforts by incorporating and extending some pre-existing datasets and annotation experiences. In some languages it is novel in that: (i) it constitutes the first attempt to annotate and analyse VMWEs in running text, e.g. in Greek and Maltese, (ii) it pays special attention, for the first time, to certain VMWE categories, e.g. to VPCs in Greek, to LVCs in Lithuanian, to IReflVs in most Slavic and Romance languages, and to distinguishing VMWEs from semi-copula-based expressions in Farsi (§4.3). But the most notable achievement going beyond the state of the art is to offer the first large highly multilingual VMWE corpus annotated according to unified guidelines and methodologies.

# 10 Conclusions and future work

We described the results of a considerable collective effort towards setting up a common framework for annotating VMWEs in 18 languages from 9 different language families. Unlike McDonald et al. (2013), our methodology is not English-centred. We draft the guidelines and test them on many languages in parallel, without giving priority to any of them (except for communication purposes). We offer a classification of VMWEs where properties hypothesised as universal or quasi-universal are treated in a homogeneous way, while leaving room to language-specific categories and features at the same time. Additionally to its importance for language modelling, and contrastive linguistic studies, this typology may be useful for various language technology tasks, notably because different VMWE types show different degrees of semantic decomposability, which influences their interpretation and translation. For instance, in LVCs nouns may translate literally and verbs may be omitted in the semantic calculus, but the same usually does not hold for IDs. Our annotation guidelines are organised in decision trees, so as to maximise the replicability of the annotators’ decisions.
Our efforts also pave the way towards unified terminology and notation con-
ventions. In particular, we stress the relations between words and tokens, which
are crucial for defining the scope of the MWE phenomenon. We formalise the
notion of a canonical form of a VMWE. Moreover, the notational conventions
used in this volume for citing, glossing and translating multilingual examples of
VMWEs largely result from our documentation work.

The PARSEME VMWE corpus<fnref n="33" /> and its annotation guidelines,<fnref n="34" /> both avail-
able under open licenses, are meant as dynamic resources, subject to continu-
ous enhancements and updates. The size of the corpus is still modest for many
languages and should be progressively increased. Adopting higher annotation
standards, including a double annotation and adjudication, would lead to more
reliable guidelines, increase the quality of the data, and strengthen our claims
and findings. Since the publication of version 1.0 of the corpus, rich feedback
was gathered from language teams, several dozens of issues were formulated and
were discussed in a dedicated Gitlab space<fnref n="35" /> and version 1.1³⁶ of the guidelines
was elaborated. The most important evolutions include:

* Abandoning the category-neutral identification stage, since the annotation practice showed that VMWE identification is virtually always done in a category-specific way. The previous identification tests become ID-specific tests.

* Abandoning the OTH category due to its very restricted use. VMWEs classified previously as OTH now enter the ID category (except when the interpretation of the OTH category was language-specific).

* Introducing the multverb construction (MVC) category to account for idiomatic serial verbs in Asian languages such as Hindi, Indonesian, Japanese and Chinese.

* Redesigning the tests and the decision trees for the LVC and VPC category, so as to increase the determinism in the annotation of these two categories.

* Introducing – optionally and experimentally – the category of inherently adpositional verbs (IAVs), roughly equivalent to the previously abandoned

<footnote n="33">³³http://hdl.handle.net/11372/LRT-2282</footnote>

<footnote n="34">³⁴http://parsemefr.lif.univ-mrs.fr/parseme-st-guidelines/1.0/</footnote>

<footnote n="35">³⁵https://gitlab.com/parseme/sharedtask-guidelines/issues (restricted access, new users are welcome upon registration with the project leaders)</footnote>

<footnote n="36">³⁶http://parsemefr.lif.univ-mrs.fr/parseme-st-guidelines/1.1/</footnote>
inherently prepositional verbs (IPrepVs). The IAV should be addressed in
the post-annotation step, i.e. once the VMWEs of all other categories have
been identified.

* Renaming the IReflV category by IRV, for an easier pronunciation.

* Renaming the ID category to VID (verbal idiom), to explicitly account for the verbal-only scope.

Adjustments of the previously annotated corpus to the guidelines version 1.1 are ongoing. The corpus should also significantly grow, as new portions of data are being annotated and new language teams (Arabic, Basque, Croatian, English and Hindi) are joining the project. Edition 1.1 of the PARSEME shared task (cf. Savary et al. 2017 for edition 1.0), based on the enhanced guidelines and corpus, is taking place as this volume is being edited.

In the long run, we intend to include other categories of MWEs (nominal, adjec-
tival, adverbial, prepositional, named entities, etc.) under the annotation scope,
as well as pave the way towards consistent representation and processing of both
MWEs and syntax.

Acknowledgments

The work described in this chapter was supported by: (i) the IC1207 PARSEME COST action,<fnref n="37" />; (ii) national funded projects: LD-PARSEME<fnref n="38" /> (LD14117) in the Czech Republic, PARSEME-FR<fnref n="39" /> (ANR-14-CERA-0001) in France, and PASTO-VU<fnref n="40" /> in Lithuania; (iii) European Union’s Horizon 2020 research and innovation programme (Marie Skłodowska-Curie grant No 713567); (iv) Science Foundation Ireland in the ADAPT Centre<fnref n="41" /> (Grant 13/RC/2106) at Dublin City University.

We are grateful to all language teams for their contributions to preparing the
annotation guidelines and the annotated corpora. The full composition of the
annotation team is the following.

Balto-Slavic languages:

* (BG) Ivelina Stoyanova (LGL, LL), Tsvetana Dimitrova, Svetla Koeva, Svetlozara Leseva, Valentina Stefanova, Maria Todorova;

<footnote n="37">³⁷http://www.parseme.eu</footnote>

<footnote n="38">³⁸https://ufal.mff.cuni.cz/grants/ld-Parseme</footnote>

<footnote n="39">³⁹http://parsemefr.lif.univ-mrs.fr/</footnote>

<footnote n="40">⁴⁰http://mwe.lt/en_US/</footnote>

<footnote n="41">⁴¹www.adaptcentre.ie</footnote>
• (CS) Eduard Bejček (LL), Zdeňka Urešová, Milena Hnátková;

• (LT) Jolanta Kovalevskaitė (LL), Loic Boizou, Erika Rimkutė, Ieva Bumbulienė;

• (SL) Simon Krek (LL), Polona Gantar, Taja Kuzman;

• (PL) Agata Savary (LL), Monika Czerepowicka.

Germanic languages:

• (DE) Fabienne Cap (LGL, LL), Glorianna Jagfeld, Agata Savary;

• (EN) Ismail El Maarouf (LL), Teresa Lynn, Michael Oakes, Jamie Findlay, John McCrae, Veronika Vincze;

• (SV) Fabienne Cap (LL), Joakim Nivre, Sara Stymne.

Romance languages:

• (ES) Carla Parra Escartín (LL), Cristina Aceta, Itziar Aduriz, Uxoa Iñurrieta, Carlos Herrero, Héctor Martínez Alonso, Belem Priego Sanchez;

• (FR) Marie Candito (LGL, LL), Matthieu Constant, Ismail El Maarouf, Carlos Ramisch (LGL), Caroline Pasquer, Yannick Parmentier, Jean-Yves Antoine;

• (IT) Johanna Monti (LL), Valeria Caruso, Manuela Cherchi, Anna De Santis, Maria Pia di Buono, Annalisa Raffone;

• (RO) Verginica Barbu Mititelu (LL), Monica-Mihaela Rizea, Mihaela Ionescu, Mihaela Onofrei;

• (PT) Silvio Ricardo Cordeiro (LL), Aline Villavicencio, Carlos Ramisch, Leonardo Zilio, Helena de Medeiros Caseli, Renata Ramisch;

Other languages:

• (EL) Voula Giouli (LGL,LL), Vassiliki Foufi, Aggeliki Fotopoulou, Sevi Louisou;

• (FA) Behrang Qasemi Zadeh (LL);

• (HE) Chaya Liebeskind (LL), Yaakov Ha-Cohen Kerner (LL), Hevi Elyovich, Ruth Malka;

• (HU) Veronika Vincze (LL), Katalin Simkó, Viktória Kovács;

• (MT) Lonneke van der Plas (LL), Luke Galea (LL), Greta Attard, Kirsty Azzopardi, Janice Bonnici, Jael Busuttil, Ray Fabri, Alison Farrugia, Sara Anne Galea, Albert Gatt, Anabelle Gatt, Amanda Muscat, Michael Spagnol, Nicole Tabone, Marc Tanti;

• (TR) Kübra Adalı (LL), Gülşen Eryiğit (LL), Tutkum Dinç, Ayşenur Miral, Mert Boz, Umut Sulubacak.
We also thank Mozhgan Neisani from University of Isfahan and Mojgan Seraji from the Uppsala Universitet for their contribution to the inter-annotator agreement calculation.

## Abbreviations

<table><tr><td>FUT</td><td>future</td><td>MTW</td><td>multitoken word</td></tr><tr><td>GEN</td><td>genitive</td><td>MWT</td><td>multiword token</td></tr><tr><td>IAA</td><td>inter-annotator-agreement</td><td>NLP</td><td>natural language processing</td></tr><tr><td>ID</td><td>idiom</td><td>OTH</td><td>other VMWEs</td></tr><tr><td>IREFLV</td><td>inherently reflexive verb</td><td>PART</td><td>particle</td></tr><tr><td>LGL</td><td>language group leader</td><td>REFL</td><td>reflexive clitic</td></tr><tr><td>LL</td><td>language leader</td><td>SG</td><td>singular</td></tr><tr><td>LV</td><td>light verb</td><td>UD</td><td>Universal Dependencies</td></tr><tr><td>LVC</td><td>light-verb construction</td><td>VID</td><td>verbal idiom</td></tr><tr><td>MAD</td><td>mean absolute deviation</td><td>VMWE</td><td>verbal multiword expression</td></tr><tr><td>MASC</td><td>masculine</td><td>VPC</td><td>verb-particle construction</td></tr><tr><td>MWE</td><td>multiword expression</td><td>1, 2, 3</td><td>first, second, third person</td></tr></table>

## References

Abeillé, Anne, Lionel Clément & François Toussenel. 2003. Building a treebank for French. In Anne Abeillé (ed.), *Treebanks: Building and using parsed corpora*, 165–187. Dordrecht, The Netherlands: Kluwer Academic Publishers.

Adalı, Kübra, Tutkum Dinç, Memduh Gokirmak & Gülşen Eryiğit. 2016. Comprehensive annotation of multiword expressions for Turkish. In *TurCLing 2016, The First International Conference on Turkic Computational Linguistics at CLING 2016*.

Al Saied, Hazem, Marie Candito & Matthieu Constant. 2018. A transition-based verbal multiword expression analyzer. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 209–226. Berlin: Language Science Press. DOI:10.5281/zenodo.1469561

Baggio, Giosuè, Michiel van Lambalgen & Peter Hagoort. 2012. The processing consequences of compositionality. In *The Oxford handbook of compositionality*, 655–672. New York: Oxford University Press.

Baldwin, Timothy. 2005. Deep lexical acquisition of verb-particle constructions. *Computer Speech and Language* 19(4). 398–414. DOI:10.1016/j.csl.2005.02.004
Baldwin, Timothy & Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya & Fred J. Damerau (eds.), *Handbook of Natural Language Processing, Second edition*, 267–292. Boca Raton: CRC Press.

Bauer, Laurie. 1983. *English word-formation* (Cambridge Textbooks in Linguistics). Cambridge: Cambridge University Press. https://books.google.pl/books?id=yGfUHs6FCvIC.

Bejček, Eduard, Jan Hajič, Pavel Straňák & Zděnska Urěnsová. 2017. Extracting verbal multiword data from rich treebank annotation. In *Proceedings of the 15th International Workshop on Treebanks and Linguistic Theories* (TLT 15), 13–24.

Bejček, Eduard & Pavel Straňák. 2010. Annotation of multiword expressions in the prague dependency treebank. *Language Resources and Evaluation* 44(1–2). 7–21.

Bejček, Eduard, Pavel Straňák & Daniel Zeman. 2011. Influence of treebank design on representation of multiword expressions. In Alexander F. Gelbukh (ed.), *Computational Linguistics and intelligent text processing - 12th International Conference*, (CICLing 2011), Tokyo, Japan, February 20-26, 2011. *Proceedings, Part I*, vol. 6608 (Lecture Notes in Computer Science), 1–14. Springer. DOI:10.1007/978-3-642-19400-9_1

Branco, António, Francisco Costa, João Silva, Sara Silveira, Sérgio Castro, Maria Avelãs, Clara Pinto & João Graça. 2010. Developing a deep linguistic databank supporting a collection of treebanks: The CINTIL DeepGramBank. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner & Daniel Tapias (eds.), *Proceedings of the 7th conference on international language resources and evaluation* (LREC 2010). European Language Resources Association (ELRA).

Brants, Sabine, Stefanie Dipper, Peter Eisenberg, Silvia Hansen-Schirra, Esther König, Wolfgang Lezius, Christian Rohrer, George Smith & Hans Uszkoreit. 2005. TIGER: Linguistic interpretation of a German corpus. *Research on Language and Computation* 2(4). 597–620. DOI:10.1007/s11168-004-7431-3

de Marneffe, Marie-Catherine, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre & Christopher D. Manning. 2014. Universal Stanford dependencies: A cross-linguistic typology. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asunción Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the 9th International Conference on Language Resources and Evaluation* (LREC 2014), 4585–4592. European Language Resources Association (ELRA).

De Smedt, Koenraad, Victoria Rosén & Paul Meurer. 2015. *MWEs in universal dependency treebanks*. IC1207 COST PARSEME 5th general meeting. Iași, Ro-
mania. http://typo.uni-konstanz.de/parseme/index.php/2-general/138-admitted-posters-iasi-23-24-september-2015.

Erjavec, Tomaz, Darja Fiser, Simon Krek & Nina Ledinek. 2010. The JOS linguistically tagged corpus of Slovene. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner & Daniel Tapias (eds.), *Proceedings of the 7th International Conference on Language Resources and Evaluation* (LREC 2010), 1806–1809. European Language Resources Association (ELRA).

Eryiğit, Gülşen, Kübra Adali, Dilara Torunoğlu-Selamet, Umut Sulubacak & Tuğba Pamay. 2015. Annotation and extraction of multiword expressions in Turkish treebanks. In *Proceedings of the 11th Workshop on Multiword Expressions* (MWE ’15), 70–76. Association for Computational Linguistics. http://www.aclweb.org/anthology/W15-0912.

Fort, Karën & Benoît Sagot. 2010. Influence of pre-annotation on POS-tagged corpus development. In *Proceedings of the 4th Linguistic Annotation Workshop* (LAW IV ’10), 56–63. Association for Computational Linguistics. http://dl.acm.org/citation.cfm?id=1868720.1868727.

Gross, Gaston. 1988. Degré de figement des noms composés. *Langages* 90. 57–71.

Gross, Maurice. 1986. Lexicon-grammar: The representation of compound words. In *Proceedings of the 11th conference on computational linguistics (COLING ’86)*, 1–6. Association for Computational Linguistics. DOI:10.3115/991365.991367

Gross, Maurice. 1994. The lexicon-grammar of a language: Application to French. In Ashley R. E. (ed.), *The encyclopedia of language and linguistics*, 2195–2205. Oxford: Oxford/NewYork/Seoul/Tokyo: Pergamon. https://hal-upec-upem.archives-ouvertes.fr/hal-00621380.

Hajič, Jan, Eva Hajičová, Marie Mikulová & Jiří Mírovský. 2017. Prague Dependency Treebank. In *Handbook on Linguistic Annotation* (Springer Handbooks), 555–594. Berlin, Germany: Springer Verlag.

Janssen, Theo M. V. 2001. Frege, contextuality and compositionality. *Journal of Logic, Language and Information* 10(1). 115–136. DOI:10.1023/A:1026542332224

Jespersen, Otto. 1965. *A Modern English grammar on historical principles, Part VI, Morphology*. London: Allen & Unwin.

Kaalep, Heiki-Jaan & Kadri Muischnek. 2006. Multi-word verbs in a flective language: The case of Estonian. In *Proceedings of the EACL Workshop on Multi-Word Expressions in a Multilingual Context* (MWE ’06), 57–64. Association for Computational Linguistics. http://www.aclweb.org/anthology/W/W06/W06-2400.pdf.
Kaalep, Heiki-Jaan & Kadri Muischnek. 2008. Multi-word verbs of Estonian: A database and a corpus. In *Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions* (MWE '08), 23–26. Association for Computational Linguistics. http://www.lrec-conf.org/proceedings/lrec2008/workshops/W20_Proceedings.pdf.

Kim, Su Nam. 2008. *Statistical modeling of multiword expressions*. Melbourne: University of Melbourne dissertation.

Koeva, Svetla, Ivelina Stoyanova, Maria Todorova & Svetlozara Leseva. 2016. Semi-automatic compilation of the dictionary of Bulgarian multiword expressions. In *Proceedings of the Workshop on Lexicographic Resources for Human Language Technology* (GLOBALEX 2016), 86–95.

Kracht, Marcus. 2007. Compositionality: The very idea. *Research on Language and Computation* 5(3). 287–308. DOI:10.1007/s11168-007-9031-5

Krenn, Brigitte. 2008. Description of evaluation resource – German PP-verb data. In *Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions* (MWE '08), 7–10. Association for Computational Linguistics.

Laporte, Éric. 2018. Choosing features for classifying multiword expressions. In Manfred Sailer & Stella Markantonatou (eds.), *Multiword expressions: Insights from a multi-lingual perspective* (Phraseology and Multiword Expressions). Language Science Press.

Laporte, Éric, Takuya Nakamura & Stavroula Voyatzi. 2008. A French corpus annotated for multiword expressions with adverbial function. In *Proceedings of the 2nd Linguistic Annotation Workshop*, 48–51. https://halshs.archives-ouvertes.fr/halshs-00286541.

Lipka, Leonhard, Susanne Handl & Wolfgang Falkner. 2004. Lexicalization & institutionalization. The state of the art in 2004. *SKASE Journal of Theoretical Linguistics* 1(1). 2–19. http://www.skase.sk/Volumes/JTL01/lipka.pdf.

Maldonado, Alfredo & Behrang QasemiZadeh. 2018. Analysis and Insights from the PARSEME Shared Task dataset. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 149–175. Berlin: Language Science Press. DOI:10.5281/zenodo.1469557

Marcus, Mitchell P., Mary Ann Marcinkiewicz & Beatrice Santorini. 1993. Building a large annotated corpus of English: The penn treebank. *Computational Linguistics* 19(2). 313–330. http://dl.acm.org/citation.cfm?id=972470.972475.

Mathet, Yann, Antoine Widlöcher & Jean-Philippe Métivier. 2015. The unified and holistic method Gamma (γ) for inter-annotator agreement measure and alignment. *Computational Linguistics* 41(3). 437–479. DOI:10.1162/COLI_a_00227
McDonald, Ryan, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló & Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In *Proceedings of the 51st annual meeting of the association for computational linguistics (volume 2: Short papers)*, 92–97. Association for Computational Linguistics. http://www.aclweb.org/anthology/P13-2017.

Mel'čuk, Igor A. 2010. La phraséologie en langue, en dictionnaire et en TALN. In *Actes de la 17ème conférence sur le traitement automatique des langues naturelles 2010*.

Moreau, Erwan, Ashjan Alsulaimani, Alfredo Maldonado, Lifeng Han, Carl Vogel & Koel Dutta Chowdhury. 2018. Semantic reranking of CRF label sequences for verbal multiword expression identification. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 177–207. Berlin: Language Science Press. DOI:10.5281/zenodo.1469559

Nivre, Joakim, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty & Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)*, 1659–1666. European Language Resources Association (ELRA). 23-28 May, 2016.

Nivre, Joakim & Veronika Vincze. 2015. *Light verb constructions in universal dependencies*. IC1207 COST PARSEME 5th general meeting. Iași, Romania. http://typo.uni-konstanz.de/parseme/index.php/2-general/138-admitted-postersiasi-23-24-september-2015.

Nunberg, Geoffrey, Ivan A. Sag & Thomas Wasow. 1994. Idioms. *Language* 70(3). 491–538.

Pagin, Peter & Dag Westerståhl. 2001. Compositionality II: Arguments and problems. *Philosophy Compass* 5. 250–264. DOI:10.1111/j.1747-9991.2009.00228.x

Partee, Barbara H., Alice ter Meulen & Robert E. Wall. 1990. *Mathematical methods in linguistics (Studies in Linguistics and Philosophy 30)*. Dordrecht: Kluwer.
Pasquer, Caroline. 2017. Expressions polylexicales verbales : étude de la variabilité en corpus. In *Actes de la 18e rencontre des étudiants chercheurs en informatique pour le traitement automatique des langues* (TALN-RÉCITAL 2017).

Pausé, Marie-Sophie. 2017. *Structure lexico-sentaxique des locutions du français et incidence sur leur combinatoire*. Nancy, France: Université de Lorraine dissertation.

QasemiZadeh, Behrang & Saeed Rahimi. 2006. Persian in MULTEXT-East framework. In *Advances in Natural Language Processing, 5th International Conference on NLP, FinTAL 2006, Turku, Finland, August 23-25, 2006, Proceedings*, 541–551.
DOI:10.1007/11816508_54

Rosén, Victoria, Gyri Smørdal Losnegaard, Koenraad De Smedt, Eduard Bejček, Agata Savary, Adam Przepiórkowski, Petya Osenova & Verginica Barbu Mitetelu. 2015. A survey of multiword expressions in treebanks. In *Proceedings of the 14th International Workshop on Treebanks & Linguistic Theories Cconference*. https://hal.archives-ouvertes.fr/hal-01226001.

Sag, Ivan A., Timothy Baldwin, Francis Bond, Ann A. Copestake & Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In *Proceedings of the 3rd International Conference on Computational Linguistics and Intelligent Text Processing*, vol. 2276/2010 (CICLing ’02), 1–15. Springer-Verlag.

Savary, Agata, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova & Antoine Doucet. 2017. The PARSEME Shared Task on automatic identification of verbal multiword expressions. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 31–47. Association for Computational Linguistics. DOI:10.18653/v1/W17-1704

Savary, Agata, Jakub Waszczuk & Adam Przepiórkowski. 2010. Towards the annotation of named entities in the National Corpus of Polish. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner & Daniel Tapias (eds.), *Proceedings of the 7th conference on international language resources and evaluation* (LREC 2010). European Language Resources Association (ELRA).

Schneider, Nathan, Spencer Onuffer, Nora Kazour, Nora Emily Danchik, Michael T. Mordowanec, Henrietta Conrad & Smith Noah A. 2014. Comprehensive annotation of multiword expressions in a social web corpus. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the 9th International Conference on Language Resources and Evaluation*.
tion (LREC 2014), 455–461. European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2014/pdf/521_Paper.pdf.

Seraji, Mojgan, Carina Jahani, Beáta Megyesi & Joakim Nivre. 2014. A Persian treebank with Stanford typed dependencies. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the 9th International Conference on Language Resources and Evaluation* (LREC 2014). European Language Resources Association (ELRA).

Sheinfux, Livnat Herzig, Tali Arad Greshler, Nurit Melnik & Shuly Wintner. Forthcoming. Verbal MWEs: Idiomaticity and flexibility. In Yannick Parmen-tier & Jakub Waszczuk (eds.), *Representation and parsing of multiword expres-sions* (Phraseology and Multiword Expressions), 5–38. Berlin: Language Sci-ence Press.

Siemieniec-Gołaś, Ewa. 2010. On some Turkish auxiliary verbs in giovanni molino's dittionario della lingua italiana, turchesca (1641). *Studia Linguistica Universitatis Iagellonicae Cracoviensis* 127(1). 57–77.

Simkó, Katalin Ilona, Viktória Kovács & Veronika Vincze. 2018. Identify-ing verbal multiword expressions with POS tagging and parsing tech-niques. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended pa-pers from the MWE 2017 workshop*, 227–243. Berlin: Language Science Press. DOI:10.5281/zenodo.1469563

Szarvas, György, Veronika Vincze, Richárd Farkas, György Móra & Iryna Gurevych. 2012. Cross-genre and cross-domain detection of semantic uncer-tainty. *Computational Linguistics – Special Issue on Modality and Negation* 38(2). 335–367.

Taslimipoor, Shiva, Omid Rohanian, Ruslan Mitkov & Afsaneh Fazly. 2018. Iden-tification of multiword expressions: A fresh look at modelling and evalu-ation. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended pa-pers from the MWE 2017 workshop*, 299–317. Berlin: Language Science Press. DOI:10.5281/zenodo.1469569

Tu, Yuancheng & Dan Roth. 2011. Learning English light verb constructions: Con-textual or statistical. In *Proceedings of the Workshop on Multiword Expressions: From Parsing and Generation to the Real World* (MWE ’11), 31–39. Association for Computational Linguistics. http://www.aclweb.org/anthology/W11-0807.

Tu, Yuancheng & Dan Roth. 2012. Sorting out the most confusing English phrasal verbs. In *Proceedings of the First Joint Conference on Lexical and Computational
*Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval '12), 65–69.* Association for Computational Linguistics. http://dl.acm.org/citation.cfm?id=2387636.2387648.

van Gompel, Maarten & Martin Reynaert. 2013. FoLia: A practical XML format for linguistic annotation - a descriptive and comparative study. *Computational Linguistics in the Netherlands Journal* 3. 63–81.

van Gompel, Maarten, Ko van der Sloot, Martin Reynaert & Antal van den Bosch. 2017. FoLiA in practice: The infrastructure of a linguistic annotation format. In, (To appear). Ubiquity Press.

Vincze, Veronika. 2012. Light verb constructions in the SzegedParalelFX English–Hungarian parallel corpus. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk & Stelios Piperidis (eds.), *Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC-2012), 2381–2388*. European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2012/pdf/177_Paper.pdf.

Vincze, Veronika & János Csirik. 2010. Hungarian corpus of light verb constructions. In *Proceedings of the 23rd International Conference on Computational Linguistics* (COLING '10), 1110–1118. Association for Computational Linguistics. http://www.aclweb.org/anthology/C10-1125.

Vincze, Veronika, István Nagy T. & Gábor Berend. 2011. Multiword expressions and named entities in the Wiki50 corpus. In *Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, 289–295*. RANLP 2011 Organising Committee. http://aclweb.org/anthology/R11-1040.
<figure><img src="image_26.png" /></figure>
# Chapter 5

## Analysis and Insights from the PARSEME Shared Task dataset

### Alfredo Maldonado

ADAPT Centre, Trinity College Dublin

### Behrang QasemiZadeh

University of Düsseldorf

The PARSEME Shared Task on the automatic identification of verbal multiword expressions (VMWEs) was the first collaborative study on the subject to cover a wide and diverse range of languages. One observation that emerged from the official results is that participating systems performed similarly on each language but differently across languages. That is, intra-language evaluation scores are relatively similar whereas inter-language scores are quite different. We hypothesise that this pattern cannot be attributed solely to the intrinsic linguistic properties in each language corpus, but also to more practical aspects such as the evaluation framework, characteristics of the test and training sets as well as metrics used for measuring performance. This chapter takes a close look at the shared task dataset and the systems' output to explain this pattern. In this process, we produce evaluation results for the systems on VMWEs that only appear in the test set and contrast them with the official evaluation results, which include VMWEs that also occur in the training set. Additionally, we conduct an analysis aimed at estimating the relative difficulty of VMWE detection for each language. This analysis consists of a) assessing the impact on performance of the ability, or lack-thereof, of systems to handle discontinuous and overlapped VMWEs, b) measuring the relative sparsity of sentences with at least one VMWE, and c) interpreting the performance of each system with respect to two baseline systems: a system that simply tags every verb as a VMWE, and a dictionary lookup system. Based on our data analysis, we assess the suitability of the official evaluation methods, specifically the token-based method, and propose to use Cohen's kappa score as an additional evaluation method.

<figure><img src="image_27.png" /></figure>
# 1 Introduction

Multiword expressions (MWEs) have been studied extensively due to the fact that many natural language processing (NLP) pipelines depend on their correct identification and processing (Sag et al. 2002). However, there has been relatively little work on *Verbal* MWEs (VMWEs). The PARSEME<fnref n="1" /> Shared Task on VMWEs (Savary et al. 2017) was the first initiative focusing on the problem of identifying VMWEs for a relatively large number of languages, 18 in total. This initiative produced an array of annotated training and test sets for each language. Using these training sets, shared task participants developed and trained VMWE-identification systems, which were then evaluated on separate test sets also produced by PARSEME.

Several patterns have emerged from the evaluation results in this pioneering shared task. One is that individual systems tend to perform very differently across languages (inter-language performance) and yet different systems performed similarly in most languages (intra-language performance). In particular, participating systems scored highest on Farsi, Romanian, Czech and Polish, and lowest on Swedish, Hebrew, Lithuanian and Maltese, whilst ranging somewhere in between for the rest of the languages. It has been observed that the inter-language performance is positively correlated with the proportion of VMWEs shared by the training and test sets in each language (Maldonado et al. 2017). This observation suggests that the reported systems' performance and ranking could potentially be dependent on the proportion of shared VMWEs across languages. At the very least, it is clear that inter-language performance differences cannot be attributed to linguistic differences among languages alone, but to particularities of the dataset that interplay with these linguistic differences.

This chapter conducts a detailed data analysis of the PARSEME dataset and the official systems' submissions in order to try to understand how these particularities impact systems' performance and to propose possible modifications to the dataset in order to balance out said particularities among the language corpora.

To this end, we start our discussion in §2 by computing statistics for each language to get a sense of their differences. We then measure the relative difficulty in identifying VMWEs in each language corpus by focusing on three factors that could potentially pose challenges to the systems: 1) the relative sparsity of VMWEs in each language corpus (by measuring the proportion of sentences with and without VMWEs); 2) the prevalence and significance of discontinuous VMWEs and embedded (or overlapped) VMWEs; and 3) corpus similarity and

<footnote n="1">¹http://parseme.eu</footnote>
homogeneity measures between the training and test portions for each language section. We observe that the importance of these factors varies across languages: while some are inherent to each language's linguistic properties (e.g., proportion of continuous vs discontinuous VMWEs or the dominant category of VMWEs in a language), others (e.g., relative sparsity of VMWEs) can be controlled by altering the size of the training and test sets, the proportion of shared VMWEs between these two sets, and, in general, the homogeneity of the distribution of VMWEs in these sets for each of the languages.

We then turn our attention to the shared task official evaluation scores on the participating systems in §3 and §4. In §3, we focus on the effect of the proportion of shared VMWEs between the training and test sets in each language corpus. We evaluate the systems on shared VMWEs and on VMWEs occurring exclusively in the test set. We also introduce two baseline systems (a system that simply tags every verb as a VMWE and a simple dictionary look-up system) and observe that the performance of the participating systems follows trends that the performance of these baselines shows.

In §4, we concentrate on the evaluation metrics used in the shared task: one that measures the ability of retrieving full VMWEs (MWE-based evaluation) and another that gives credit to systems on partially identified VMWEs (Token-based evaluation). We observe that the Token-based evaluation measure gives more weight to long VMWEs and, in addition, can be exploited by a system that simply detects verbs. Lastly, we use Cohen's κ inter-annotator agreement measure as an evaluation metric based on the intuition that it provides a 'chance-corrected' degree of similarity between a system output and a gold standard.

In §5, we conclude that the PARSEME VMWE dataset is a valuable resource for evaluating VMWE identification systems as long as certain variables are controlled for and purpose-specific evaluation frameworks are considered. We also propose avenues for future work.

Before we delve into the analysis and discussion, it should be mentioned that systems were considered to be participating in one of two separate tracks under the original shared task rules: a) an open track in which participants were free to use any external data (other than the training data provided) to train and develop their systems, and b) a closed track, where participants were allowed to use the provided training data only. Given that only one system (LATL) participated in the open track and for only one language (French), this chapter completely ignores the open/closed distinction and compares all systems on the same evaluation scores.
## 2 Shared task dataset

This section explores several numerical properties of the dataset developed for the shared task in order to gain an insight into differences among languages and to identify potential difficulty factors in the corpora. We consider difficulty factors to be corpus-specific characteristics (such as corpus size, sparsity of VMWEs or corpus heterogeneity) that could potentially hinder an algorithm's ability to identify VMWEs. We assess a factor's degree of difficulty by observing the overall systems' performance on languages that present the factor in question, in comparison to languages that do not present that factor. The performance of the systems is measured by the official shared task evaluation F1 scores, shown in Table 1. That table also contains the averages all systems' scores for a given language (avg column) and the ranks of the languages according to these averages (rnk column). Recall that two evaluation modalities were measured in the shared task: MWE-based evaluation, which counts as a success the matching of a full VMWE, and Token-based evaluation, which gives partial credit to partially matched VMWEs. Figure 1 summarises these scores per language as box plots.

<figure><img src="image_28.png" /><figcaption>Figure 1: Box plots summarising F1 scores achieved by all systems on each language, using the MWE-based and Token-based evaluation modalities</figcaption></figure>
<table><caption>Table 1: F1 evaluation scores by language and system with averages (avg), rank (rnk) in Token-based, MWE-based and Cohen's κ evaluations. Baselines: dictionary look-up (BD) and verb detection (BV).</caption><thead><tr><th></th><th>ADAPT</th><th>LATL</th><th>LIF</th><th>MUMULS</th><th>RACAI</th><th>SZEGED</th><th>TRANSITION</th><th>avg</th><th>rnk</th><th>BD</th><th>BV</th></tr></thead><tbody><tr><td rowspan="3">BG</td><td>Token-based</td><td></td><td></td><td>59.16</td><td></td><td></td><td>66.15</td><td>62.66</td><td>4</td><td>47.44</td><td></td></tr><tr><td>MWE-based</td><td></td><td></td><td>34.68</td><td></td><td></td><td>61.27</td><td>47.98</td><td>6</td><td>34.67</td><td></td></tr><tr><td>Cohen's κ</td><td></td><td></td><td>21.36</td><td></td><td></td><td>53.57</td><td>37.47</td><td>6</td><td>21.27</td><td></td></tr><tr><td rowspan="3">CS</td><td>Token-based</td><td>72.86</td><td></td><td>23.52</td><td>70.76</td><td></td><td>73.65</td><td>60.20</td><td>5</td><td>64.34</td><td>20.41</td></tr><tr><td>MWE-based</td><td>57.72</td><td></td><td>16.67</td><td>64.18</td><td></td><td>71.67</td><td>52.56</td><td>4</td><td>51.91</td><td>0</td></tr><tr><td>Cohen's κ</td><td>46.49</td><td></td><td>8.82</td><td>55.04</td><td></td><td>64.36</td><td>43.68</td><td>5</td><td>37.87</td><td>-18.54</td></tr><tr><td rowspan="3">DE</td><td>Token-based</td><td>40.48</td><td></td><td>34.45</td><td>28.30</td><td>45.45</td><td>41.09</td><td>37.95</td><td>13</td><td>40.70</td><td>28.52</td></tr><tr><td>MWE-based</td><td>22.80</td><td></td><td>21.14</td><td>19.17</td><td>40.53</td><td>41.10</td><td>28.95</td><td>13</td><td>41.34</td><td>9.22</td></tr><tr><td>Cohen's κ</td><td>5.86</td><td></td><td>5.01</td><td>5.82</td><td>24.97</td><td>26.44</td><td>13.62</td><td>16</td><td>26.42</td><td>-15.49</td></tr><tr><td rowspan="3">EL</td><td>Token-based</td><td>43.14</td><td></td><td>42.17</td><td>38.71</td><td>40.75</td><td>46.88</td><td>42.33</td><td>12</td><td>34.16</td><td>9.14</td></tr><tr><td>MWE-based</td><td>31.34</td><td></td><td>23.08</td><td>31.74</td><td>31.88</td><td>40.07</td><td>31.62</td><td>12</td><td>21.81</td><td>0.02</td></tr><tr><td>Cohen's κ</td><td>23.28</td><td></td><td>12.46</td><td>25.2</td><td>22.9</td><td>31.57</td><td>23.08</td><td>11</td><td>9.46</td><td>-7.41</td></tr><tr><td rowspan="3">ES</td><td>Token-based</td><td>49.17</td><td></td><td>48.75</td><td>30.93</td><td>44.18</td><td>58.39</td><td>46.28</td><td>9</td><td>50.97</td><td>15.56</td></tr><tr><td>MWE-based</td><td>44.33</td><td></td><td>33.62</td><td>30.06</td><td>33.99</td><td>57.39</td><td>39.88</td><td>9</td><td>44.22</td><td>0</td></tr><tr><td>Cohen's κ</td><td>35.84</td><td></td><td>21.18</td><td>23.41</td><td>17.81</td><td>48.83</td><td>30.91</td><td>9</td><td>32.7</td><td>-13.41</td></tr><tr><td rowspan="3">FA</td><td>Token-based</td><td>85.36</td><td></td><td></td><td></td><td></td><td>90.20</td><td>87.78</td><td>1</td><td>65.75</td><td>47.73</td></tr><tr><td>MWE-based</td><td>80.08</td><td></td><td></td><td></td><td></td><td>86.64</td><td>83.36</td><td>1</td><td>55.92</td><td>0</td></tr><tr><td>Cohen's κ</td><td>63.13</td><td></td><td></td><td></td><td></td><td>74.77</td><td>68.95</td><td>2</td><td>22.71</td><td>-50.01</td></tr><tr><td rowspan="3">FR</td><td>Token-based</td><td>61.52</td><td>54.61</td><td>10.00</td><td>29.40</td><td>50.09</td><td>33.64</td><td>60.28</td><td>42.79</td><td>10</td><td>45.73</td><td>18.28</td></tr><tr><td>MWE-based</td><td>50.88</td><td>47.46</td><td>10.82</td><td>9.29</td><td>47.55</td><td>5.73</td><td>57.74</td><td>32.78</td><td>11</td><td>38.42</td><td>0.21</td></tr><tr><td>Cohen's κ</td><td>40.12</td><td>33.77</td><td>7.98</td><td>-4.75</td><td>38.74</td><td>-14.42</td><td>48.98</td><td>21.49</td><td>13</td><td>24.19</td><td>-15.99</td></tr><tr><td rowspan="3">HE</td><td>Token-based</td><td></td><td></td><td>0.00</td><td></td><td></td><td>31.30</td><td>15.65</td><td>16</td><td>33.80</td><td></td></tr><tr><td>MWE-based</td><td></td><td></td><td>0.00</td><td></td><td></td><td>33.44</td><td>16.72</td><td>16</td><td>37.44</td><td></td></tr><tr><td>Cohen's κ</td><td></td><td></td><td>0.00</td><td></td><td></td><td>27.74</td><td>13.87</td><td>14</td><td>32.69</td><td></td></tr><tr><td rowspan="3">HU</td><td>Token-based</td><td>66.10</td><td></td><td>68.86</td><td>62.26</td><td>70.81</td><td>67.47</td><td>67.10</td><td>3</td><td>68.13</td><td>12.49</td></tr><tr><td>MWE-based</td><td>66.89</td><td></td><td>62.21</td><td>65.08</td><td>74.01</td><td>69.87</td><td>67.61</td><td>3</td><td>68.09</td><td>2.44</td></tr><tr><td>Cohen's κ</td><td>50.6</td><td></td><td>42.13</td><td>49.45</td><td>60.04</td><td>52.03</td><td>50.85</td><td>3</td><td>49.01</td><td>-35.81</td></tr><tr><td rowspan="3">IT</td><td>Token-based</td><td>25.11</td><td></td><td>18.24</td><td>34.90</td><td></td><td>43.57</td><td>30.46</td><td>14</td><td>37.85</td><td>14.4</td></tr><tr><td>MWE-based</td><td>23.09</td><td></td><td>16.90</td><td>15.31</td><td></td><td>39.90</td><td>23.80</td><td>15</td><td>29.03</td><td>0</td></tr><tr><td>Cohen's κ</td><td>14.26</td><td></td><td>10.01</td><td>-10.01</td><td></td><td>25.33</td><td>9.9</td><td>17</td><td>8.27</td><td>-14.44</td></tr><tr><td rowspan="3">LT</td><td>Token-based</td><td></td><td></td><td>0.00</td><td></td><td></td><td>25.33</td><td>12.67</td><td>17</td><td>28.85</td><td></td></tr><tr><td>MWE-based</td><td></td><td></td><td>0.00</td><td></td><td></td><td>28.35</td><td>14.18</td><td>17</td><td>30.08</td><td></td></tr><tr><td>Cohen's κ</td><td></td><td></td><td>0.00</td><td></td><td></td><td>27.25</td><td>13.62</td><td>15</td><td>28.82</td><td></td></tr><tr><td rowspan="3">MT</td><td>Token-based</td><td>8.87</td><td></td><td>0.00</td><td>4.69</td><td></td><td>16.29</td><td>7.46</td><td>18</td><td>11.42</td><td>6.79</td></tr><tr><td>MWE-based</td><td>6.41</td><td></td><td>0.00</td><td>5.00</td><td></td><td>14.44</td><td>6.46</td><td>18</td><td>6.75</td><td>0.02</td></tr><tr><td>Cohen's κ</td><td>3.5</td><td></td><td>0.00</td><td>2.99</td><td></td><td>6.6</td><td>3.27</td><td>18</td><td>-5.25</td><td>-5.74</td></tr><tr><td rowspan="3">PL</td><td>Token-based</td><td>72.74</td><td></td><td>69.77</td><td></td><td></td><td>70.56</td><td>53.27</td><td>7</td><td>74.40</td><td>18.33</td></tr><tr><td>MWE-based</td><td>67.95</td><td></td><td>59.61</td><td></td><td></td><td>69.09</td><td>49.16</td><td>5</td><td>69.98</td><td>0</td></tr><tr><td>Cohen's κ</td><td>61.53</td><td></td><td>51.33</td><td></td><td></td><td>62.72</td><td>43.9</td><td>4</td><td>63.46</td><td>-15.01</td></tr><tr><td rowspan="3">PT</td><td>Token-based</td><td>70.18</td><td></td><td>60.01</td><td></td><td></td><td>70.94</td><td>57.98</td><td>6</td><td>59.97</td><td>14.32</td></tr><tr><td>MWE-based</td><td>58.14</td><td></td><td>44.05</td><td></td><td></td><td>67.33</td><td>42.63</td><td>8</td><td>54.49</td><td>0</td></tr><tr><td>Cohen's κ</td><td>51.35</td><td></td><td>35.98</td><td></td><td></td><td>62.03</td><td>34.46</td><td>7</td><td>46.35</td><td>-11.86</td></tr><tr><td rowspan="3">RO</td><td>Token-based</td><td>81.90</td><td></td><td>83.58</td><td>77.99</td><td></td><td>79.12</td><td>80.65</td><td>2</td><td>63.76</td><td>11.51</td></tr><tr><td>MWE-based</td><td>73.38</td><td></td><td>77.21</td><td>77.75</td><td></td><td>75.31</td><td>75.91</td><td>2</td><td>57.74</td><td>0</td></tr><tr><td>Cohen's κ</td><td>71.28</td><td></td><td>75.35</td><td>76.12</td><td></td><td>73.18</td><td>73.98</td><td>1</td><td>53.75</td><td>-7.32</td></tr><tr><td rowspan="3">SL</td><td>Token-based</td><td>45.06</td><td></td><td>45.62</td><td>33.20</td><td></td><td>46.55</td><td>42.61</td><td>11</td><td>28.47</td><td>0.08</td></tr><tr><td>MWE-based</td><td>37.08</td><td></td><td>31.08</td><td>30.19</td><td></td><td>43.22</td><td>35.39</td><td>10</td><td>21.65</td><td>0</td></tr><tr><td>Cohen's κ</td><td>29</td><td></td><td>20.49</td><td>23.45</td><td></td><td>33.17</td><td>26.53</td><td>10</td><td>5.23</td><td>-0.07</td></tr><tr><td rowspan="3">SV</td><td>Token-based</td><td>31.49</td><td></td><td>26.69</td><td>31.19</td><td></td><td>30.70</td><td>30.02</td><td>15</td><td>8.94</td><td>13.23</td></tr><tr><td>MWE-based</td><td>30.32</td><td></td><td>25.17</td><td>27.03</td><td></td><td>30.36</td><td>28.22</td><td>14</td><td>7.32</td><td>0</td></tr><tr><td>Cohen's κ</td><td>24.44</td><td></td><td>20.78</td><td>16.56</td><td></td><td>24.75</td><td>21.63</td><td>12</td><td>-5.62</td><td>-10.29</td></tr><tr><td rowspan="3">TR</td><td>Token-based</td><td>52.85</td><td></td><td>45.40</td><td>51.59</td><td></td><td>55.28</td><td>51.28</td><td>8</td><td>16.60</td><td>10.45</td></tr><tr><td>MWE-based</td><td>42.83</td><td></td><td>34.49</td><td>51.76</td><td></td><td>55.40</td><td>46.12</td><td>7</td><td>5.95</td><td>0</td></tr><tr><td>Cohen's κ</td><td>25.88</td><td></td><td>19.05</td><td>38.88</td><td></td><td>42.14</td><td>31.49</td><td>8</td><td>-8.57</td><td>-17.81</td></tr><tr><td rowspan="3">avg</td><td>Token-based</td><td>53.79</td><td>54.61</td><td>10.00</td><td>40.71</td><td>41.12</td><td>36.86</td><td>54.10</td><td></td><td>43.40</td><td>16.08</td></tr><tr><td>MWE-based</td><td>46.22</td><td>47.46</td><td>10.82</td><td>29.81</td><td>38.71</td><td>25.50</td><td>52.37</td><td></td><td>37.60</td><td>0.79</td></tr><tr><td>Cohen's κ</td><td>36.44</td><td>33.77</td><td>7.98</td><td>20.56</td><td>30.82</td><td>11.82</td><td>43.64</td><td></td><td>24.6</td><td>-16.92</td></tr></tbody></table>
## 2.1 Corpora sizes, VMWE sparsity and frequency distributions

We start by discussing the sizes of the training and test portions in each language corpus, depicted in Figure 2. Sizes are measured in terms of the total number of sentences. Traditionally, corpora sizes are discussed in terms of number of words, rather than number of sentences. We use number of sentences instead for a variety of reasons: 1) Each language corpus in the dataset consists of a collection of individual sentences. So the sentence is a natural unit to describe the dataset. 2) A sentence is expected to have a single main verb. On average, we can expect to have a little more than one verb per sentence. However, we would like to know what this average is for the case of *verbal* MWEs (VMWEs). That is, we would like to know how sparse VMWEs are in a given language corpus, and what impact this sparsity may have. 3) Measures such as the rate of VMWEs per *n* tokens could also be used, but are less linguistically motivated. Finally, 4) the training-to-test size ratios in terms of number of words are largely the same in this dataset as in terms of number of sentences.

Notice that Romanian and Czech have by far the largest training sets, dwarfing corpora of all other languages. This seems to work in favour of these two languages as, on average, Romanian ranked 2nd place in both evaluation modalities and Czech ranked at 4th and 5th places in the MWE-based and Token-based modalities, respectively. Swedish is the language with the smallest training set (only 200 sentences). The average F1 score of systems participating in Swedish is around 30% for both evaluation modalities. Indeed, the size of the training set is somewhat positively correlated with the average system evaluation scores for each language. The Pearson correlation coefficients for MWE-based and Token-based evaluations are 0.33 and 0.35, respectively.

The size of the test set relative to its corresponding training set varies widely across languages. The test-to-training proportions vary from 8% to 60% for most languages, except for Maltese (79%), Spanish (85%) and most notably, Swedish, with a test set about 8 times larger than its training set.<fnref n="2" /> Although both Maltese and Swedish performed rather poorly (Maltese actually ranked last), there is no clear pattern between the test-to-training proportion of a language corpus and the performance of systems. In fact, Spanish ranked exactly in the middle at 9th place. These proportions were found to be mildly negatively correlated against MWE-based and Token-based evaluations: -0.20 and -0.23, respectively (Pearson correlation coefficients).

<footnote n="2">²200 training sentences vs. 1600 test sentences, making the proportion of the training set almost invisible in Figure 2</footnote>
<figure><img src="image_29.png" /><figcaption>Figure 2: Relative sizes (in sentences) of the training and test portions of each language corpus.</figcaption></figure>

<figure><img src="image_30.png" /><figcaption>Figure 3: VMWE Sparsity – Percentage of sentences with VMWEs; horizontal lines depict average percentages across languages for training (TRN) and test (TST) sets, respectively.</figcaption></figure>