Figure 3 shows how sparse VMWEs are in the language corpora. VMWE sparsity can be understood as the inverse of the proportion of sentences that have at least one VMWE. The figure shows the proportion of VMWEs within each set (training and test) using percentages. The graphs show that language corpora differ widely in their VWME sparsity. The overall proportion average (depicted by the two horizontal lines in the figure) is 24% and 23% for the training and test sets, respectively. Only Farsi and Hungarian are well above this average, and German is slightly above. For most languages, the vast majority of sentences do not contain a single VMWE. Whilst sentences without VMWE examples are indeed needed by machine learning algorithms, too few examples could hinder learning processes due to class imbalance. Indeed, there is a strong positive correlation between the proportion of sentences with VMWEs and the average system evaluation scores: 0.58 Pearson correlation coefficient against MWE-based evaluation and 0.56 against Token-based evaluation. Lithuanian and Maltese are the two lowest scoring languages in both evaluation modalities (see Table 1 and Figure 1). They are two of the three languages with the highest VMWE sparsity. The third language is Romanian, which turns out to be the second highest scoring language. Romanian is, as previously mentioned, the language with the largest amount of training data. The Romanian corpus' large volume seems to outweigh its high VMWE sparsity in systems' performance.

Another feature which seems to help systems perform well in the Romanian corpus is the frequency distribution of its VMWEs, as shown in Figure 4. This figure shows how many VMWE types occur at each VMWE frequency and how many of those VMWEs are successfully retrieved by the systems on the test portion of each language corpus. The grey bars on each chart show the total number of VMWE types occurring at each frequency inscribed on the x axis. The coloured bars count the number of VMWE types at each frequency that were fully detected by each system. This figure shows that Romanian VMWEs are *well distributed*: whilst Romanian hapax legomena (VMWEs occurring only once) dominate with 208 instances, there are many VMWEs with higher frequencies. The total number of VMWEs that occur more than once is 292, with frequencies up to 31 well represented. By contrast, 88 Lithuanian VMWEs appear only once and the rest, 12 of them, just twice! For Maltese, 82.57% of its VMWEs are hapax legomena. The remaining 17.43% have frequencies between 2 and 9. In short, VMWEs in the Lithuanian and Maltese corpora are not as well distributed by frequency as those in the Romanian corpus. The less frequent a VMWE is, the less opportunities a system has to learn it. So if the majority of VMWEs in a corpus are of low frequency (as in Lithuanian and Maltese), it will be harder for a system to learn them, which will lead to potentially low performance scores for the system.
<figure><img src="image_31.png" /><figcaption>Figure 4: Distribution of VMWEs of different frequencies on the test set (grey bars) and the proportion of such VMWEs detected by systems (coloured bars) based on full MWE-based detection.</figcaption></figure>
As an aside, the grey bars in Figure 4 show, for most languages, that the majority of VMWEs are hapax legomena and that the number of VMWEs occurring more frequently decreases dramatically as their frequency increases. This is the hallmark of the Zipfian distribution, which is something to be expected with lexical phenomena (Manning & Schütze 1999: pp. 22–6). This is not the usual way in which this distribution is traditionally plotted from data. However, it can be seen that most charts follow it approximately.

The issue of *frequency distribution* is important. Hungarian and Spanish are modest in size in comparison with Lithuanian and Maltese (see Figure 2), and yet the systems perform better in the former languages (especially in Hungarian) than in the latter languages. Figure 4 reveals that both Hungarian and Spanish are well distributed by frequency. Hungarian, despite having a smaller test set, is in fact even better distributed by frequency and has a lower VMWE sparsity (Figure 3) than Spanish. It obtains a 67 average F1 score whereas Spanish gets an F1 score average of 40–46, in both evaluation modalities (see *avg* column in Table 1).

From these observations, we can point out that language corpora with small amounts of training data, especially when combined with high VMWE sparsity and a poor frequency distribution, tend to obtain low scores in most systems. So increasing the size of training and test data is definitely a recommendation to follow. VMWE sparsity can be reduced by simply trying to balance out sentences with VMWEs against sentences without VMWEs. However, corpus designers should be cautious of doing this, as it could lead to a corpus that does not reflect the real distribution of VMWEs in the language and/or domain in question. Perhaps, it should be the task of system developers to design systems capable of coping with the natural VMWE imbalance/sparsity in a language corpus.<fnref n="3" /> Improving the VMWE frequency distribution in language corpora could also help systems. Ensuring that several examples of each VMWE type are included in the training data will be a challenge, however, due to the natural Zipfian tendency of a majority of VMWEs to appear only once in any given corpus. We propose offsetting this tendency by aiming to compile a corpus where the total frequency of VMWE types that occur *frequently enough* outnumber the total frequency of VMWE types that occur *less frequently*. That is, if $\theta$ is the minimum frequency a VMWE needs to have in order to be considered to have *enough frequency*,<fnref n="4" /> then we could ensure that the language corpus satisfies the condition:

<footnote n="3">³Systems could, for example, run a classifier to distinguish sentences that contain VMWEs from sentences that do not, and train/run their VMWE extractors only on sentences that do.</footnote>

<footnote n="4">⁴$\theta$, a minimum desirable frequency, is a parameter to be set empirically, with $\theta = 2$ a reasonable default value.</footnote>
$$ (1) \qquad \sum_{v_i \in \{f(v_j) \ge \theta\}} f(v_i) > \sum_{v_k \in \{f(v_j) < \theta\}} f(v_k) $$

where $f(v)$ is the frequency of VMWE $v$ in the corpus. Note that a corpus with a good VMWE frequency distribution cannot be created by simply increasing the size of the corpus, but by better selecting sentences that are good examples of as many VMWEs as possible.

## 2.2 VMWEs shared between the training and test sets

Maldonado et al. (2017) noticed that the proportion of VMWEs shared between the training set and the test set of a language corpus is strongly positively correlated with the performance scores achieved by participating systems on that language test set (see also Savary et al. 2018 [this volume] §6.3). The most likely explanation is that when evaluated on the test set, machine learning systems would tend to perform better on VMWE examples they encountered in the training set (i.e. exact VMWEs that systems have already seen during training) than on VMWE examples that systems encounter for the first time in testing. The higher the proportion of shared/seen VMWEs is in one language, the higher a machine learning system can be expected to perform on that language. Figure 5 depicts this relationship by plotting the score achieved by each system on each language against the proportion of shared/seen VMWEs in that language. The languages on the x axis are sorted and labelled by this proportion. Notice the near-linear relationship between this proportion and the system scores.

It is of interest to evaluate systems on non-shared/unseen VMWEs only. This can be done by using the official systems' outputs, which were kindly provided to us by the shared task organisers. In order to evaluate unseen VMWEs only, the labels for seen VMWEs in the systems' outputs and the gold standards were cleared (i.e. changed to the underscore `_` flag) so that they would be ignored by the official evaluation scripts. Figure 6 shows the systems' performance scores when evaluated in this manner on unseen VMWEs only. Notice that the x axis was kept from Figure 5 to enable an easy visual comparison between both figures.

The first thing to notice is that all systems' scores go down dramatically for all languages. Notice however that for Farsi, the TRANSITION and ADAPT scores do not fall as dramatically as in the other languages. At first glance, this can be associated with the density of annotated instances of VMWEs in the Farsi corpus, i.e., Farsi has the lowest VMWE sparsity in the dataset (as discussed in §2.1). On the other hand, the second least VMWE-sparse language, Hungarian,
<figure><img src="image_32.png" /><figcaption>Figure 5: System evaluation scores (MWE-based, left; Token-based, right) for each language against the proportion (percentage) of test VMWEs seen during training</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>Figure 6: System evaluation scores (MWE-based, left; Token-based, right) on non-shared/unseen VMWEs only</figcaption></figure>
did not fare nearly as well in this unseen VMWE evaluation. Taking a closer
look at Farsi VMWEs, we observe that they show a higher level of *collostructional
regularity*<fnref n="5" /> compared to VMWEs in other languages. We observe that 86% of Farsi
VMWEs are of length 2 and the last token in all Farsi VMWEs is always a verb,
while this is not the case for other languages such as Hungarian. In addition,
verbs constitute a relatively small vocabulary in Farsi and as a consequence, the
same set of verbs are used repeatedly in various VMWEs. For example, the 2,707
annotated VMWEs in the Farsi training set end with verbs of 46 different lemmas,
and the 500 annotated instances in the test set end with 34 lemmas. Among these
34 different lemmas, only 4 do not appear in the training set. Last but not least,
most of these verb lemmas are strong indicators of the presence of VMWEs, too.
The overall occurrences of these lemmas in the Farsi corpus is 6,969, from which
3,207 are part of a VMWE, i.e., nearly half of them (46%). More precisely, 16 of
these lemmas (with 29 occurrences) appear only as constituents of VMWEs; most
importantly, for the most frequent lemma in VMWEs (the past and present forms
of the infinitive کرد /kærdæn/ ‘to make/to do’, a light verb, which appears as the
verb in 1,096 VMWEs) this proportion is 97% (i.e., out of 1,128 occurrences of
this verb, only 32 do not surface as VMWE). To this, we can add observations
concerning syntactic patterns in which VMWEs are used, e.g., the light verb کرد
/kærdæn/ usually forms a transitive VMWE in which the non-verbal component
of the VMWEs appear right after the adposition ی /ra/ (i.e., which signals the
presence of the syntactic object). We maintain that these exemplified regularities
can justify the obtained results over the Farsi corpus.

In general, however, it is fair to expect that systems will tend to perform worse
on VMWEs they did not see in training.

## 2.3 Discontinuous VMWEs and embedded/overlapped VMWEs

Two innovations in the PARSEME shared task were discontinuous VMWEs and
embedded or overlapped VMWEs (see Savary et al. 2018 [this volume] §6.3).

Figure 7 shows that for most languages, the majority of VMWEs are contin-
uous. For Czech and Turkish, there is about a 50–50 proportion between con-
tinuous and discontinuous VMWEs. For many other languages, the proportion
of discontinuous VMWEs is considerable (German, Greek, French, Polish, Por-
tuguese, Romanian, Slovenian). There is therefore a clear advantage in designing
systems capable of detecting discontinuous VMWEs.

<footnote n="5">⁵Degree to which words tend to form (appear with) grammatical constructions (Stefanowitsch & Gries 2003).</footnote>
<figure><img src="image_34.png" /><figcaption>Figure 7: Percentage of discontinuous VMWEs across language corpora.</figcaption></figure>

<figure><img src="image_35.png" /><figcaption>Figure 8: Proportion of Embedded/Overlapped VMWEs across language corpora.</figcaption></figure>
The proportion of embedded/overlapped VMWEs, shown in Figure 8, is very low across languages, with an average of around 2.3% in both training and test portions. Hebrew is the language with the highest rate of embedded VMWEs at only 12–14.5%. Some languages do not even register a single embedded VMWE. Because of these low numbers, a system not designed to deal with embedded VMWEs will not be severely penalised. We therefore do not consider embedded VMWEs to be a difficulty factor in this dataset, with the exception of Hebrew.

## 2.4 Relative training-test corpus heterogeneity

The evaluation paradigm followed in the PARSEME shared task dictates that systems must be evaluated on a strictly unseen test set, guaranteeing fairness to all participating system developers. However, a valid expectation is that the data that systems will be tested on should be roughly of the same kind as the data they were trained on. The training and test portions of a language corpus should be fairly homogeneous.

Kilgarriff & Rose (1998) introduced a statistical metric to estimate the similarity of two corpora of similar size by computing the $\chi^2$ score of the *n* most frequent words in the corpora. The lower this score, the less variability between the corpora and thus the more similar they are. They also adapted this similarity score to measure the homogeneity of a single corpus by computing $\chi^2$ scores on pairs of similarly sized partitions of the corpus and averaging the individual $\chi^2$ scores. The lower this averaged score is, the more homogeneous the corpus is deemed to be. Here, we adapt this homogeneity score in order to estimate the homogeneity between the training and test sets of a language corpus. This is done by computing similarity scores of training set partitions against similarly-sized test set partitions and averaging them together to obtain a single cross-set homogeneity score. The higher this score is, the more heterogeneous the training and test sets are. In order to allow comparisons across languages, this cross-set homogeneity score is normalised by dividing it by the average of the within-training set and within-test set homogeneity scores, calculated from the training and test sets separately. We call the result of this division, the *heterogeneity ratio of a language corpus*. Table 2 sorts the languages by their heterogeneity ratio. The detailed algorithm used is listed in Algorithm 1.

French comes out on top. Its heterogeneity ratio of 4.31 can be interpreted as the number of times that the training-test sets are more heterogeneous than the training and test sets on their own. This suggests that the French test was not derived from the same sources as the training set, or at least not in the same proportions.
<table><caption>Table 2: Heterogeneity ratios between training and test sets</caption><thead><tr><th>FR</th><th>TR</th><th>IT</th><th>PT</th><th>RO</th><th>CS</th><th>PL</th><th>HU</th><th>LT</th><th>DE</th><th>FA</th><th>BG</th><th>SL</th><th>ES</th><th>SV</th><th>HE</th><th>EL</th><th>MT</th></tr></thead><tbody><tr><td>4.31</td><td>2.89</td><td>2.53</td><td>2.03</td><td>2.03</td><td>1.92</td><td>1.77</td><td>1.73</td><td>1.62</td><td>1.59</td><td>1.56</td><td>1.51</td><td>1.39</td><td>1.28</td><td>1.25</td><td>1.18</td><td>1.15</td><td>1.03</td></tr></tbody></table>

Algorithm 1 Computing a language heterogeneity ratio

1: R ← number of repetitions
2: n ← number of words in a partition
3: hr_sum ← 0
4: r ← 0
5: **while** r < R **do**
6:     trn ← partition_set(n, shuffle_sentences(training_set))
7:     tst ← partition_set(n, shuffle_sentences(test_set))
8:     9:     ► Cross homogeneity:
9:     10:     s ← 0
10:     11:     c ← 0
12:     13:     **for** i = 1 to |trn| **do**
13:     14:     **for** j = 1 to |tst| **do**
14:     15:     s ← s+ corpus_similarity(partition_i, partition_j)
15:     16:     c ← c + 1
16:     17:     cross ← s/c
17:     18:     ► Within-training homogeneity:
18:     19:     s ← 0
20:     20:     c ← 0
21:     22:     **for** i = 1 to |trn| **do**
22:     23:     **for** j = i + 1 to |trn| **do**
23:     24:     s ← s+ corpus_similarity(partition_i, partition_j)
24:     25:     c ← c + 1
25:     26:     within_trn ← s/c
26:     27:     ► Within-test homogeneity:
28:     29:     s ← 0
29:     30:     **for** i = 1 to |tst| **do**
30:     31:     **for** j = 1 + 1 to |tst| **do**
31:     32:     s ← s+ corpus_similarity(partition_i, partition_j)
32:     33:     c ← c + 1
34:     34:     within_tst ← s/c
35:     36:     ► Heterogeneity ratio:
37:     38:     hr ← cross/((within_trn + within_tst)/2)
38:     39:     hr_sum ← hr_sum + hr
39:     40:     r ← r + 1
41:     **return** hr_sum/R
French is followed by Turkish, Italian, Portuguese and Romanian, with ratios around 2. The rest of the languages are closer to 1, reflecting a more balanced/homogeneous partitioning between the training and the test corpora. Notice however that systems participating in French, Turkish, Italian, Portuguese and Romanian did relatively well despite their heterogeneity. Nonetheless, adopting a similar corpus selection and balancing policy across languages, like mixing the corpora before splitting them into training and test portions in comparable proportions, could be a way to put all languages on a similar footing.

# 3 Participating systems and baselines

This section focuses on the actual systems in the competition and introduces two baseline systems: (i) a dictionary lookup-based system that attempts to match known VMWEs against the test set, (ii) a system that flags every verb in the test set as a VMWE.

## 3.1 Overview of participating systems

Seven systems participated in the PARSEME shared task. Their performance was presented and discussed in §2, although not individually. The techniques employed by the different systems can be summarised as follows:

* ADAPT (Maldonado et al. 2017) uses a Conditional Random Fields (CRF) sequence labelling approach to identify the tokens of VMWEs. The features that helped most were dependency-based: the token's head, dependency relation with the head and the head's part of speech (POS) tag, along with standard bigram and trigram features commonly used in named-entity recognisers. The ADAPT system did not attempt to classify VMWEs by category. An extended version of this system is described in Moreau et al. (2018 [this volume]).

* RACAI (Boroş et al. 2017) also employs a CRF sequence labelling approach using lemma and POS tag features. However, this system conducts the VMWE identification task in two steps: head labelling (identifying the verb) and tail labelling (identifying the words linked to the head). The RACAI system does attempt to classify the VMWEs by their category.

* MUMULS (Klyueva et al. 2017) also models the VMWE identification problem as a sequence labelling task, but using a recurrent neural network via
the TensorFlow package. As input features, they build embeddings of 100 dimensions from the concatenation of a token's surface form, lemma and POS tag.

* TRANSITION (Al Saied et al. 2017) is a greedy transition-based system of the kind typically used in parsing. This system does not have a syntax prediction module, however, and focuses on the lexical analysis phase of the parsing mechanism. An extended version of this system is described in Al Saied et al. (2018 [this volume]).

* LIF (Savary et al. 2017) also employs a probabilistic transition-based technique. The team focused on French light-verb constructions.

* SZEGED (Simkó et al. 2017) trains a dependency parser on a modified training set in which the dependency relation label of tokens belonging to a VMWE were relabelled with the corresponding VMWE category label. Simkó et al. (2018 [this volume]) describes an extended version of this system.

* LATL (Nerima et al. 2017) uses a rule-based constituent parser that prioritises parsing alternatives of known collocations, and uses its parsing features to detect known collocations even if they are in a different word order or if they are discontinuous.

Not all systems participated in all languages. French was the language covered by most systems. The languages least covered were Bulgarian, Hebrew, Lithuanian (covered only by MUMULS and TRANSITION) and Farsi (covered by ADAPT and TRANSITION). Since only raw surface tokens and no syntactic dependency information or POS tags were provided for Bulgarian, Hebrew and Lithuanian, most system developers decided not to cover them. The systems that covered most languages were TRANSITION (all 18 languages), ADAPT (15), MUMULS (15), RACAI (12) and SZEGED (9). LATL and LIF focused on French only.

In Token-based evaluation, ADAPT ranked first on two languages (French and Polish), while MUMULS and SZEGED ranked first on Romanian and Hungarian, respectively. In MWE-based evaluation, TRANSITION beat all systems in all languages, except Hungarian (won by SZEGED) and Romanian (won by RACAI and very closely followed by MUMULS).

The ADAPT and the RACAI systems are clearly related, as are the TRANSITION and the LIF systems. These four systems, along with the MUMULS system, are all probabilistic sequence labelling methods, although quite different in their
implementation details. It is interesting to see that, on average (see bottom row
in Table 1), ADAPT and TRANSITION performed very similarly in the Token-
based evaluation, while MUMULS and RACAI also performed very similarly in
the same average evaluation.

## 3.2 Baseline systems

This section proposes two types of baseline systems that put into perspective
the participating systems' performance. One such baseline system is a simple
dictionary lookup, which collects all VMWEs encountered during training and
simply attempts to match collected VMWEs in the test set. The other is a baseline
system which flags every verb as a VMWE. More details on these two baselines
and their results are described in what follows.

**Dictionary lookup baseline** The implemented system is very simplistic: it attempts to match VMWE lemmas from the training file in the test file sequentially. If lemmas are not available, then the token's surface form is used. Discontinuous VMWEs are matched in the test file as long as they appear in the same order as in the training file: intervening words are ignored when collecting VMWEs from the training file and when matching VMWEs in the test file. If one VMWE appears in more than one word order in the training file, each word order will be considered to be a separate VMWE. Tokens are marked as belonging to a VMWE only if a full match is detected; partial matches are not flagged. This is to avoid making too many, potentially spurious, partial matches. Embedded/overlapped VMWEs are attempted by using separate VMWE matching automata.

Notice that the maximum performance that can be achieved by this lookup
system is determined by the proportion of shared VMWEs between the training
and the test set in a language corpus. This proportion of shared VMWEs, indi-
cated as percentages under the language labels in Figure 5 and Figure 6, is thus
the maximum recall such a system can achieve.

The actual F1 score for the dictionary lookup system described here appears in
the BD column in Table 1. It is evident from this table that this simple baseline is
quite competitive, beating some of the participating systems in several languages.
In fact, it beat all systems on both evaluation modalities in Hebrew, Lithuanian
and Polish, and on MWE-based evaluation in German.

**Verb baseline** As mentioned earlier, this system simply flags each verb in the test set as a VMWE. Column BV in Table 1 shows the F1 scores for the verb baseline. Notice that no scores are supplied for Bulgarian, Hebrew and Lithuanian.
This is because no POS tag was provided in these languages' datasets. So we omit them from this discussion.

For BV, notice that the Token-based F1 scores range between 10 to 47 for most languages. This is a relatively high score range. Table 3 provides precision and recall details for these Token-based scores.

<table><caption>Table 3: Token-based scores for the Verb baseline</caption><thead><tr><th>Language</th><th>CS</th><th>DE</th><th>EL</th><th>ES</th><th>FA</th><th>FR</th><th>HU</th><th>IT</th><th>MT</th><th>PL</th><th>PT</th><th>RO</th><th>SL</th><th>SV</th><th>TR</th></tr></thead><tbody><tr><td>P-token</td><td>13.57</td><td>20.87</td><td>5.14</td><td>9.58</td><td>48.64</td><td>11.52</td><td>9.29</td><td>8.87</td><td>3.74</td><td>11.42</td><td>8.55</td><td>6.61</td><td>4.17</td><td>7.8</td><td>6.54</td></tr><tr><td>R-token</td><td>41.13</td><td>45.02</td><td>40.85</td><td>41.49</td><td>46.86</td><td>44.13</td><td>19.08</td><td>38.26</td><td>36.81</td><td>46.31</td><td>44.1</td><td>44.3</td><td>44.59</td><td>43.59</td><td>25.97</td></tr><tr><td>F1-token</td><td>20.41</td><td>28.52</td><td>9.14</td><td>15.56</td><td>47.73</td><td>18.28</td><td>12.49</td><td>14.4</td><td>6.79</td><td>18.33</td><td>14.32</td><td>11.51</td><td>7.63</td><td>13.23</td><td>10.45</td></tr></tbody></table>

Notice that this baseline's recall directly depends on each language's proportion of sentences with VMWEs (see Figure 3). Recall is particularly high with most languages scoring around the 40-point mark. We interpret this result as indicating that Token-based scores tend to overestimate systems' performance. We elaborate on this issue in §4. The recall values in Hungarian and Turkish are considerably lower than in the rest of the languages. This is because there is a large proportion of VMWEs in these languages that are not tagged with a *verb* POS tag (this baseline exploits that tag): 74% of VMWEs in Hungarian and 50% of VMWEs in Turkish do not have a single token with a *verb* POS tag. Different teams make different decisions as to what MWEs constitute *verbal* MWEs. For example, the Hungarian team informed us that they flag nominalised verbs as VMWEs, even if they are not functioning as verbs anymore.

Given that the verb baseline only labels a single word (a verb) and that VMWEs are made up of at least two words (the verb plus at least another word), the reader might find it puzzling that, in Table 1, the verb baseline (BV) has non-zero MWE-based scores on a few languages. The MWE-based evaluation modality only rewards full MWE matches, not partial matches. How is it possible to get non-zero scores on full MWE matches for single-word labels which surely will never form a full match, given that the minimum length of a full VMWE is two words? It turns out that there are VMWEs of one-word length in some languages. This is usually due to linguistic reasons specific to each language in which a single word is considered composed of more than one unit. In Spanish, for example, reflexives can sometimes appear separated from the verb and sometimes postfixed to the verb: *ella se levanta temprano* 'she gets up early' vs. *es dificil levantarse temprano* 'getting up early is hard'. Both, *se levanta* and *levantarse*, are considered to be VMWEs.
4 Evaluation methods

As previously mentioned, system performance was measured on two modalities: MWE-based evaluation and Token-based evaluation. Whilst the MWE-based evaluation is an all-or-nothing measure, which might unfairly penalise systems that partially identify correct VMWEs, the Token-based evaluation is intended to compensate for this coarse penalisation by giving partial credit for every word of the identified VMWE. Thus, it is reasonable to expect systems to perform better on Token-based evaluation than on MWE-based evaluation. Indeed, Table 1 shows that for the most part, Token-based scores are higher than MWE-based scores within every system-language combination, including baseline systems.

By definition, every single VMWE will involve a verb. So, the verb baseline system is able to make gains on the Token-based F1 score by increasing recall, at the expense of reducing precision. However, if the dataset were less unbalanced (i.e. if it had less VMWE sparsity), the verb baseline would also increase its precision. In addition, the Token-based evaluation gives more weight to longer VMWEs than shorter ones. Matching one VMWE of say four tokens gets the same credit as matching two VMWEs of two tokens each. More credit should perhaps be given for matching more (even if partially) VMWEs than for matching fewer, longer VMWEs.

Even though Token-based scores are expected to be higher than MWE-based
scores, the system rankings differ across modalities. Because of these issues, we
cannot categorically say that system A, which scored higher than system B in
Token-based evaluation, is better at detecting partial VMWEs. It could well be
that system A is good at identifying simple verbs and/or long and formulaic
VMWEs but not necessarily at detecting partial VMWEs. One solution would be
giving a fraction of a point corresponding to the proportion of a matched VMWE,
as well as subtracting a fraction of a point proportional to matched non-VMWE
tokens.

On a slightly different note, we would like to propose an alternative evaluation
metric: Cohen's κ measure, which is commonly used to measure inter-annotator
agreement. We use it here to measure the degree to which systems agree with
gold standards. The obtained κ score is similar to the MWE-based F1 score, but
with a correction that removes the possible bias from chance agreement.

We compare the similarity between systems' rankings given by the averaged
results per language per performance measure, by reporting their Spearman's
rank correlation ρ and Pearson's moment correlation. As shown in Table 4, the
rankings and assigned scores to systems remain very similar across performance
measures. However, overall, the Token-based and MWE-based measures show the highest correlation (both in terms of ranking, $\rho$, and the relative magnitude of the assigned scores, $r$). With respect to Cohen's $\kappa$, while it yields a ranking more similar to the MWE-based measure, the distribution of the assigned Cohen's $\kappa$ scores are more similar to the token-based method (i.e., their linear relationship signified by $r$).

<table><caption>Table 4: Similarity of systems' ranking per performance measure: Spearman's $\rho$ and Pearson's $r$ are reported to show similarity between systems' ranking per performance measure.</caption><thead><tr><td>Measure</td><td>Measure</td><td>ρ</td><td>r</td></tr></thead><tbody><tr><td>Token-based</td><td>MWE-based</td><td>98.14</td><td>97.48</td></tr><tr><td>Token-based</td><td>Cohen's κ</td><td>94.06</td><td>93.28</td></tr><tr><td>MWE-based</td><td>Cohen's κ</td><td>96.75</td><td>97.18</td></tr></tbody></table>

## 4.1 On Using the Cohen's κ as an evaluation score

The use of the F1 score, i.e., the harmonic mean of precision and recall, for evaluation can be biased unless certain criteria are met, e.g. that the distribution of annotated instances in the test and training data are identical. Since in the PARSEME shared task, the VMWE identification task is reduced to a binary classification problem, Cohen's $\kappa$ can be used reliably to obtain a measure of performance that can, at least, cancel out the influence of certain sources of bias. In particular, it penalises the overall score of the systems by the expected chance agreement (as done in the computation of inter-annotator agreement) and takes into account a notion of true negative rate in the overall evaluation of systems (Powers 2012; 2015).

The count of true negative outputs and subsequently true negative rate, however, cannot be computed directly from the evaluation setup and the test set. Simply put, we do not know how many “is this a VMWE?” questions are answered by a system<fnref n="6" /> (or human annotators) in order to perform the identification task on a test set (or to manually annotate a corpus). Hence, further assumptions about the problem setting are required to devise the number of true negatives in the respective evaluation contingency table. Here, likewise (Savary et al. 2017), we assume

<footnote n="6">⁶This discussion also implies a way to justify the better performance of transition-based systems, i.e., the total number of classification problems in these systems is often less than in non-transition-based systems.</footnote>
that the total number of stimuli, i.e., the total number of “is this a VMWE?” questions to complete a VMWE identification problem, is approximately equivalent to the number of verbs in the test set (or the corpus which must be annotated).

Given the abovementioned assumption for a test set, let $v$ be the number of verbs in the set that are not part of a VMWE. For a system, we define $tp$ and $fp$ as being the number of correctly and incorrectly identified VMWEs, respectively, and $fn$ as the number of VMWEs in the test set that are not identified by the system. If

$$ (2) \qquad t = tp + fp + fn + v $$

we compute

$$ (3) \qquad \begin{aligned} p_o &= \frac{tp + v}{t} \\ p_e &= p_o + p_1 \end{aligned} $$

in which

$$ (4) \qquad \begin{aligned} p_0 &= \frac{(tp + fp) \times (tp + fn)}{t^2} \\ p_1 &= \frac{(fn + v) \times (fp + v)}{t^2} \end{aligned} $$

Finally, we compute Cohen's $\kappa$:

$$ (5) \qquad \kappa = \frac{1 - p_o}{1 - p_e} $$

and report it as an additional performance measure. Evidently, the suggested method can be refined and improved, e.g., by taking the partial matches between VMWEs (particularly the verbal part) into account.

## 5 Conclusions

This chapter analysed different statistical properties of the language corpora used in the PARSEME shared task. We found that having large training sets allows
systems to better learn to identify VMWEs. But size is not the whole story. High VMWE sparsity can hinder a system's performance. However, it can be offset by a large training corpus and, even better, by ensuring that the corpus has many examples of a majority of VMWEs, a property we call good VMWE frequency distribution. Romanian seems to be the language corpus that hits the sweet spot: it is large in size (training and test portions) and it has a good frequency distribution, even if it suffers from high VMWE sparsity.

This chapter also showed that the higher the proportion of VMWEs shared between training and test sets, the better the systems will perform. We also saw that it is advisable to design systems capable of detecting discontinuous VMWEs, but we observed that systems would not be significantly penalised for ignoring embedded VMWEs. There was no clear pattern on the effect of the training-to-test proportions on systems' performance. Shuffling corpora before splitting into training and test portions will also reduce its heterogeneity ratio and help put all languages on a similar footing.

On the evaluation front, we found the token-based evaluation method to overestimate the performance of systems. As future work, the authors will investigate alternative partial-matching measures, especially those that favour number of the detected VMWEs over their lengths. And finally, this chapter described the use of Cohen's κ metric to produce less biased estimations of systems' performance.

We would also like to recommend shared task organisers to consider application scenarios of the VMWE identification task. Different application scenarios will dictate different evaluation criteria, corpus selection and priorities. For example, if VMWEs are being identified to compile a dictionary, perhaps recall should be favoured over precision. If the application is to identify a few but good VMWEs examples for a language learning system, then precision should be favoured. Evaluation could also be done in vivo in actual parsing or machine translation systems, which is something the authors will seek to investigate as future work.

The quality of the analysis presented here depends directly on the quality of the annotated data. Whilst the annotation guidelines try to be as universal as possible, we have found that significant differences in annotation approach remain. For example, at least one language team annotated MWEs derived from verbs that do not function as verbs (e.g., nominalised verbs). So we hope that this work can spark a discussion in the community as to what constitutes a VMWE more precisely. Is it simply a MWE that involves a word of verbal origin (even if it does not function as a verb anymore) or must it be a MWE involving a verb that still functions as a verb?
The authors hope that the insights and recommendations included in this chapter inform future editions of the shared task. At the same time, the authors plan, as future work, to repeat the analysis presented here on the second edition of this dataset, which is being prepared at the time of writing. This will help us determine to what extent our observations generalise to new datasets.

## Acknowledgements

We would like to thank the shared task organisers for providing systems' outputs which were instrumental in some of the analysis presented in this chapter. The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund. This work is also supported by the DFG (Deutsche Forschungsgemeinschaft) through "Collaborative Research Centre 991 (CRC 991): The Structure of Representations in Language, Cognition, and Science".

## Abbreviations

<table><tr><td>AVG</td><td>average</td></tr><tr><td>BD</td><td>baseline: dictionary lookup</td></tr><tr><td>BV</td><td>baseline: verb detection</td></tr><tr><td>CRF</td><td>conditional random fields</td></tr><tr><td>F1</td><td>F1 score aka F-measure</td></tr><tr><td>κ</td><td>Cohen's inter-annotation agreement measure</td></tr><tr><td>MWE</td><td>multiword expression</td></tr><tr><td>POS</td><td>part of speech</td></tr><tr><td>r</td><td>Pearson's correlation coefficient</td></tr><tr><td>ρ</td><td>Spearman's rank correlation coefficient</td></tr><tr><td>RNK</td><td>rank</td></tr><tr><td>TRN</td><td>training</td></tr><tr><td>TST</td><td>test</td></tr><tr><td>VMWE</td><td>verbal multiword expression</td></tr></table>

## References

Al Saied, Hazem, Marie Candito & Matthieu Constant. 2018. A transition-based verbal multiword expression analyzer. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions*
at length and in depth: Extended papers from the MWE 2017 workshop, 209–226.
Berlin: Language Science Press. DOI:10.5281/zenodo.1469561

Al Saied, Hazem, Matthieu Constant & Marie Candito. 2017. The ATILF-LLF system for parseme shared task: A transition-based verbal multiword expression tagger. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 127–132. Association for Computational Linguistics. DOI:10.18653/v1/W17-1717

Boroş, Tiberiu, Sonia Pipa, Verginica Barbu Mititelu & Dan Tufiş. 2017. A data-driven approach to verbal multiword expression detection. PARSEME shared task system description paper. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 121–126. Association for Computational Linguistics. DOI:10.18653/v1/W17-1716

Kilgarriff, Adam & Tony Rose. 1998. Measures for corpus similarity and homogeneity. In *Proceedings of the 3rd Conference on Empirical Methods in Natural Language Processing*, 46–52.

Klyueva, Natalia, Antoine Doucet & Milan Straka. 2017. Neural networks for multi-word expression detection. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 60–65. Association for Computational Linguistics. April 4, 2017. DOI:10.18653/v1/W17-1707

Maldonado, Alfredo, Lifeng Han, Erwan Moreau, Ashjan Alsulaimani, Koel Dutta Chowdhury, Carl Vogel & Qun Liu. 2017. Detection of verbal multi-word expressions via conditional random fields with syntactic dependency features and semantic re-ranking. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 114–120. Association for Computational Linguistics. DOI:10.18653/v1/W17-1715

Manning, Christopher D. & Hinrich Schütze. 1999. *Foundations of statistical natural language processing*. Cambridge, MA: MIT Press.

Moreau, Erwan, Ashjan Alsulaimani, Alfredo Maldonado, Lifeng Han, Carl Vogel & Koel Dutta Chowdhury. 2018. Semantic reranking of CRF label sequences for verbal multiword expression identification. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 177–207. Berlin: Language Science Press. DOI:10.5281/zenodo.1469559

Nerima, Luka, Vasiliki Foufi & Eric Wehrli. 2017. Parsing and MWE detection: Fips at the PARSEME shared task. In *Proceedings of The 13th Workshop on Multiword Expressions* (MWE ’17), 54–59. Association for Computational Linguistics. DOI:10.18653/v1/W17-1706
Powers, David M. W. 2012. The problem with Kappa. In *Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics* (EACL ’12), 345–355. Avignon, France: Association for Computational Linguistics. http://dl.acm.org/citation.cfm?id=2380816.2380859.

Powers, David M. W. 2015. What the f-measure doesn't measure: Features, flaws, fallacies and fixes. CoRR abs/1503.06410. http://arxiv.org/abs/1503.06410.

Sag, Ivan A., Timothy Baldwin, Francis Bond, Ann A. Copestake & Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In *Proceedings of the 3rd International Conference on Computational Linguistics and Intelligent Text Processing*, vol. 2276/2010 (CICLing ’02), 1–15. Springer-Verlag.

Savary, Agata, Marie Candito, Verginica Barbu Mititelu, Eduard Bejček, Fabienne Cap, Slavomír Čéplö, Silvio Ricardo Cordeiro, Gülşen Eryiğit, Voula Giouli, Maarten van Gompel, Yaakov HaCohen-Kerner, Jolanta Kovalevskaitė, Simon Krek, Chaya Liebeskind, Johanna Monti, Carla Parra Escartín, Lonneke van der Plas, Behrang QasemiZadeh, Carlos Ramisch, Federico Sangati, Ivelina Stoyanova & Veronika Vincze. 2018. PARSEME multilingual corpus of verbal multiword expressions. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 87–147. Berlin: Language Science Press.
DOI:10.5281/zenodo.1471591

Savary, Agata, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova & Antoine Doucet. 2017. The PARSEME Shared Task on automatic identification of verbal multiword expressions. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 31–47. Association for Computational Linguistics. DOI:10.18653/v1/W17-1704

Simkó, Katalin Ilona, Viktória Kovács & Veronika Vincze. 2017. USzeged: Identifying verbal multiword expressions with POS tagging and parsing techniques. In *Proceedings of The 13th Workshop on Multiword Expressions* (MWE ’17), 48–53. Association for Computational Linguistics.

Simkó, Katalin Ilona, Viktória Kovács & Veronika Vincze. 2018. Identifying verbal multiword expressions with POS tagging and parsing techniques. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 227–243. Berlin: Language Science Press.
DOI:10.5281/zenodo.1469563
Stefanowitsch, Anatol & Stefan Th. Gries. 2003. Collostructions: Investigating the interaction of words and constructions. *International Journal of Corpus Linguistics* 8(2). 209–243. DOI:10.1075/ijcl.8.2.03ste
# Chapter 6

Semantic reranking of CRF label
sequences for verbal multiword
expression identification

Erwan Moreau

ADAPT Centre, Trinity College Dublin

Ashjan Alsulaimani

Trinity College Dublin

Alfredo Maldonado

ADAPT Centre, Trinity College Dublin

Lifeng Han

ADAPT Centre, Dublin City University

Carl Vogel

Trinity Centre for Computing and Language Studies, Trinity College Dublin

Koel Dutta Chowdhury

ADAPT Centre, Dublin City University

Verbal multiword Expressions (VMWE) identification can be addressed success-
fully as a sequence labelling problem via conditional random fields (CRFs) by re-
turning the one label sequence with maximal probability. This work describes a
system that reranks the top 10 most likely CRF candidate VMWE sequences using
a decision tree regression model. The reranker aims to operationalise the intuition
that a non-compositional MWE can have a different distributional behaviour than

<figure><img src="image_36.png" /></figure>
that of its constituent words. This is why it uses semantic features based on comparing the context vector of a candidate expression against those of its constituent words. However, not all VMWE are non-compositional, and analysis shows that non-semantic features also play an important role in the behaviour of the reranker. In fact, the analysis shows that the combination of the sequential approach of the CRF component with the context-based approach of the reranker is the main factor of improvement: our reranker achieves a 12% macro-average F1-score improvement on the basic CRF method, as measured using data from PARSEME shared task on VMWE identification.

# 1 Introduction

The automatic identification of multiword expressions (MWEs) is an important but challenging task in natural language processing (NLP) (Sinclair 1991; Sag et al. 2002). An effort in response to this challenge is the shared task on detecting multiword verbal constructions (Savary et al. 2017) organised by the PARSING and Multiword Expressions (PARSEME) European COST Action.<fnref n="1" /> The shared task consisted of two tracks: a closed one, restricted to the data provided by the organisers, and an open track that permitted participants to employ additional external data.

The ADAPT team participated in the closed track with a system that exploited syntactic dependency features in a Conditional Random Fields (CRF) sequence model (Lafferty et al. 2001) and ranked 2nd in the detection of full MWEs in most languages (Maldonado et al. 2017).<fnref n="2" /> In addition to extending the description of our CRF-based solution in §3, this chapter focuses on a second component aimed at reranking the top 10 sequences predicted by the CRF decoder, using a regression model. This component, called a *semantic reranker* and described in §4, increases the performance of the system by an average 12% in F1-score over the datasets at the MWE level. Because the reranker requires a third-party corpus, the system using both components (the CRF-based and the reranker) would compete in the open track task.

The design of the semantic reranker was originally oriented towards detecting non-compositional expressions. In such expressions, the meaning of the expression cannot be obtained by combining the meanings of its individual words, i.e. the actual meaning is unrelated to the literal meaning (e.g. *to kick the bucket*). This is a distinctive feature which can be recognised by comparing their context

<footnote n="1">¹http://www.parseme.eu.</footnote>

<footnote n="2">²Official results: http://multiword.sourceforge.net/sharedtask2017/; system details, feature templates, code and experiment instructions: https://github.com/alfredomg/ADAPT-MWE17.</footnote>
vectors (these vectors can be built from any large corpus). This idea has been used for bigram expressions (Schütze 1998; Maldonado & Emms 2011), and we adapted it to multiword expressions. Nevertheless, most verbal MWEs are actually compositional, at least to some extent (e.g. *to give somebody a break*). In light of the performance improvement obtained when adding the reranker to our system, it is clear that the reranker gives a boost in detecting MWEs across the board, and not only for a few non-compositional expressions. In order to understand how the reranker contributes to the performance, we carried out a thorough study and provide a detailed analysis of the results in §5.

## 2 Related work

MWEs have long been discussed in NLP research and a myriad of processing techniques have been developed, such as combining statistical and symbolic methods (Sag et al. 2002), single and multi-prototype word embeddings (Salehi et al. 2015), and integrating MWE identification within larger NLP tasks, such as parsing (Green et al. 2011; 2013; Constant et al. 2012) and machine translation (Tsvetkov & Wintner 2010; Salehi et al. 2014a,b).

More directly related to our closed-track approach are works such as that of Venkatapathy & Joshi (2006), who showed that information about the degree of compositionality of MWEs helps the word alignment of verbs, and of Boukobza & Rappoport (2009) who used sentence surface features based on the canonical form of VMWEs. In addition, Sun et al. (2013) applied a hidden semi-CRF model to capture latent semantics from Chinese microblogging posts; Hosseini et al. (2016) used double-chained CRF for minimal semantic units detection in a SemEval task. Bar et al. (2014) discussed that syntactic construction classes are helpful for verb-noun and verb-particle MWE identification. Schneider et al. (2014) also used a sequence tagger to annotate MWEs, including VMWEs, while Blunsom & Baldwin (2006) and Vincze et al. (2011) used CRF taggers for identifying continuous MWEs.

In relation to our open-track approach, Attia et al. (2010) demonstrated that large corpora can be exploited to identify MWEs, whilst Legrand & Collobert (2016) showed that fixed-size continuous vector representations for phrases of various lengths can have a performance comparable to CRF-based methods in the same task. Finally, Constant et al. (2012) used a reranker for MWEs in an *n*-best parser. We combine these ideas by reranking the *n* best CRF VMWE predictions for each sentence using regression scores computed from vectors that represent different combinations of VMWE candidates. The vectors are computed from a large corpus, namely EUROPARL's individual language subcorpora.
# 3 VMWE identification via CRF

We decided to model the problem of VMWE identification as a sequence labelling and classification problem. We operationalise our solution through CRFs (Laferty et al. 2001), implemented using the CRF++ system.<fnref n="3" /> CRFs have been successfully applied to such sequence-sensitive NLP tasks such as segmentation, named-entity recognition (Han et al. 2013; 2015) and part-of-speech (POS) tagging. Our team attempted 15 out of the 18 languages involved in the shared task. It should be noted that of these 15 languages, four (Czech, Farsi, Maltese and Romanian) were provided without syntactic dependency information, although morphological information (i.e. tokens' lemmas and POS) was indeed supplied. The data for the languages we did not attempt (Bulgarian, Hebrew and Lithuanian) lacked even morpho-syntactic information, leaving the CRF with only tokens as features; so we felt that we were unlikely to obtain good results with them, and chose to focus on the richer datasets.

## 3.1 Features

We assume that features based on the relationships between the different types of morpho-syntactic information provided by the organisers will help identify VMWEs. Ideally, one feature set (or *feature template* in the terminology of CRF++) per language should be developed. Due to time constraints, we developed a feature set for three languages (German, French and Polish), then for every language the feature template that performed best in cross-validation among these three was selected.

For each token in the corpus, the direct linguistic features available are its word surface (W), word lemma (L) and POS (P). In the languages where syntactic dependency information is provided, each token also has its head's word surface (HW), its head's word lemma (HL), its head's POS (HP) and the dependency relation between the token and its head (DR). It is possible to create CRF++ feature templates that combine these features. In addition, it is also possible to use the predicted output label of the previous token (B).

The three final feature templates, which we call FT3, FT4 and FT5,<fnref n="4" /> are shown in Table 1. Whilst the feature templates in this table are expressed in the CRF++ format, a comment (starting with #) at each feature (line) expresses the type of feature and its relative position to the current token. For instance, L-2 refers to

<footnote n="3">³https://taku910.github.io/crfpp/. Release 0.58, Last verified 2017-12-29.</footnote>

<footnote n="4">⁴The feature template numbering starts at 3 for consistency with their original description in Maldonado et al. (2017).</footnote>
6 Semantic reranking of CRF label sequences for VMWE identification

<table><caption>Table 1: CRF++ Feature Templates developed. Example: template U32:%x[0,3] indicates current token (row 0, i.e. current row) and lemma (column 3) while template U41:%x[-1,2] refers to the previous token (row -1 from current row) and POS tag (column 2), etc.</caption><thead><tr><th>FT3</th><th>FT4</th><th>FT5</th></tr></thead><tbody><tr><td># L-2<br>U30:%[-2,3]<br># L-1<br>U31:%[-1,3]<br># L<br>U32:%[0,3]<br># L+1<br>U33:%[1,3]<br># L+2<br>U34:%[2,3]<br># L-2/L-1<br>U35:%[-2,3]/%[-1,3]<br># L-1/L<br>U36:%[-1,3]/%[-1,3]<br># L/L+1<br>U37:%[0,3]/%[1,3]<br># L+1/L+2<br>U38:%[1,3]/%[2,3]</td><td># L-2<br>U30:%[-2,3]<br># L-1<br>U31:%[-1,3]<br># L<br>U32:%[0,3]<br># L+1<br>U33:%[1,3]<br># L+2<br>U34:%[2,3]<br># L-2/L-1<br>U35:%[-2,3]/%[-1,3]<br># L-1/L<br>U36:%[-1,3]/%[0,3]<br># L/L+1<br>U37:%[0,3]/%[1,3]<br># L+1/L+2<br>U38:%[1,3]/%[2,3]</td><td># L-2<br>U30:%[-2,3]<br># L-1<br>U31:%[-1,3]<br># L<br>U32:%[0,3]<br># L+1<br>U33:%[1,3]<br># L+2<br>U34:%[2,3]<br># L-2/L-1<br>U35:%[-2,3]/%[-1,3]<br># L-1/L<br>U36:%[-1,3]/%[0,3]<br># L/L+1<br>U37:%[0,3]/%[1,3]<br># L+1/L+2<br>U38:%[1,3]/%[2,3]</td></tr><tr><td># P<br>U00:%[0,2]<br># HL/DR<br>U01:%[0,4]/%[0,6]<br># P/DR<br>U02:%[0,2]/%[0,6]<br># HP/DR<br>U03:%[0,5]/%[0,6]</td><td># HL/DR<br>U01:%[0,4]/%[0,6]<br># P/DR<br>U02:%[0,2]/%[0,6]<br># HP/DR<br>U03:%[0,5]/%[0,6]</td><td># HL/DR<br>U01:%[0,4]/%[0,6]<br># P/DR<br>U02:%[0,2]/%[0,6]<br># HP/DR<br>U03:%[0,5]/%[0,6]</td></tr><tr><td># Previous token's label<br>B</td><td># Previous token's label<br>B</td><td># Previous token's label<br>B</td></tr><tr><td># P-2<br>U40:%[-2,2]<br># P-1<br>U41:%[-1,2]<br># P<br>U42:%[0,2]<br># P+1<br>U43:%[1,2]<br># P+2<br>U44:%[2,2]<br># P-1/P<br>U45:%[-1,2]/%[0,2]<br># P/P+1<br>U46:%[0,2]/%[1,2]</td><td># P-2<br>U40:%[-2,2]<br># P-1<br>U41:%[-1,2]<br># P<br>U42:%[0,2]<br># P+1<br>U43:%[1,2]<br># P+2<br>U44:%[2,2]<br># P-1/P<br>U45:%[-1,2]/%[0,2]<br># P/P+1<br>U46:%[0,2]/%[1,2]</td><td># P-2<br>U40:%[-2,2]<br># P-1<br>U41:%[-1,2]<br># P<br>U42:%[0,2]<br># P+1<br>U43:%[1,2]<br># P+2<br>U44:%[2,2]<br># P-1/P<br>U45:%[-1,2]/%[0,2]<br># P/P+1<br>U46:%[0,2]/%[1,2]</td></tr><tr><td># L/HP<br>U52:%[0,3]/%[0,5]</td><td></td><td></td></tr></tbody></table>
the lemma of the token at position *i* – 2 relative to the current token at position *i*, P+1 refers to the part of speech of the token at position *i* + 1, and HL/DR refers to the combination of head’s lemma of the current token and the dependency relation between the current token and its head.<fnref n="5" />

FT5 is based on FT4, which in turn, is based on FT3. FT3 is itself based on the CRF++ example feature template, commonly used in NER experiments. The difference between FT3 and this example feature template is that FT3 exploits syntactic dependency information.

We conducted preliminary 5-fold cross validation experiments on German, French and Polish training data using the FT3 template. We then started tweaking the template independently for each of these three languages based on successive 5-fold cross validation results. This exercise resulted in the three final templates: FT3 for French and FT5 for German and Polish. Given that FT4’s performance was very similar to that of FT5, we decided not to discard it.

During this preliminary experimentation, we also observed that templates exploiting token word surface features (W) performed unsurprisingly worse than those based on token lemmas (L) and POS (P). Templates using head features (HL, HP, DR) in addition to token features (L, P) fared better than those relying on token features only.

We also attempted to test the assumption that these feature templates would perform similarly in other languages of the same language family. That is, that FT3 would also perform better than FT4 and FT5 in other Romance languages and that FT5 would score higher than FT3 and FT4 in the rest of the languages. So we conducted a final set of 5-fold cross validation experiments on all 15 languages, this time trying each feature template (FT3, FT4 and FT5) independently on each language. The results are shown in Table 2. The F1 scores in bold italic are the maximum scores per language. For each given language, the results of the three templates are very similar. Therefore we are not able to comfortably confirm or refute our language family assumption. Nonetheless, we decided to choose for the final challenge the template that maximised the MWE-based F1 score for each language.

In order to use these templates with the provided data, we combined the supplied PARSEMETSV (VMWE annotations) and CONLLU files (linguistic features) into a single file. The training and blind test files were combined separately. The resulting file is also columnar in format, with column 0 representing the token ID as per the original PARSEMETSV file, column 1 the token’s surface form, col-

<footnote n="5">⁵CRF++ feature template format described in https://taku910.github.io/crfpp/#templ Last verified 2017-12-19.</footnote>
6 Semantic reranking of CRF label sequences for VMWE identification

<table><caption>Table 2: F1 scores from cross validation experiments on 15 languages using feature templates FT3, FT4 and FT5.</caption><thead><tr><th>Lang</th><th>CS</th><th>DE</th><th>EL</th><th>ES</th><th>FA</th></tr><tr><th colspan="6">Eval. Type MWE Token MWE Token MWE Token MWE Token MWE Token</th></tr></thead><tbody><tr><th>FT3</th><td>54.23</td><td>70.79</td><td>23.84</td><td>39.02</td><td><strong>50.41</strong></td><td><strong>62.03</strong></td><td><strong>56.04</strong></td><td>60.74</td><td>76.09</td><td>83.52</td></tr><tr><th>FT4</th><td>55.91</td><td>71.81</td><td>24.41</td><td>39.62</td><td>50.16</td><td>61.76</td><td>55.72</td><td>60.77</td><td>77.88</td><td>84.75</td></tr><tr><th>FT5</th><td><strong>57.12</strong></td><td>72.57</td><td>25.23</td><td><strong>40.53</strong></td><td>49.72</td><td>62.02</td><td>55.70</td><td><strong>61.00</strong></td><td><strong>78.61</strong></td><td><strong>85.24</strong></td></tr></tbody></table>

<table><thead><tr><th>Lang</th><th>FR</th><th>HU</th><th>IT</th><th>MT</th><th>PL</th></tr><tr><th colspan="6">Eval. Type MWE Token MWE Token MWE Token MWE Token MWE Token</th></tr></thead><tbody><tr><th>FT3</th><td>3.99</td><td>6.71</td><td>66.03</td><td>70.24</td><td><strong>65.85</strong></td><td>75.81</td><td>81.44</td><td>80.96</td><td>28.31</td><td>31.12</td></tr><tr><th>FT4</th><td>4.20</td><td>6.92</td><td>65.70</td><td>70.56</td><td>65.62</td><td>76.07</td><td>81.30</td><td>81.00</td><td>28.19</td><td>30.80</td></tr><tr><th>FT5</th><td><strong>5.35</strong></td><td><strong>7.96</strong></td><td><strong>66.28</strong></td><td><strong>71.21</strong></td><td>65.6</td><td><strong>76.08</strong></td><td><strong>81.86</strong></td><td><strong>81.76</strong></td><td><strong>28.68</strong></td><td><strong>31.51</strong></td></tr></tbody></table>

<table><thead><tr><th>Lang</th><th>PT</th><th>RO</th><th>SL</th><th>SV</th><th>TR</th></tr><tr><th colspan="6">Eval. Type MWE Token MWE Token MWE Token MWE Token MWE Token</th></tr></thead><tbody><tr><th>FT3</th><td>56.56</td><td>64.81</td><td>75.87</td><td><strong>83.76</strong></td><td><strong>37.06</strong></td><td><strong>48.90</strong></td><td>19.68</td><td>20.09</td><td><strong>49.71</strong></td><td>59.42</td></tr><tr><th>FT4</th><td><strong>56.73</strong></td><td>65.32</td><td>75.91</td><td>83.47</td><td>34.04</td><td>46.70</td><td><strong>24.47</strong></td><td><strong>24.56</strong></td><td>49.49</td><td><strong>59.43</strong></td></tr><tr><th>FT5</th><td>56.64</td><td><strong>65.52</strong></td><td><strong>76.00</strong></td><td>83.69</td><td>34.65</td><td>47.57</td><td>22.09</td><td>22.33</td><td>49.46</td><td>59.38</td></tr></tbody></table>

umn 2 the token's POS tag (P), column 3 the lemma (L), column 4 the head's lemma (HL), column 5 the head's POS tag (HP), column 6 the dependency relationship between the token and its head (DR), and column 7 the VMWE label for the token.

The VMWE label was changed from the numerical values in the PARSEMETSV file to “B” for the tokens that start a VMWE and “I” for subsequent tokens that belong to a VMWE. This labelling scheme (usually called BIO, for *Begin, Inside, Outside*) is common in CRF-based implementations of NER systems. The BIO scheme can represent several consecutive VMWEs but cannot represent embedded or overlapping VMWEs, so these were ignored and a single B or I label was used for overlapping tokens.<fnref n="6" /> The proportion of overlapping VMWEs is between 2 and 6% for Czech, German, Spanish, French, Italian, Polish, Portuguese and

<footnote n="6">⁶Remark: Schneider et al. (2014) proposes several tagging schemes, some using special “o” labels for discontinuous expressions; since we use the most simple scheme (BIO), the words which appear between the lexicalized components of the expressions are labeled with a regular “O”.</footnote>
Swedish, and it is even less for the rest of the languages we studied (see Maldon-
do & QasemiZadeh 2018 [this volume] for further details). Because of these low
proportions, we consider embedded/overlapping VMWEs to have only a small
negative impact on our system’s performance. Therefore, we decided to ignore
them.

Our system does not distinguish among the different categories of VMWEs,
treating them all equally. The templates in Table 1 make reference to each feature
based on the position of the current token and the column in which they appear
in the input file.

3.2 CRF results

Table 3 shows, under the “CRF only” category, the Precision (P), Recall (R) and F1 (F) scores on the test set based on the shared task two evaluation modalities, MWE-based and token-based.<fnref n="7" /> On token-based evaluation, our system was ranked in first place in French, Polish and Swedish, second place in eight languages (Czech, Greek, Farsi, Maltese, Portuguese, Romanian, Spanish and Turkish) and third in three (German, Italian and Slovenian). For MWE-based scores, our system ranked second place in nine languages (Farsi, French, Italian, Maltese, Polish, Portuguese, Slovenian, Spanish and Swedish) and third in four languages (Czech, German, Hungarian and Turkish). If all languages’ scores are averaged per system, our system ranks in third place on both token-based and MWE-based evaluation (see Maldonado & QasemiZadeh 2018 [this volume] for average scores per system).

4 Semantic reranker

4.1 Motivations

The semantic reranker is the second component of the VMWE detection system. While the first component (CRF) offers a decent level of performance in its predictions, the reranker is intended to fix as many mistaken predictions as possible, by exploiting features that CRF are poorly equipped to deal with. These features are based on a distributional semantics approach (Schütze 1998; Maldonado & Emms 2011): they rely on comparing context vectors which are extracted from a *reference corpus* (usually a large third-party corpus). As it is often the case with

<footnote n="7">⁷We did not include the languages for which there is no Europarl data in this table.</footnote>
6 Semantic reranking of CRF label sequences for VMWE identification

<table><caption>Table 3: Performance by language according to official evaluation measures for the CRF component alone and the two components together (CRF and semantic reranker) P/R/F stands for precision/recall/f-score. All the values are expressed as percentages. The last two rows show the macro-average performance over the 12 languages.</caption><thead><tr><th rowspan="2">Lang.</th><th rowspan="2">Eval.<br>Type</th><th colspan="3">CRF only</th><th colspan="3">CRF + Reranker</th><th colspan="3">Improvement (%)</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td rowspan="2">CS</td><td>MWE</td><td>59.3</td><td>56.2</td><td>57.7</td><td>79.8</td><td>63.4</td><td>70.6</td><td>+34.5</td><td>+12.8</td><td>+22.4</td></tr><tr><td>Token</td><td>81.9</td><td>65.6</td><td>72.9</td><td>86.4</td><td>65.7</td><td>74.6</td><td>+5.4</td><td>+0.2</td><td>+2.4</td></tr><tr><td rowspan="2">DE</td><td>MWE</td><td>33.1</td><td>17.4</td><td>22.8</td><td>53.5</td><td>19.6</td><td>28.7</td><td>+61.9</td><td>+12.6</td><td>+25.9</td></tr><tr><td>Token</td><td>70.6</td><td>28.4</td><td>40.5</td><td>72.2</td><td>25.6</td><td>37.8</td><td>+2.3</td><td>-9.7</td><td>-6.6</td></tr><tr><td rowspan="2">EL</td><td>MWE</td><td>34.4</td><td>28.8</td><td>31.3</td><td>45.1</td><td>32.2</td><td>37.6</td><td>+31.2</td><td>+11.8</td><td>+19.9</td></tr><tr><td>Token</td><td>53.8</td><td>36.0</td><td>43.1</td><td>54.8</td><td>36.3</td><td>43.6</td><td>+1.8</td><td>+0.7</td><td>+1.1</td></tr><tr><td rowspan="2">ES</td><td>MWE</td><td>61.1</td><td>34.8</td><td>44.3</td><td>66.7</td><td>38.8</td><td>49.0</td><td>+9.2</td><td>+11.5</td><td>+10.6</td></tr><tr><td>Token</td><td>74.5</td><td>36.7</td><td>49.2</td><td>74.2</td><td>38.9</td><td>51.1</td><td>-0.3</td><td>+6.0</td><td>+3.9</td></tr><tr><td rowspan="2">FR</td><td>MWE</td><td>61.5</td><td>43.4</td><td>50.9</td><td>75.6</td><td>47.6</td><td>58.4</td><td>+22.9</td><td>+9.7</td><td>+14.8</td></tr><tr><td>Token</td><td>80.9</td><td>49.6</td><td>61.5</td><td>82.6</td><td>50.0</td><td>62.3</td><td>+2.1</td><td>+0.7</td><td>+1.2</td></tr><tr><td rowspan="2">HU</td><td>MWE</td><td>75.7</td><td>59.9</td><td>66.9</td><td>74.5</td><td>63.3</td><td>68.5</td><td>-1.5</td><td>+5.7</td><td>+2.4</td></tr><tr><td>Token</td><td>78.5</td><td>57.1</td><td>66.1</td><td>77.3</td><td>61.5</td><td>68.5</td><td>-1.5</td><td>+7.8</td><td>+3.7</td></tr><tr><td rowspan="2">IT</td><td>MWE</td><td>61.7</td><td>14.2</td><td>23.1</td><td>70.8</td><td>9.2</td><td>16.3</td><td>+14.6</td><td>-35.2</td><td>-29.5</td></tr><tr><td>Token</td><td>69.6</td><td>15.3</td><td>25.1</td><td>76.1</td><td>9.7</td><td>17.3</td><td>+9.2</td><td>-36.4</td><td>-31.2</td></tr><tr><td rowspan="2">PL</td><td>MWE</td><td>78.0</td><td>60.2</td><td>68.0</td><td>83.7</td><td>63.8</td><td>72.4</td><td>+7.4</td><td>+6.0</td><td>+6.6</td></tr><tr><td>Token</td><td>87.4</td><td>62.3</td><td>72.7</td><td>87.0</td><td>63.9</td><td>73.7</td><td>-0.5</td><td>+2.6</td><td>+1.3</td></tr><tr><td rowspan="2">PT</td><td>MWE</td><td>64.1</td><td>53.2</td><td>58.1</td><td>75.9</td><td>57.2</td><td>65.2</td><td>+18.3</td><td>+7.5</td><td>+12.2</td></tr><tr><td>Token</td><td>83.5</td><td>60.5</td><td>70.2</td><td>82.2</td><td>60.1</td><td>69.4</td><td>-1.5</td><td>-0.8</td><td>-1.1</td></tr><tr><td rowspan="2">RO</td><td>MWE</td><td>75.5</td><td>71.4</td><td>73.4</td><td>90.4</td><td>77.6</td><td>83.5</td><td>+19.8</td><td>+8.7</td><td>+13.8</td></tr><tr><td>Token</td><td>88.3</td><td>76.4</td><td>81.9</td><td>91.8</td><td>77.9</td><td>84.3</td><td>+4.0</td><td>+2.1</td><td>+3.0</td></tr><tr><td rowspan="2">SL</td><td>MWE</td><td>51.4</td><td>29.0</td><td>37.1</td><td>68.6</td><td>32.4</td><td>44.0</td><td>+33.5</td><td>+11.7</td><td>+18.7</td></tr><tr><td>Token</td><td>72.9</td><td>32.6</td><td>45.1</td><td>75.4</td><td>32.4</td><td>45.3</td><td>+3.5</td><td>-0.8</td><td>+0.5</td></tr><tr><td rowspan="2">SV</td><td>MWE</td><td>48.6</td><td>22.0</td><td>30.3</td><td>48.7</td><td>23.7</td><td>31.9</td><td>+0.2</td><td>+7.7</td><td>+5.2</td></tr><tr><td>Token</td><td>52.5</td><td>22.5</td><td>31.5</td><td>52.1</td><td>24.1</td><td>32.9</td><td>-0.7</td><td>+7.0</td><td>+4.6</td></tr><tr><td rowspan="2">macro-<br>average</td><td>MWE</td><td>58.7</td><td>40.9</td><td>48.2</td><td>69.4</td><td>44.1</td><td>53.9</td><td>+18.3</td><td>+7.8</td><td>+11.9</td></tr><tr><td>Token</td><td>74.5</td><td>45.3</td><td>56.3</td><td>76.0</td><td>45.5</td><td>56.9</td><td>+2.0</td><td>+0.6</td><td>+1.1</td></tr></tbody></table>
complex machine learning problems, the orthogonality of the information, obtained on the one hand from the sequential CRF model and computed on the other hand from an independent semantic vector space, proves fruitful; this will be demonstrated in §5.

## 4.2 Design

Intuitively, the goal is to estimate whether a candidate expression should be considered a MWE or not. Thus, the core part of the reranker is to generate features which are relevant to assess the likelihood of a given candidate MWE being an actual MWE. These expression-level features are then combined to produce a set of sentence-level features, which in turn are used to train a decision tree regression model. Later, this model is applied to the candidate sequences provided by the CRF component, so that their predicted scores can be compared; finally, the sequence with the highest score (among a set of *N* candidate sequences) is selected as the final answer. This is why the reranker receives the output produced by CRF++ in the form of the 10 most likely predictions for every sentence.

## 4.3 A distributional semantics approach

Distributional semantics is a well known method to represent the meaning of a single word as a context vector (Schütze 1998). However, our algorithm must calculate a context vector for the multiple words in an expression whether they are continuous or discontinuous (see §5.7). This raises new questions about the optimal way to take the co-occurrences into account in the algorithm. To the authors' knowledge, such questions have not been previously studied.

In order to calculate the context vector, we count the words co-occurring with the MWE in a *reference corpus* (see §4.4). Given a candidate MWE identified by the CRF, the reference corpus is searched for every occurrence of this MWE. This consists in matching the words which compose the expression; because MWEs are not continuous in general, the matching only requires that the words appear in the same order in a sentence, i.e. allows other words to appear between the words of the expression.<fnref n="8" /> As a consequence, false positive matches may happen, in particular if the words of the MWE appear in a sentence by chance, without any direct relation between them (neither syntactic or semantic).

<footnote n="8">⁸This implies that ambiguities can arise if the same word is used several times in a sentence. In such cases, the matching always selects the shortest possible distance between the first and the last word if the MWE.</footnote>
Once an occurrence of the MWE expression is identified, the words which appear within a fixed-size window around its lexicalized components are counted as its co-occurrences. The number of times a given word co-occurs with the MWE across all the sentences in the reference corpus is recorded; by doing this for every word which co-occurs with a component of an MWE, we obtain a vector which represents the meaning of the MWE. However this method is traditionally used with single words, and its adaptation to sequences of words raises new questions. This is why we propose several options, as detailed below, that determine how the co-occurrences are extracted and counted. The combinations of these options offer multiple possibilities that we analyse in §5.8.

* *IncludeWordsInExpression (WIE)*: This option determines whether to add the actual expression words to the context vector as contexts of other components or to exclude them, when they fall within the scope of the window.

* *MultipleOccurencesPosition (MO)*: This option determines whether or not to count multiple occurrences of a word within the window scope. Such a word can be part of the actual expression or not, depending on the first option.

* *ContextNormalization (CN)*: This option determines whether the frequencies of the co-occurrences are normalized for every occurrence of the expression, i.e. divided by the number of co-occurrences found.<fnref n="9" /> This is meant to account for the differences in the number of context words across different sentences or expressions (since a longer expression generally has more words in its context window).

Table 4 illustrates the impact of these options when applied to the following example, in which the words of the expressions are in bold:

(1) French (Indo-European; FR training data)

> Les gens ne se rendent pas bien compte du coût énorme de
> The people NEG self give not well account of the cost enormous of
> l' opération.
> the operation.

> 'People do not fully realize the enormous cost of the operation.'

<footnote n="9">⁹Otherwise absolute frequencies are used at the level of a single expression. In both cases a different stage of normalization is carried out once all the occurrences of the expression have been collected, where the values are divided the number of occurrences.</footnote>
<table><caption>Table 4: Example of how context words are counted (for one occurrence of the expression). A context window of size 2 is assumed on both sides. IWIE and MO represent the options *IncludeWordsInExpression*, *MultipleOccurrences*, respectively. The values are represented in the case where *ContextNormalization* is false, i.e. they are not normalized; in the case where *ContextNormalization* is true, every value is divided by the sum of the values in the row.</caption><thead><tr><th>IWIE</th><th>MO</th><th>gens</th><th>ne</th><th>se</th><th>rendent</th><th>pas</th><th>bien</th><th>compte</th><th>du</th><th>coût</th></tr></thead><tbody><tr><td>False</td><td>False</td><td>1</td><td>1</td><td>0</td><td>0</td><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td></tr><tr><td>True</td><td>True</td><td>1</td><td>2</td><td>1</td><td>1</td><td>3</td><td>2</td><td>0</td><td>1</td><td>1</td></tr><tr><td>False</td><td>True</td><td>1</td><td>2</td><td>0</td><td>0</td><td>3</td><td>2</td><td>0</td><td>1</td><td>1</td></tr><tr><td>True</td><td>False</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td></tr></tbody></table>

## 4.4 Third-party reference corpus: Europarl

We use Europarl (Koehn 2005) as reference corpus, because it is large and conveniently contains most of the languages addressed in the shared task data. However there is no Europarl data for Farsi, Maltese and Turkish. This is why these languages are excluded from this part of the process. For each of the 12 remaining languages, we use only the monolingual Europarl corpus, and we tokenise it using the generic tokeniser provided by the shared task organisers. However this tokeniser was not necessarily used for the shared task data, because each language team was free to use their own tools or to use existing pre-tokenised data.<fnref n="10" /> Therefore, discrepancies are to be expected between the tokenisation of the shared task corpus and the one performed on Europarl. Additionally, Europarl consists of raw text, so the reranker cannot use the morpho-syntactic information (POS tags and lemmas) provided with the input data.

## 4.5 Features

The core component of the reranker computes a set of feature values for every sequence proposed by the CRF; these features are later used for training or applying the decision tree model. First, these features include a few simple values which are either available directly from the CRF output or easily computed from the reference corpus:

* The confidence score assigned by the CRF component to the sequence (i.e. the probability of the sequence according to the CRF);

<footnote n="10">¹⁰For instance, the French dataset originates from several existing corpora; their tokenisation follows language-specific patterns which cannot be obtained with a generic tokeniser.</footnote>
* The number of expressions labeled by the CRF in the sequence;

* The minimum/mean/maximum number of words in the expression, over the expressions of the sequence;

* The minimum/mean/maximum frequency of the expression in the reference corpus, over the expressions of the sequence.

The original intuition behind the semantic reranker is to calculate features which give indications about the level of compositionality between the words in the expression. The underlying assumption is that such features might help detect at least non-compositional expressions. In the past this idea has been used successfully to detect collocations among two-words sequences: in Maldonado & Emmes (2011), for every two-words collocation *xy* word vectors are computed for the word *x*, the word *y* and the full sequence *xy*; the cosine similarity is used to compare (1) the vector for *x* with the vector for *xy* and (2) the vector for *y* with the vector for *xy*. In a compositional expression both scores are expected to be high, because the semantic overlap between the full expression and each of its words is high, as opposed to a non-compositional expression. Hence the average similarity score between (1) and (2) can be used as a measure of the compositionality of the pair.

This idea is generalized to the case of VMWEs of any length by comparing different parts of the expression against the full candidate expression; we call *pseudo-expressions* these “parts of the expression”. Every pseudo-expression extracted from the candidate expression is analyzed as if it was a regular candidate expression: the reference corpus is searched to identify all its occurrences, then a context vector is built, as described in §4.3. Examples of pseudo-expressions based on the expression *avoir sa place* (‘have their place’) are provided below.

Four different similiarity measures have been implemented for comparing pairs of context vectors: Jaccard index, min/max similarity, and cosine similarity with or without inverse document frequency (IDF) weighting. Additionally to the semantic context vectors comparison, the frequencies of the pseudo-expressions are compared with respect to the frequency of the full MWE: this feature is the ratio of the full expression frequency divided by the pseudo-expression frequency. Finally every candidate MWE is compared against every other candidate MWE in the 10 predicted sequences. Thus, for each of the three groups of features below, the frequency ratio and the similarity score obtained between the context vectors of the pseudo-MWEs and the full MWE are added as features.

* Features comparing each pseudo-MWE consisting of a single word of the MWE against the full MWE. Example: for the candidate expression *avoir sa*
place ('have their place'), three comparisons are performed: avoir vs. avoir sa place, sa vs. avoir sa place, place vs. avoir sa place.

* Features comparing each pseudo-MWE consisting of the MWE minus one word against the full MWE. Example: for the candidate expression avoir sa place, three comparisons are performed: sa place vs. avoir sa place, avoir place vs. avoir sa place, avoir sa vs. avoir sa place.

* Features comparing one of the other MWEs found in the 10 predicted sequences<fnref n="11" /> against the current MWE.<fnref n="12" />

The main difficulty in representing a predicted sequence as a fixed set of features is that each sentence can contain any number of MWEs, and each MWE can contain any number of words. We opted for a simple method which consists in “summarising” any non-fixed number of features with three statistics: minimum, mean and maximum. For instance, the similarity scores between each individual word and the corresponding MWE ($n$ scores, where $n$ is the number of words) are represented with these three statistics, each computed across these $n$ scores. Similarly, if the sequence contains $m$ expressions, a feature $f$ has $m$ values $f_1, ..., f_m$, with each value $f_i$ corresponding to one expression; here again the minimum, mean and maximum are calculated across these $m$ values, i.e. every expression-level feature $f$ is converted into three features $f_{min}, f_{mean}$ and $f_{max}$ in the final set of sequence-level features.

## 4.6 Supervised regression and cross-validation process

As explained above, the reranker has to assign a score to each of the 10 sequences provided by the CRF component, in order to select the highest one. We use regression, rather than classification, because a categorical answer would cause some sentences to have either no positive answer or multiple positive answers in its set of predicted sequences, thus making the decision impossible.

<footnote n="11">¹¹This includes the 9 other sequences as well as the other candidate expressions in the current one, if any.</footnote>

<footnote n="12">¹²The last group is not meant to measure compositionality of the expression. The rationale is that such features might help eliminate candidate expressions which are very different from the other candidates, under the hypothesis that likely candidates tend to share words together (such features are unlikely to help with sentences which contain several expressions). As explained in §5.5, this group of features turned out to be the least useful.</footnote>
In training mode, an instance (i.e. sequence) is assigned the score 1 if it cor-
responds exactly to the sequence in the gold standard, or 0 otherwise.<fnref n="13" /> Since
the 10 candidate sequences are all different, there should always be only one cor-
rect sequence. This way, the regression model assigns scores in [0, 1] to every
instance, with the highest values expected to be more likely correct answers. As
regression model, we use the Weka (Hall et al. 2009) implementation of decision
trees regression (Quinlan 1992): the final score is determined by the decision tree
rules, then by a linear combination of the features values. This choice has the
advantage of simplicity, but also of the interpretability of decision trees models.
Of course, other regression models could be considered as well.

Because of the two-components approach (CRF and reranker), the training
process requires special care. In order to train the reranker with the kind of data
that it will receive in testing mode, predictions from the CRF are needed. The
testing data cannot be used for this, which is why we use 5 fold cross-validation
on the training data: on each of the five 20% subsets of the data, a model trained
from the 80% data left is applied. This way the reranker can be fed with real
predictions for the full training data, including the classification errors of the CRF.
If necessary, the cross-validation process can be repeated, for instance to tune the
parameters of the reranker. Otherwise, the reranker can simply be trained with
the full training set of predictions, then applied to the test set (after the CRF
predictions have been predicted on this test set).

5 Results and analyses

In this section we present detailed results of our system and analyze how the
reranker helps improving performance with respect to various factors. All the
experiments presented in this section have been carried out using the official
PARSEME shared task 2017 training and test data. Unless otherwise stated, we
use the same “standard” configuration of options for the reranker throughout the
experiments (e.g. context window size, minimum frequency, etc.).

5.1 Results

Table 3 shows the performance of both the CRF component and the semantic
reranker by language, as well as the improvement brought by the reranker. De-
spite differences by language, all but one language (Italian) show a significant

<footnote n="13"><sup>13</sup>It might happen that none of the 10 sequences corresponds to the gold sequence; in such cases all the instances are left as negative cases.</footnote>
increase in MWE-level F-score, with a macro-average F-score improvement of +11.9%. The increase in token-level F-score is much smaller, with even a decrease in a few languages; the macro-average token-level F-score improvement is only +1.1%. This means that the reranker does not drastically change the expressions predicted by the CRF (hence the little improvement at the token level), but instead tends to fix the proposed expressions by finding their exact boundaries, thus bringing the MWE F-score closer to the token F-score. This is because the top 10 CRF predictions tend to be variants of one another, rather than drastically different labelings; they frequently focus on the same part(s) of the sentence, varying only by labeling one or two words differently; thus the reranker seldom introduces a new expression that the top prediction would have missed, but can select a more likely variant among the remaining 9 predictions. This hypothesis is also backed by the observation that the increase in precision is larger in general than the increase in recall: the reranker mostly follows the top CRF prediction and possibly fixes it, hence turning a false positive instance into a true positive. These observations are consistent with the design of the system and validate the reranking approach in general.

## 5.2 Error analysis methodology

The performance of the reranker can be evaluated straightforwardly using the official evaluation measures, as presented above; these measures are useful to compare against other systems or between datasets. However, in order to get a clear understanding of how the system works, we also look at the different combinations of error status between the CRF component and the semantic reranker: the CRF is said to be right if and only if it ranks the actual (gold standard) answer as its top prediction; similarly, the reranker is right if and only if it assigns the top score to the actual answer. Thus the four following categories are defined:

* RR stands for right-right, which means that the CRF component ranked the right answer as first sequence (first *R*) and the reranker kept it as the final answer (second *R*).

* WR stands for wrong-right: the CRF answer was wrong, but the reranker successfully selected an alternative answer.

* RW stands for right-wrong: in this case the reranker mistakenly changed the CRF answer, which was correct in the first place.

* WW stands for wrong-wrong: the CRF answer was wrong, and the reranker either kept it or changed it to another wrong answer.
6 Semantic reranking of CRF label sequences for VMWE identification

These four categories cover all the cases, except when the correct answer is
not present in the 10 most likely sequences that the CRF component provides.
For this case we use the special label **GoldNotIn10**.

It is worth noticing that the reranker works at the sentence level (as opposed
to the expression level or the token level). This is why these categories apply
to complete sentences, in accordance with the design of the two-components
system. In particular, the number of expressions in a sentence is not taken into
account in this categorization. As a consequence, sentences which contain no
expression at all are considered as important as sentences which contain one or
multiple expressions.

<figure><img src="image_37.png" /><figcaption>Figure 1: Reranker score w.r.t CRF confidence for the gold sequence in every sentence, by error type (all languages together). A sentence is represented in the right half (resp. left half) if the CRF assigned a high confidence (resp. low) to its gold sequence, i.e. the CRF answer is correct (resp. incorrect). Similarly, a sentence appears in the top half (resp. bottom half) if the reranker assigned a high score (resp. low) to its gold sequence, i.e. the reranker answer is correct (resp. incorrect).<fnref n="14" /></figcaption></figure>

Figure 1 gives an overview of how the reranker improves performance over
the CRF predictions. Every point in this graph represents a sentence, positioned
according to the CRF confidence (X axis) and reranker final score (Y axis) for
its gold sequence. This way, if the CRF finds the right answer for a sentence,

<footnote n="14">¹⁴Remark: Category *GoldNotIn10* is not visible on this graph, since in such cases the gold sequence cannot be assigned a CRF confidence nor a reranker score.</footnote>
i.e. the gold sequence obtains the highest confidence among the 10 sequences, it is represented in the right half of the graph, and conversely for wrong answers. If the reranker finds the right answer, it assigns a high score to the gold sequence, so the sentence appears in the top half, and conversely. This explains why the four error types appear mostly clustered each in its own quadrant: the top right quadrant contains sentences for which the correct sequence is recognized by both the CRF and the reranker (hence in the **RR** error category), and the bottom left contains sentences for which neither finds the right answer (**WW**). The last two quadrants are the interesting ones, since this is where the reranker changes the CRF prediction to another sequence: in the top left quadrant, the **WR** points correspond to successful changes, whereas the **RW** cases, in the bottom right quadrant, correspond to mistakes introduced by the reranker.<fnref n="15" /> It can be observed that the former category outnumbers the latter, thus confirming the positive contribution of the reranker.

## 5.3 Insight: what the reranker actually does

The vast majority of the sentences (85.3% of a dataset in average) fall into the **RR** category, i.e. the reranker simply confirms the correct CRF answer. The **WW** cases account for 7.6% of the sentences, and the *GoldNotIn10* cases for 4.4%. The reranker actually changes only 2.7% of the answers, and when it does, it does it correctly 81.5% of the time (2.2% **WR**, 0.5% **RW**).

Figure 2 shows how the positions of the sequences selected by the reranker are distributed. The reranker strongly favors top positions for its selected sequence; more precisely, as the position of the sequence decreases, the number of sentences for which this position is selected decreases exponentially (this is why a logarithmic scale is used in Figure 2). This trend is regular from the top position, which is selected 92.5% of the time, down to the 9th position, selected in only two cases.<fnref n="16" /> This shows that increasing the number of candidate sequences supplied by the CRF (10 in all our experiments) would not improve the performance of the reranker, since it seldom selects a sequence associated with a low CRF confidence (the importance of the CRF confidence as a feature is shown more clearly

<footnote n="15">¹⁵This graph can give the impression that one could easily prevent **RW** mistakes (bottom right quadrant) by accepting any CRF answer with high confidence, but this is due to the fact that only the gold sequence is represented here. Thus, for cases where the gold sequence has low confidence, some other (wrong) sequence has high confidence. Therefore selecting the highest confidence sequence would simply prevent any reranking to happen, in particular for the CRF mistakes which can be fixed.</footnote>

<footnote n="16">¹⁶The small rebound in position 10 is not significant, as it represents only 3 cases.</footnote>
in §5.5 below). Finally the fact that the reranker makes more correct changes than mistakes is confirmed in this graph again, by observing that the number of *WR* cases is higher or equal than the number of *RW* cases at every position.

<figure><img src="image_38.png" /><figcaption>Figure 2: Distribution of the position selected by the reranker (logarithmic scale, all languages together). For every position, the large dark grey bar shows the total number of sentences, while the coloured bars show the number of sentences for every possible error type.<fnref n="17" /></figcaption></figure>

The distribution of errors is not uniform over the data, and the number of expressions in a sentence is one of the most obvious factors: For example, Figure 3 shows that 96% of the sentences with no expression in the gold sequence are correctly identified by both the CRF and the reranker (RR), whereas only 9% of the sentences with three expressions are. The difference is mostly due to the proportion of sentences for which the CRF does not propose the right answer in the top 10 candidate sequences (*GoldNotIn10*), which is naturally higher in the more complex cases with multiple expressions in the sentence. Figure 3 also shows that the reranker is more useful with the sentences which contain one or two expressions (with 7.4% and 7.5% of changes, respectively), because these contain more mistakes to correct compared to sentences with no expressions, and contain more possibilities to correct the mistakes compared to sentences with three (or more) expressions (since the reranker cannot correct anything in the *GoldNotIn10* cases).

<footnote n="17">¹⁷This means that the dark grey bar represents the sum of the coloured bars, although the logarithmic scale makes this difficult to observe. Since the sequence selected by the CRF is the top one, position 1 is the only way for both the CRF and the reranker answers to be correct, thus it contains all the *RR* cases. Similarly, it cannot contain any *WR* or *RW* case, by definition.</footnote>
<figure><img src="image_39.png" /><figcaption>Figure 3: Proportion of error type by number of expressions in the gold sequence (all languages). Sentences with more than 3 expressions were discarded (30 cases, 0.1% of the data). Example: 16.5% of the sentences contain exactly one expression; among these, 18% belong to the *God-NotIn10* category, and 41%, 1%, 6%, 32% belong respectively to categories *RR*, *RW*, *WR*, *WW*.</figcaption></figure>

## 5.4 Reranker-specific evaluation

Based on these error types, a new reranker-centered evaluation method can be defined. Indeed, using this categorization, the reranker can be seen as a binary classifier: from this perspective, the job of the reranker is to detect the sentences for which the CRF answer was wrong, and leave the right ones as they are. Thus, for every sentence, either the answer is changed (positive instances) or not (negative instances). With this idea in mind, the four main categories can be translated to the standard true/false positive/negative categories in a straightforward way: if the reranker changes the answer correctly (WR), the instance is a true positive; if it changes the answer incorrectly (RW), the instance is a false positive, and similarly for the last two cases: RR and WW correspond respectively to true negative and false negative instances (the former was rightly not changed, and the latter should have been changed). Thanks to this interpretation, performance measures like precision, recall and F-score can be calculated for the semantic reranker, independently from the performance of the CRF component.<fnref n="18" /> An example of using such performance scores is given in Table 5.

<footnote n="18">¹⁸The GoldNotIn10 category is ignored when calculating these performance measures for the reranker, consistently with the idea of evaluating the reranker on its own: since these cases are impossible to solve, they should not be taken into account.</footnote>
<table><caption>Table 5: Reranker-specific performance by number of expressions in the gold sequence (all languages). P/R/F stands for precision/recall/f-score; the macro-average performance is the average over languages (datasets with NaN F-scores are ignored).</caption><thead><tr><th rowspan="2">Nb exprs</th><th colspan="3">Macro-average</th><th colspan="3">Micro-average</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>0</td><td>82.1</td><td>40.6</td><td>53.8</td><td>83.9</td><td>38.3</td><td>52.6</td></tr><tr><td>1</td><td>79.5</td><td>14.5</td><td>23.5</td><td>81.1</td><td>16.1</td><td>26.8</td></tr><tr><td>2</td><td>64.3</td><td>20.7</td><td>30.4</td><td>66.0</td><td>15.9</td><td>25.6</td></tr><tr><td>3</td><td>75.0</td><td>20.8</td><td>31.0</td><td>50.0</td><td>10.0</td><td>16.7</td></tr><tr><td>all</td><td>74.9</td><td>18.9</td><td>29.1</td><td>81.4</td><td>22.4</td><td>35.2</td></tr></tbody></table>

In the rest of this section we do not detail results by dataset, since the large number of languages and the dataset particularities would make it harder to recognize the general patterns related to the reranker. Nevertheless, it is important to keep in mind that such dataset-specific factors exist even though they are not shown. Additionally, the inequal size of the datasets clearly favors large datasets over small ones when grouping all the sentences together. This is why we present both the micro-average and macro-average performance whenever relevant, like in Table 5.

## 5.5 Feature analysis

As explained in §4, the reranker uses various kinds of features to determine the likelihood of the expressions in a sequence. Table 6 gives a brief overview of the impact of the different groups of features on performance, expressed as the F-score, computed from the micro-average precision and recall of the reranker alone (column 1, see §5.4), macro-average of the same over languages (column 2) and expression-level F-score (column 3, official evaluation on the full system).

First, it should be observed that the reranker relies heavily on the CRF confidence to make its decisions: without this feature, the performance drops to a ridiculously low level. Nevertheless, the reranker needs additional features in order to improve over the CRF alone (since otherwise the best it can do is to always agree with the CRF top prediction). A few simple features allow a large gain in performance (SF in Table 6: number of candidate expressions in the sequence, min./mean/max. number of words by expression and frequency in the
reference corpus). Adding more complex features based on frequency and semantic similarity of the candidate expression allows the reranker to make even better decisions: the micro F-score reaches 35.6 with the best combination, compared to 32.0 with only SF. Among these features, frequency and semantic similarity features seem to be equally useful, and combining both of them achieves the best performance; the only group of features which performs poorly (and is apparently even counter-productive) is the one where the candidate expression is compared against all other candidates in the sentence (group III in Table 6).

<table><caption>Table 6: Performance of the reranker using various subsets of features (percentages). Simple features (SF) represents the number of expressions and words as well as the frequency in the reference corpus; Groups I, II and III represent respectively single word, expression minus one word and alternative expressions features; Groups a and b represent respectively frequency features and semantic similarity features (see §4); NaN values correspond to cases where the precision and/or recall is zero.</caption><thead><tr><th>Features</th><th>micro F-score<br>reranker</th><th>macro F-score<br>reranker</th><th>macro F-score<br>MWE-level</th></tr></thead><tbody><tr><td>baseline: CRF answer</td><td>NaN</td><td>NaN</td><td>48.2</td></tr><tr><td>all but confidence</td><td>00.6</td><td>NaN</td><td>09.6</td></tr><tr><td>confidence + SF (*)</td><td>32.0</td><td>NaN</td><td>53.1</td></tr><tr><td>(*) + Ia, Ib</td><td>34.9</td><td>29.7</td><td>53.4</td></tr><tr><td>(*) + IIa, IIb</td><td>34.3</td><td>29.0</td><td>53.4</td></tr><tr><td>(*) + IIIa, IIIb</td><td>33.2</td><td>27.8</td><td>53.6</td></tr><tr><td>(*) + IIa, IIb, IIIa, IIIb</td><td>34.4</td><td>29.1</td><td>53.7</td></tr><tr><td>(*) + Ia, Ib, IIIa, IIIb</td><td>34.2</td><td>29.0</td><td>53.4</td></tr><tr><td>(*) + Ia, Ib, IIa, IIb</td><td><strong>35.6</strong></td><td><strong>30.2</strong></td><td>53.7</td></tr><tr><td>freq. only: (*) + Ia, IIa, IIIa</td><td>33.9</td><td>28.7</td><td>53.7</td></tr><tr><td>sem. sim. only: (*) + Ib, IIb, IIIb</td><td>34.0</td><td>28.4</td><td>53.7</td></tr><tr><td>all features, with mean only</td><td>34.8</td><td>29.3</td><td>53.6</td></tr><tr><td>all features, with min/mean/max</td><td>35.2</td><td>30.1</td><td>53.9</td></tr></tbody></table>

## 5.6 Analysis: impact of the coverage in the reference corpus

Some candidate expressions might not be found in the reference corpus, either because they are simply rare or because of tokenization/lemmatization issues (see §4.4). In fact, the coverage rate of the expressions in Europarl is quite low: for 36.2% of the sentences containing at least one expression, the expression(s)
they contain are not found at all in the reference corpus. Figure 4 shows that the error type depends greatly on whether the expression appears in the reference corpus or not. First, the CRF finds the right answer much more often when the expression is covered (RR + RW = 73%) than when it is not (RR + RW = 30%). This can be explained by the fact that the least frequent expressions are hard to identify by the CRF, and they also tend not to appear in the reference corpus. While this implies that the reranker has potentially more mistakes to fix in the zero-coverage cases, it actually changes fewer sentences (4.4% against 12.5% for covered expressions), resulting in a very low recall (3.5% against 37.7%); the precision is also lower, with 67% against 81%.

As explained in §5.5, the reranker can work with only a small set of “simple features”, which is why its performance in the zero-coverage case is lower but positive. Clearly, the more advanced features which rely on the reference corpus increase performance. This means that the coverage in the reference corpus is critical for the reranker to give its best results, but our current implementation of the system is probably not optimized from this point of view; in particular, the tokenization process might not be identical between the input data (where tokenization is provided) and the reference data (for which we apply a generic tokenizer), and the reference corpus is not lemmatized (see §4.4). This might explain why the recall is low with the current implementation. Ideally, a larger corpus would also help by covering a broader range of expressions; but there are very few such large datasets available for multiple languages.

<figure><img src="image_40.png" /><figcaption>Figure 4: Sentences error types by coverage/non-coverage of their expressions in the reference corpus (all languages). Example: 70% of the sentences containing expressions which appear in the reference corpus belong to the *RR* category, whereas only 16% of the sentences with expressions not covered in the reference corpus belong to this category.</figcaption></figure>
## 5.7 Analysis: continuous vs. discontinuous expressions

Verbal multiword expressions can be classified as either continuous or discontinuous: in the former case, the expression appears as a sequence of contiguous words, as in the following idiomatic expression:

(2) French (Indo-European; FR training data)

Celles-ci peuvent à tout moment jeter l'éponge. they.3.FEM.PL can at any time throw the sponge
'They.3.FEM.PL can give up at any time.'

In the latter case, the expression appears with words inserted between its lexicalized components, e.g.:

(3) French (Indo-European; FR training data)

J'ai obtenu de Jean-Marie Molitor [...] la permission de publier. I have obtained from Jean-Marie Molitor [...] the permission to publish
'Jean-Marie Molitor gave me the permission to publish.'

It is worth noticing the same lexicalized components might appear sometimes as a continuous expression and other times as a discontinuous expression.

<figure><img src="image_41.png" /><figcaption>Figure 5: Proportion of error type by number of continuous expressions in the sentence, for sentences containing 1, 2 or 3 expressions (all languages). Example: among sentences which contain two expressions, the proportion of *RR* cases is 7% (respectively 12%, 30%), when there are no (resp. 1, 2) continuous expressions among the two.</figcaption></figure>
## 6 Semantic reranking of CRF label sequences for VMWE identification

Figure 5 shows the impact of continuity in expressions: RR+RW cases increase with the number of continuous expressions for any number of expressions in the sentence, which means that the more there are continuous expressions, the better the performance of the CRF (there are also much less GoldNotIn10 cases). Interestingly, however, the semantic reranker follows an opposite trend: the less there are continuous expressions (i.e. the more there are discontinuous expressions), the better its performance: not only it fixes more mistakes from the CRF (better recall), it also fixes them better (better precision).<fnref n="19" /> The most likely explanation for these observations is that the CRF suffers from a “sequential bias” which makes it less good with discontinuous expressions, whereas such cases are not any harder for the semantic reranker, which is “sequence-agnostic”. In our opinion, this point clearly illustrates the complementarity of the two components.

## 5.8 Analysis: context vectors options

In §4.3, we presented several options which modify the way words which co-occur with the expression are taken into account in the MWE context vector. Table 7 shows the impact on performance of these options. Although there is no decisive pattern in these results, the absence of context-level normalization (CN)

<table><caption>Table 7: Performance of the reranker depending on context vector options. Left: Overall micro-average performance; right: F-score for continuous/discontinuous cases, for sentences with one expression exactly. CN, IWIE and MO represent the options presented in §4.3, respectively: *ContextNormalization*, *IncludeWordsInExpression*, *MultipleOccurrences*. P/R/F stands for precision/recall/f-score.</caption><thead><tr><th colspan="4">Options</th><th colspan="3">Micro-average</th></tr><tr><th>CN</th><th>IWIE</th><th>MO</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>82.9</td><td>21.9</td><td>34.6</td></tr><tr><td>0</td><td>0</td><td>1</td><td>81.7</td><td><strong>22.4</strong></td><td>35.2</td></tr><tr><td>0</td><td>1</td><td>0</td><td>82.5</td><td>22.1</td><td>34.8</td></tr><tr><td>0</td><td>1</td><td>1</td><td>81.6</td><td>21.7</td><td>34.3</td></tr><tr><td>1</td><td>0</td><td>0</td><td><strong>83.3</strong></td><td>21.8</td><td>34.5</td></tr><tr><td>1</td><td>0</td><td>1</td><td>82.8</td><td>21.7</td><td>34.4</td></tr><tr><td>1</td><td>1</td><td>0</td><td>80.9</td><td>21.8</td><td>34.4</td></tr><tr><td>1</td><td>1</td><td>1</td><td>81.4</td><td><strong>22.4</strong></td><td>35.2</td></tr></tbody></table>

<table><thead><tr><th colspan="4">Options</th><th>F-score</th><th>F-score</th></tr><tr><th>CN</th><th>IWIE</th><th>MO</th><th>Continuous</th><th>Discontinuous</th></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>14.7</td><td><strong>41.4</strong></td></tr><tr><td>0</td><td>0</td><td>1</td><td><strong>16.8</strong></td><td>40.3</td></tr><tr><td>0</td><td>1</td><td>0</td><td>14.7</td><td>40.9</td></tr><tr><td>0</td><td>1</td><td>1</td><td>15.9</td><td>39.6</td></tr><tr><td>1</td><td>0</td><td>0</td><td>15.2</td><td>39.8</td></tr><tr><td>1</td><td>0</td><td>1</td><td>15.2</td><td>39.6</td></tr><tr><td>1</td><td>1</td><td>0</td><td>14.0</td><td>41.0</td></tr><tr><td>1</td><td>1</td><td>1</td><td>14.8</td><td>40.8</td></tr></tbody></table>

<footnote n="19">¹⁹Except in the case of three expressions with zero or one continuous; this is probably due to the low number of cases.</footnote>
as well as allowing multiple occurrences of the same word to be counted multiple
times (MO) obtain slightly higher performance in general. Looking at the effect
of these options on continuous/discontinuous expressions, including expressions
words in the context window (IWIE) has a negative effect on all the cases, except
if CN is selected but only in the discontinuous case. In fact, an interesting pattern
can be observed in the continuous/discontinuous table: the combinations of op-
tions which make the F-score increase for continuous expressions tend to make
the F-score in the discontinuous case decrease, and conversely. This is confirmed
by a moderate negative Pearson's correlation coefficient of -0.56 and a high neg-
ative Spearman's rank correlation coefficient of -0.79. Here again the differences
in performance are too moderate to conclude decisively; however this point sug-
gests that there is a trade-off between the continuous and discontinuous cases,
and this trade-off might be controlled through these options to some extent. This
means that the system could potentially be tuned to favor one or the other case.

6 Conclusion and future work

In this chapter we described a two stages approach for identifying VMWEs, based
on sequence labeling with CRF followed by reranking of the CRF candidates. We
showed experimentally that the reranker significantly improves the performance
of the system, with in average a 12% F1-score improvement over using the CRF
component alone. Then we proceeded to analyze how the reranker works.

We found that the reranker follows the CRF quite closely, rarely selecting a
candidate with a low CRF confidence, and selecting the CRF top prediction in
92.5% of the cases. Consistently with this observation, when the reranker di-
verges from the top CRF prediction, it does so correctly with high confidence
(81.5% of correct answers among the changed predictions).

The contribution of the reranker is more important with the sentences which
contain one or two expressions: sentences with no expressions are almost always
correctly detected by the CRF alone, whereas the cases with 3 or more expres-
sions are so complex that the CRF does not usually provide the right candidate
among its top 10 predictions, leaving the reranker unable to fix these errors. The
coverage of the MWE in the reference corpus is another major factor of perfor-
mance for the reranker, with the recall dropping 10 times for expressions which
do not appear in the reference corpus. Finally the last important finding of our
study is that the reranker seems to compensate the CRF sequential bias: while
the latter performs better with continuous MWEs, the reranker performs com-
paratively better with discontinuous cases.
The semantic reranker presented in this chapter is a proof-of-concept version, and new perspectives emerge from the fact that the combination of a CRF component with this reranker proves fruitful for detecting MWEs. There are a few obvious areas in which the reranker could be improved, especially in the tokenization/lemmatization part, and it is likely that choosing a more adequate reference corpus would help. But there are also deeper questions which are worth studying:

* Computing a context vector for a MWE is not as trivial as for a single word (especially if the words in the expression are not continuous), and the authors are not aware of any standard approach for this. While several options were tested, this question deserves to be studied on its own.

* In the same line of thought, the current state of the art in distributional semantics is based on word embeddings (Legrand & Collobert 2016). Here again, the authors are not aware of any software able to retrieve word embeddings for multiple (possibly discontinuous) words.

* Would it be possible for the reranker to work at the expression level instead of the sentence level? Indeed, the current method used to “merge” multiple expressions in a sentence is likely to lose some information in the process. One could also ask whether some of the information currently computed by the reranker could be fed directly into the CRF. An iterative process could even be considered, perhaps allowing to refine the quality of the predicted expressions over iterations.

## Acknowledgements

We are grateful to the editors and anonymous reviewers for their valuable comments. Lifeng Han also thanks Qun Liu for his continued support.

The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.

The graphics in this chapter were created with R (R Core Team 2012), using the ggplot2 library (Wickham 2009).
## Abbreviations

<table><tr><td>CRF</td><td>conditional random fields</td><td>RW</td><td>right-wrong</td></tr><tr><td>IDF</td><td>inverse document frequency</td><td>VMWE</td><td>verbal multiword expression</td></tr><tr><td>MWE</td><td>multiword expression</td><td>WR</td><td>wrong-right</td></tr><tr><td>NLP</td><td>Natural Language Processing</td><td>WW</td><td>wrong-wrong</td></tr><tr><td>RR</td><td>right-right</td><td></td><td></td></tr></table>

## References

Attia, Mohammed, Lamia Tounsi, Pavel Pecina, Josef van Genabith & Antonio Toral. 2010. Automatic extraction of Arabic multiword expressions. In Proceedings of the COLING Workshop on Multiword Expressions: From Theory to Applications (MWE '10), 19–27. Association for Computational Linguistics.

Bar, Kfir, Mona Diab & Abdelati Hawwari. 2014. Arabic multiword expressions. In Nachum Dershowitz & Ephraim Nissan (eds.), Language, culture, computation. Computational Linguistics and Linguistics: Essays dedicated to Yaacov Choueka on the occasion of his 75th birthday, Part III, 64–81. Berlin: Springer. DOI:10.1007/978-3-642-45327-4_5

Blunsom, Phil & Timothy Baldwin. 2006. Multilingual deep lexical acquisition for HPSGs via supertagging. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, 164–171. Sydney, Australia: Association for Computational Linguistics. DOI:10.3115/1610075.1610101

Boukobza, Ram & Ari Rappoport. 2009. Multi-word expression identification using sentence surface features. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 468–477. August 6-7, 2009.

Constant, Matthieu, Anthony Sigogne & Patrick Watrin. 2012. Discriminative strategies to integrate multiword expression recognition and parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, 204–212. Association for Computational Linguistics.

Green, Spence, Marie-Catherine de Marneffe, John Bauer & Christopher D. Manning. 2011. Multiword expression identification with tree substitution grammars: A parsing tour de force with French. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 725–735. Association for Computational Linguistics. http://www.aclweb.org/anthology/D11-1067.

Green, Spence, Marie-Catherine de Marneffe & Christopher D. Manning. 2013. Parsing models for identifying multiword expressions. Computational Linguistics 39(1). 195–227. DOI:10.1162/COLI_a_00139
Hall, Mark, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann & Ian H. Witten. 2009. The WEKA data mining software: An update. *ACM SIGKDD Explorations Newsletter* 11(1). 10–18.

Han, Aaron Li-Feng, Derek F. Wong & Lidia S. Chao. 2013. Chinese named entity recognition with conditional random fields in the light of Chinese characteristics. In Mieczysław A. Kłopotek, Jacek Koronacki, Małgorzata Marciniak, Agnieszka Mykowiecka & Sławomir T. Wierzchoń (eds.), *Language processing and intelligent information systems. iis 2013*, vol. 7912 (Language Processing and Intelligent Information Systems), 57–68. Berlin Heidelberg: Springer.

Han, Aaron Li-Feng, Xiaodong Zeng, Derek F. Wong & Lidia S. Chao. 2015. Chinese named entity recognition with Graph-based semi-supervised learning model. In *Proceedings of the 8th SIGHAN Workshop on Chinese Language Processing (SIGHAN-8)*, 15–20. Association for Computational Linguistics & Asian Federation of Natural Language Processing. July 30-31, 2015.

Hosseini, Mohammad Javad, Noah A. Smith & Su-In Lee. 2016. UW-CSE at SemEval-2016 task 10: Detecting multiword expressions and supersenses using double-chained conditional random fields. In *Proceedings of the 10th International Workshop on Semantic Evaluation*, 931–936. http://aclweb.org/anthology/S/S16/S16-1143.pdf. June 16-17, 2016.

Koehn, Philipp. 2005. Europarl: A parallel corpus for statistical machine translation. In *Proceedings of the 10th Machine Translation Summit*, 79–86. Phuket, Thailand.

Lafferty, John D., Andrew McCallum & Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In *Proceedings of the Eighteenth International Conference on Machine Learning* (ICML ’01), 282–289. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. http://dl.acm.org/citation.cfm?id=645530.655813.

Legrand, Joël & Ronan Collobert. 2016. Phrase representations for multiword expressions. In *Proceedings of the 12th Workshop on Multiword Expressions* (MWE ’16), 67–71. Association for Computational Linguistics. http://anthology.aclweb.org/W16-1810.

Maldonado, Alfredo & Martin Emms. 2011. Measuring the compositionality of collocations via word co-occurrence vectors: Shared task system description. In *Proceedings of the Workshop on Distributional Semantics and Compositionality*, 48–53. http://dl.acm.org/citation.cfm?id=2043121.2043130.

Maldonado, Alfredo, Lifeng Han, Erwan Moreau, Ashjan Alsulaimani, Koel Dutta Chowdhury, Carl Vogel & Qun Liu. 2017. Detection of verbal multi-word expressions via conditional random fields with syntactic dependency features
and semantic re-ranking. In Proceedings of the 13th Workshop on Multiword Expressions (MWE '17), 114–120. Association for Computational Linguistics.
DOI:10.18653/v1/W17-1715

Maldonado, Alfredo & Behrang QasemiZadeh. 2018. Analysis and Insights from the PARSEME Shared Task dataset. In Stella Markantonatou, Carlos Ramisch, Agata Savary & Veronika Vincze (eds.), *Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop*, 149–175. Berlin: Language Science Press. DOI:10.5281/zenodo.1469557

Quinlan, J. Ross. 1992. Learning with continuous classes. In *Proceedings of the 5th Australian Joint Conference on Artificial Intelligence*, 343–348.

R Core Team. 2012. *R: A language and environment for statistical computing*. R Foundation for Statistical Computing. Vienna, Austria. https://www.R-project.org/. ISBN 3-900051-07-0.

Sag, Ivan A., Timothy Baldwin, Francis Bond, Ann A. Copestake & Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In *Proceedings of the 3rd International Conference on Computational Linguistics and Intelligent Text Processing*, vol. 2276/2010 (CICLing'02), 1–15. Springer-Verlag.

Salehi, Bahar, Paul Cook & Timothy Baldwin. 2014a. Detecting non-compositional MWE components using Wiktionary. In *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014)*, 1792–1797. Doha, Qatar. http://aclweb.org/anthology/D/D14/D14-1189.pdf.

Salehi, Bahar, Paul Cook & Timothy Baldwin. 2014b. Using distributional similarity of multi-way translations to predict multiword expression compositionality. In *Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2014)*, 472–481. Gothenburg. http://aclweb.org/anthology/E/E14/E14-1050.pdf.

Salehi, Bahar, Paul Cook & Timothy Baldwin. 2015. A word embedding approach to predicting the compositionality of multiword expressions. In *Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics – Human Language Technologies (NAACL '15)*, 977–983. http://dblp.uni-trier.de/db/conf/naacl/naacl2015.html#SalehiCB15.

Savary, Agata, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova & Antoine Doucet. 2017. The PARSEME Shared Task on automatic identification of verbal multiword expressions. In *Proceedings of the 13th Workshop on Multiword Expressions (MWE '17)*, 31–47. Association for Computational Linguistics. DOI:10.18653/v1/W17-1704
Schneider, Nathan, Emily Danchik, Chris Dyer & Noah A. Smith. 2014. Discriminative lexical semantic segmentation with gaps: Running the MWE gamut. *Transactions of the Association for Computational Linguistics* 2(1). 193–206. https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/281.

Schütze, Hinrich. 1998. Automatic word sense discrimination. *Computational Linguistics* 24(1). 97–123. http://dl.acm.org/citation.cfm?id=972719.972724.

Sinclair, John. 1991. *Corpus, concordance, collocation*. Oxford: Oxford University Press.

Sun, Xiao, Chengcheng Li, Chenyi Tang & Fuji Ren. 2013. Mining semantic orientation of multiword expression from Chinese microblogging with Discriminative Latent Model. In *Proceedings of the 2013 International Conference on Asian Language Processing*, 117–120. http://dblp.uni-trier.de/db/conf/ialp/ialp2013.html#SunLTR13.

Tsvetkov, Yulia & Shuly Wintner. 2010. Extraction of multi-word expressions from small parallel corpora. In *Proceedings of the 23rd International Conference on Computational Linguistics* (COLING-10), 1256–1264. http://dl.acm.org/citation.cfm?id=1944566.1944710.

Venkatapathy, Sriram & Aravind K. Joshi. 2006. Using information about multi-word expressions for the word-alignment task. In *Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Properties* (MWE ’06), 20–27. http://dl.acm.org/citation.cfm?id=1613692.1613697.

Vincze, Veronika, István Nagy T. & Gábor Berend. 2011. Multiword expressions and named entities in the Wiki50 corpus. In *Proceedings of the International Conference Recent Advances in Natural Language Processing 2011*, 289–295. RANLP 2011 Organising Committee. http://aclweb.org/anthology/R11-1040.

Wickham, Hadley. 2009. *Ggplot2: Elegant graphics for data analysis*. New York: Springer-Verlag.
<figure><img src="image_42.png" /></figure>
# Chapter 7

## A transition-based verbal multiword expression analyzer

Hazem Al Saied

ATILF UMR 7118, Université de Lorraine/CNRS

Marie Candito

LLF UMR 7110, Université Paris Diderot/CNRS

Matthieu Constant

ATILF UMR 7118, Université de Lorraine/CNRS

We describe a robust transition-based analyzer for identifying and categorizing Verbal Multiword Expressions (VMWEs). The system was developed and evaluated using the datasets of the PARSEME shared task on VMWE identification (Savary et al. 2017).

We accommodate the variety of linguistic resources provided for each language, in terms of accompanying morphological and syntactic information. Our system produces very competitive scores, for both VMWE identification and categorization, with respect to the shared task results.

## 1 Introduction

We present a generic system for the identification and categorization of verbal multiword expressions (hereafter VMWEs). With respect to grammatical or nominal multiword expressions, VMWEs tend to exhibit more morphological and syntactic variation than other MWEs, if only because in general the verb is inflected and can receive adverbial modifiers. Furthermore, some VMWE types, in particular light-verb constructions, allow for the full range of syntactic variation

<figure><img src="image_43.png" /></figure>
(extraction, coordination etc...). This renders the VMWE identification task even more challenging than general MWE identification, in which fully frozen and continuous expressions contribute to an increase in the overall performance.

Our objective was to design a data-driven system applicable to several languages, with limited language-specific tuning. We took advantage of the datasets provided within the shared task on automatic identification of VMWEs (Savary et al. 2017) to train and test our system. These datasets concern 18 languages, and consist of tokenized sentences in which VMWEs are annotated. One VMWE instance is a set made either of several tokens, potentially non-continuous, or of one single token (i.e. a multiword token, hereafter MWT, such as amalgamated verb-particle in German).<fnref n="1" /> A VMWE may be embedded in another longer one, and two VMWEs can overlap. Each annotated VMWE is tagged with a category among Light-Verb Constructions (LVC), IDioms (ID), Inherently REFLexive Verbs (IReflV), Verb-Particle Constructions (VPC) and OTHer verbal MWEs (OTH). The datasets are quite heterogeneous, both in terms of size and of accompanying resources: 4 languages have none (Bulgarian, Spanish, Hebrew, Lithuanian), for 4 languages, morphological information such as lemmas and POS is provided (Czech, Maltese, Romanian, Slovene), and for the 10 remaining languages, full dependency parses are provided.

The system we describe in the current paper is an extension of the ATILF-LLF system (Al Saied et al. 2017), a one-pass greedy transition-based system which participated in the shared task, obtaining very competitive results (hereafter ATILF-LLF 1). The new system (ATILF-LLF 2) categorizes the VMWEs on top of identifying them, and has an extended expressive power, handling some cases of VMWE embedded in another one. Both for ATILF-LLF 1 and 2, we tuned a set of feature template for each language, relying exclusively on training data, accompanying CoNLL-U files when available, and basic feature engineering.

The remainder of the article is organized as follows: we describe our system in Section 2, and comment its expressive power as opposed to ATILF-LLF 1. We then describe the experimental setup in Section 3, and comment the results in Section 4. Section 5 is devoted to related work. We conclude in Section 6 and give perspectives for future work.

<footnote n="1">¹The majority of annotated VMWEs are multi-token. The prevalence of MWTs varies greatly among languages. While absent from seven languages and very rare for nine other languages, they are very frequent in German and Hungarian.</footnote>
## 2 System description

The analyzers we developed (ATILF-LLF 1 and 2) are simplified versions of the system proposed by Constant & Nivre (2016). Building on the classic arc-standard dependency parser (Nivre 2004), Constant & Nivre (2016) designed a parsing algorithm that jointly predicts a syntactic dependency tree and a forest of multiword lexical units. Their system uses both a syntactic and a lexical stack and specific transitions merge tokens to create MWEs, as proposed by Nivre (2014). We have simplified this formal apparatus keeping only the lexical stack, and predicting MWEs only.

A transition-based system applies a sequence of actions (usually called *transitions*) to incrementally build the expected output structure in a bottom-up manner. Each transition is usually predicted by a classifier given the current state of the system (namely a *configuration*).

A configuration in our system consists of a triplet $c = (\sigma, \beta, L)$, where $\sigma$ is a stack containing units under processing, $\beta$ is a buffer containing the remaining input tokens, and $L$ is a set of output MWEs.

The *initial configuration* for a sentence $x = x_1, ..., x_n$, i.e. a sequence of $n$ tokens, is represented by $c_s$ as: $c_s(x) = ([])$, $[x_1, ..., x_n]$, $\emptyset$) and the set of *terminal configurations* $C_t$ contains any configuration of the form $c_t = ([])$, $[]$, $L)$. At the end of the analysis, the identified VMWEs are simply extracted from $L$.

Single tokens are brought to the stack by the *SHIFT* transition, and are potentially marked as (mono-token) VMWE using the *MARK AS C* transition, whereas trees are formed using merge transitions (cf. §2.1).

The output VMWEs are the units added to $L$, either by the *MARK AS C* or *MERGE AS C* transitions. When one VMWE is embedded in another one, both VMWEs appear separately in $L$ (which is thus redundant).<fnref n="2" />

### 2.1 Transition set

Our system uses the following transitions:

1. the **SHIFT** transition moves the first element of the buffer to the stack
   **Precondition:** the buffer is not empty.

2. the **REDUCE** removes the top element of the stack
   **Precondition:** the stack is not empty.

<footnote n="2">²For instance, if we represent the binary tree in bracketing format and the categorization with a subscript, $((a, b)_{\text{IRefIV}}, c)_{\text{ID}}$ represents an IRefV $a+b$ embedded within an ID $a+b+c$. Both $((a, b)_{\text{IRefIV}}, c)_{\text{ID}}$ and $(a, b)_{\text{IRefIV}}$ will appear in $L$.</footnote>
3. the **WHITE MERGE** transition combines the two top elements of the stack into a single element;

**Precondition:** the stack contains at least two elements.

4. five **MERGE AS C** transitions (where C stands for a VMWE category) perform a white merge, mark the resulting unit as a VMWE of category C, and add it to L.

**Precondition:** the stack contains at least two elements.

5. In order to cope with MWTs, we added five **MARK AS C** transitions, which mark the top stack element as a VMWE, assign to it the category C, and add it to L.

**Precondition:** The stack is not empty and its head is a non-marked single token.<fnref n="3" />

Figure 1 shows the analysis of a German sentence containing a multiword token VPC embedded within an IReflV.

In the input, each token is associated with linguistic attributes (form, and depending on the data sets, lemma and POS). When a merge transition is applied, the newly created element gets its attributes using basic concatenation over forms, and over lemmas and POS tags when available.<fnref n="4" />

## 2.2 Parsing algorithm and training oracle

In all the following, at parsing time we use a greedy algorithm, starting with an initial configuration $c_s$, and applying in sequence the best-scoring legal transition until a terminal configuration is reached.<fnref n="5" />

The training phase learns the transition-scoring classifier. This is done through supervised learning, by converting the training sentences into sequences of [con-

<footnote n="3">³Because mono- and multi-tokens have very different linguistic properties, we preferred to distinguish transitions coping with both kinds of VMWEs. Without this restriction, as noted by a reviewer, **MERGE AS C** would be equivalent to **WHITE MERGE + MARK AS C**. The effectiveness of this alternative solution remains to be tested.</footnote>

<footnote n="4">⁴This would deserve to be improved in future experiments, with finer procedures to predict the lemmatized form and more importantly to predict the POS tag of the merged node, although in the special case of VMWE prediction the POS is verbal.</footnote>

<footnote n="5">⁵Sentence analysis is composed of exactly $2n + r$ transitions, with $n$ being the number of tokens and $r$ the number of MWTs. Every single token not entering a VMWE requires a **SHIFT** and a **REDUCE**, every multi-token VMWE of length $m$ requires $2m$ transitions (**m** **SHIFTS**, $m-2$ **WHITE MERGES**, one **MERGE AS C** and one **REDUCE**), while every MWT requires three transitions: a **SHIFT**, a **MARK AS C** and a **REDUCE**.</footnote>
<table><caption>Figure 1: Transition sequence for tagging the German sentence *Damit müsste man sich nun herumschlagen* 'With-that must-SUBJUNCTIVE one REFLEXIVE now around-struggle' ⇒ 'One would have to struggle with that', containing two VMWEs: *SICH HERUMSCHLAGEN* tagged as inherently reflexive verb (IRefV), in which *HERUMSCHLAGEN* is itself a multiword token tagged as verb-particle combination (VPC).</caption><thead><tr><th>Transition</th><th>Configuration</th></tr></thead><tbody><tr><td>Shift</td><td>[ ], [Damit, müsste, man, sich, nun, herumschlagen], [ ]</td></tr><tr><td>Reduce</td><td>[ ], [müsste, man, sich, nun, herumschlagen], [ ]</td></tr><tr><td>Shift</td><td>[müsste], [man, sich, nun, herumschlagen], [ ]</td></tr><tr><td>Reduce</td><td>[ ], [man, sich, nun, herumschlagen], [ ]</td></tr><tr><td>Shift</td><td>[man], [sich, nun, herumschlagen], [ ]</td></tr><tr><td>Reduce</td><td>[ ], [sich, nun, herumschlagen], [ ]</td></tr><tr><td>Shift</td><td>[sich], [nun, herumschlagen], [ ]</td></tr><tr><td>Shift</td><td>[sich, nun], [herumschlagen], [ ]</td></tr><tr><td>Reduce</td><td>[sich], [herumschlagen], [ ]</td></tr><tr><td>Shift</td><td>[sich, herumschlagen], [ ], [ ]</td></tr><tr><td>Mark as VPC</td><td>[sich, herumschlagen<sub>VPC</sub>], [ ], [herumschlagen<sub>VPC</sub>]</td></tr><tr><td>Merge as IrefV</td><td>[(sich, herumschlagen<sub>VPC</sub>)<sub>IRefIV</sub>], [ ], [herumschlagen<sub>VPC</sub>, (sich, herumschlagen<sub>VPC</sub>)<sub>IRefIV</sub>]</td></tr><tr><td>Reduce</td><td>[ ], [ ], [herumschlagen<sub>VPC</sub>, (sich, herumschlagen<sub>VPC</sub>)<sub>IRefIV</sub>]</td></tr></tbody></table>

figuration, gold transition to apply] pairs, by using a static oracle.<fnref n="6" /> Our static oracle returns for a given configuration the first applicable transition using the following priority order: MARK AS C, MERGE AS C, WHITE MERGE, REDUCE and SHIFT. Applicability here means not only the standard preconditions for transitions, but also that the output configuration is compatible with the gold annotations. We added the constraint that the WHITE MERGE is only applicable to the right suffix of a gold VMWE. For instance, suppose we have a gold continuous VMWE *kick the bucket*, when the first two elements are on top of the stack, the WHITE MERGE is not applicable yet, it will be applied when the right suffix *the* and *bucket* is on top of the stack.

To produce our training examples, we start by generating the initial configuration for each sentence, and apply in sequence the transition predicted by the static oracle, until a terminal configuration is reached. The analysis in Figure 1 corresponds to the oracle transition sequence for the example sentence.

<footnote n="6">⁶Using (Goldberg & Nivre 2013)'s terminology, a static oracle is both incomplete (defined for configurations obtained from previous oracle transitions only) and deterministic (at each such configuration, there is a single oracle transition to apply).</footnote>
## 2.3 Expressive power

As far as expressive power is concerned, ATILF-LLF 2 is slightly more powerful than ATILF-LLF 1. ATILF-LLF 2 now performs VMWE categorization and not just identification. Both systems cannot analyze interleaving MWEs, but while ATILF-LLF 1 could cope with no overlapping at all, ATILF-LLF 2 can cope with some cases of embeddings, i.e. some cases of VMWE fully contained in another one.<fnref n="7" />

In effect, ATILF-LLF 1 contained a SHIFT transition, a WHITE MERGE, a MERGE AS C+REDUCE for identifying multi-token VMWEs, and a MARK AS C+REDUCE for MWTs.<fnref n="8" /> Because the MERGE AS C+REDUCE transition identifies a MWE and removes it from the stack, no cases of embeddings were covered.

In ATILF-LLF 2, some cases of embeddings are now covered (e.g. the example in Figure 1). More precisely, the covered cases are those where one can form a projective tree by attaching to a fictitious root all the binary trees representing the VMWEs of a sentence, ignoring tokens not belonging to any VMWE. An alternative formulation is that given any VMWE composed of the $t_1 t_2 \dots t_m$ tokens and any gap $g_1 g_2 \dots g_n$ appearing between a pair of components $t_i t_{i+1}$, the condition is that the $g_i$ tokens cannot belong to a MWE having components outside the set $g_1 \dots g_n$. So a non-covered case is found for instance in $LET_{1,2} THE_1 CAT_1 OUT_{1,2} OF_1 THE_1 BAG_1$: the VMWE $LET OUT$ has a gap containing tokens $THE_1 CAT_1$, which belong to a VMWE with tokens outside the gap.

## 3 Experimental setup

For a given language, and a given train/dev split, we use the oracle-based resulting transition sequences to train a multi-class SVM classifier.<fnref n="9" /> We describe in the next subsections the feature templates we used (§3.1) and how we tuned them (§3.2).

<footnote n="7">⁷It is worth noting that embedded VMWEs are very rare in the datasets: there are twenty to thirty embedded VMWEs in German, Hebrew and Hungarian and about 150 in Czech.</footnote>

<footnote n="8">⁸ATILF-LLF 1 used hard-coded procedures for matching MWTs (if seen in the training set), which we replaced by features used by the classifier.</footnote>

<footnote n="9">⁹The whole system was developed using Python 2.7, with 4,739 lines of code, using the Scikit-learn 0.19. We used the Error-correcting output codes framework for the multi-class SVM classifier. The code is available on Github: https://goo.gl/1j8mVu under the MIT license.</footnote>
<table><caption>Table 1: System setting code descriptions.</caption><thead><tr><td>Code</td><td>Setting description</td></tr></thead><tbody><tr><td>A</td><td>use of POS and lemmas</td></tr><tr><td>A'</td><td>use of suffixes</td></tr><tr><td>B</td><td>use of syntactic dependencies</td></tr><tr><td>C</td><td>use of bigrams S<sub>1</sub>S<sub>0</sub>, S<sub>0</sub>B<sub>0</sub>, S<sub>1</sub>B<sub>0</sub> and S<sub>0</sub>B<sub>1</sub></td></tr><tr><td>D</td><td>use of the trigram S<sub>1</sub>S<sub>0</sub>B<sub>0</sub></td></tr><tr><td>E</td><td>use of the S<sub>0</sub>B<sub>2</sub> bigram</td></tr><tr><td>F</td><td>use of transition history (length 1)</td></tr><tr><td>G</td><td>use of transition history (length 2)</td></tr><tr><td>H</td><td>use of transition history (length 3)</td></tr><tr><td>I</td><td>use of distance between S<sub>0</sub> and S<sub>1</sub></td></tr><tr><td>J</td><td>use of distance between S<sub>0</sub> and B<sub>0</sub></td></tr><tr><td>K</td><td>use of B<sub>1</sub></td></tr><tr><td>L</td><td>use of training corpus VMWE lexicon</td></tr><tr><td>M</td><td>use of stack length</td></tr><tr><td>N</td><td>use of MWT dictionary</td></tr></tbody></table>

## 3.1 Feature templates

A key point in a classical transition-based system is feature engineering, known to have great impact on performance. We have gathered feature templates into groups, for which we provide short descriptions in Table 1, along with code letters that we use in §4 to describe which feature groups were used for each language in the final shared task results. We describe in this section each feature group. We hereafter use symbol $B_i$ to indicate the ith element in the buffer. $S_0$ and $S_1$ stand for the top and the second elements of the stack. For every unit X in the stack or buffer, we denote $X_w$ its word form, $X_l$ its lemma and $X_p$ its POS tag. The concatenation of two elements X and Y is noted XY.

### Basic linguistic features

For each language, we used a precise set of stack or buffer elements, hereafter the *focused elements*, to derive unigram, bigram and trigram features. By default, the focused elements are $S_0$, $S_1$ and $B_0$. For some languages, $B_1$ was also used (code K in Table 1). If bigrams are on (code C in Table 1) features are generated for the element pairs $S_1S_0$, $S_0B_0$, $S_1B_0$, plus $S_0B_1$ if K is on, and plus $S_0B_2$ for a few
languages (code E). For trigrams, we only used the features of the $S_1S_0B_0$ triple (code D).

For any resulting unigram, bigram or trigram, we use by default the word form (e.g. $S_0w$). For languages whose datasets comprise morphological information, we further use the lemmas and POS tags (code A in Table 1), i.e. $X_l$ and $X_p$. The precise features for a bigram XY are $X_wY_w$, $X_pY_p$, $X_lY_l$, $X_pY_l$ and $X_lY_p$. Those for a trigram XYZ are $X_wY_wZ_w$, $X_lY_lZ_l$, $X_pY_pZ_p$, $X_lY_pZ_p$, $X_pY_lZ_p$, $X_pY_pZ_l$, $X_lY_lZ_p$, $X_lY_pZ_l$, $X_pY_lZ_l$.

For the languages lacking companion morphological information, we tried to mimic that information using suffixes (code A' in Table 1), more precisely the last two and last three letters, which we used for unigram elements only.

## Syntax-based features

After integrating classical linguistic attributes, we investigated using more lin-
guistically sophisticated features. First of all, syntactic structure is known to help
MWE identification (Fazly et al. 2009; Seretan 2011; Nagy T. & Vincze 2014). So
for datasets comprising syntactic information, we introduced features capturing
the existence of syntactic dependencies between elements of the buffer and of
the stack (code B in Table 1). More precisely, provided that $S_0$ is a single token,
we generate (i) the features RIGHTDEP ($S_0$, $B_i$) = TRUE and RIGHTDEPLAB ($S_0$, $B_i$) = L for each buffer token $B_i$ that is a syntactic dependent of $S_0$ with label $l$,
and (ii) the features LEFTDEP ($S_0$, $B_i$) = TRUE and LEFTDEPLAB ($S_0$, $B_i$) = L when
a buffer element $B_i$ is $S_0$'s syntactic governor.

Other syntax-based features aim at modeling the direction and label of a syn-
tactic relation between the top two tokens of the stack (feature SYNTACTICRELATION ($S_0, S_1$) = $\pm L$ is used for $S_0$ governing/governed by $S_1$, provided $S_0$ and $S_1$ are single tokens).<fnref n="10" /> All these syntactic features try to capture syntactic regular-
ities between the tokens composing a VMWE.

## Miscellaneous features

We found that other traditional features, used in transition-based systems, were
sometimes useful like (local) transition history of the system. We thus added
**History-based features** to represent the sequence of previous transitions (of
length one, two or three, cf. codes F, G and H in Table 1).

<footnote n="10">¹⁰For ATILF-LLF 1, we used gold syntactic features for the languages accompanied with gold dependency companion files, as authorized in the closed track. Performance when using predicted syntax will be evaluated in future work.</footnote>
We also added **Distance-based features**, known to help transition-based dependency parsing (Zhang & Nivre 2011), more precisely the distance between S₀ and S₁ and between S₀ and B₀ (respectively codes I and J in Table 1). We also extracted **Stack-length-based features** (code M in Table 1).

The VMWE identification task is highly lexical so we found it useful to use **dictionary-based features**, which use “dictionaries” extracted from the training set, both for multi-token VMWEs and MWTs. The dictionaries are lemma-based when lemmas are available, and form-based otherwise. These dictionary-based features include (i) a boolean feature set to true when S₀ belongs to the MWT dictionary (code N in Table 1), and (ii) boolean features firing when S₀, S₁, B₀, B₁ or B₂ belong to an entry of the extracted VMWE dictionary (code L in Table 1).

## 3.2 Feature tuning

We first divided the data sets into three groups, based on the availability of CoNLL-U files: (a) for **Bulgarian**, **Hebrew** and **Lithuanian** neither morphological nor syntactic information is available on top of tokens and VMWE annotation; (b) **Czech**, **Spanish**, **Farsi**, **Maltese** and **Romanian** are accompanied by CoNLL-U files with morphological information only, and (c) the **other languages**<fnref n="11" /> are accompanied by a fully annotated CoNLL-U file.

In the first tuning phase, we used one “pilot” language for each group (Bulgarian, Czech and French). Then, German was added as pilot language to investigate features for languages with high percentage of MWTs and embedded VMWEs. We tuned the features using both development sets extracted from the provided training sets, and using cross-validation.

Finally, we used the discovered feature groups as a source of inspiration for producing specialized feature groups for all other languages. Note that given the combinatorial explosion of feature combinations, we could not apply a full grid-search for the pilot languages, and a fortiori for all languages.

## 4 Results

We provide the identification results in Table 2, in which the performance of ATILF-LLF 2, both in the shared task test sets and in cross-validation, can be compared with (i) a baseline system, (ii) the best performing system of the shared

<footnote n="11">¹¹These languages are German, Greek, French, Hungarian, Italian, Polish, Portuguese, Slovene, Swedish, and Turkish.</footnote>
<table><caption>Table 2: VMWE identification: The first column provides the language, it is shown whether the companion file contains morpho and syntax, morpho only (') or nothing (''). The last column lists the feature groups used for that language (using the codes of Table 1). Columns 2, 3, 4: VMWE-based F-scores on test sets, for the baseline system, ATILF-LLF 2, and the best performing Shared task systems (Best of ST). Columns 5, 6, 7: same as 2, 3, 4 but token-based. Columns 8, 9, 10: VMWE-based results in 5-fold cross-validation over training sets, for the baseline system, ATILF-LLF 1, and ATILF-LLF 2. The last row (Avg) provides the average results weighted by the size of the test sets (or train sets for cross-validation results). The stars in columns Best of ST are those for which ATILF-LLF 1 did not rank first.</caption><thead><tr><th rowspan="3">Language</th><th colspan="6">Test dataset</th><th colspan="2">Cross validation</th><th rowspan="3">Feature Settings</th></tr><tr><th colspan="3">VMWE-based F<sub>1</sub></th><th colspan="3">Token-based F<sub>1</sub></th><th colspan="2">VMWE-based F<sub>1</sub></th></tr><tr><th>Baseline</th><th>ATILF-LLF 2</th><th>Best of ST</th><th>Baseline</th><th>ATILF-LLF 2</th><th>Best of ST</th><th>Baseline</th><th>ATILF-LLF 1</th><th>ATILF-LLF 2</th></tr></thead><tbody><tr><td>BG**</td><td>47.6</td><td>55.8</td><td><strong>61.3</strong></td><td>50.7</td><td>60.2</td><td><strong>66.2</strong></td><td>48.3</td><td><strong>57.1</strong></td><td>53.0</td><td>A' C D F G I L</td></tr><tr><td>CS*</td><td>61.6</td><td>70.9</td><td><strong>71.7</strong></td><td>66.7</td><td><strong>73.9</strong></td><td>73.7</td><td>60.1</td><td><strong>71.4</strong></td><td>68.9</td><td>A C F G H I J K L M</td></tr><tr><td>DE</td><td>37.9</td><td><strong>45.8</strong></td><td>41.1</td><td>33.3</td><td>44.8</td><td>*45.5</td><td>39.9</td><td>27.9</td><td><strong>47.6</strong></td><td>A B C D E J L N</td></tr><tr><td>EL</td><td>35.6</td><td><strong>42.8</strong></td><td>40.1</td><td>39.9</td><td>46.3</td><td><strong>46.9</strong></td><td>48.0</td><td>56.2</td><td><strong>57.3</strong></td><td>A B C E J K L</td></tr><tr><td>ES*</td><td>56.9</td><td><strong>58.9</strong></td><td>57.4</td><td>56.7</td><td><strong>60.3</strong></td><td>58.4</td><td>61.2</td><td>63.5</td><td><strong>66.0</strong></td><td>A C D F G H I J K L</td></tr><tr><td>FA*</td><td>72.2</td><td>84.3</td><td><strong>86.6</strong></td><td>72.5</td><td>84.8</td><td><strong>90.2</strong></td><td>67.3</td><td><strong>87.7</strong></td><td>81.1</td><td>A C I J K</td></tr><tr><td>FR</td><td>44.6</td><td><strong>60.6</strong></td><td>57.7</td><td>49.3</td><td><strong>62.6</strong></td><td>*61.5</td><td>66.0</td><td>71.1</td><td><strong>73.8</strong></td><td>A B C E I J K L</td></tr><tr><td>HE**</td><td><strong>33.4</strong></td><td>29.9</td><td><strong>33.4</strong></td><td>29.6</td><td>30.5</td><td><strong>31.3</strong></td><td><strong>30.0</strong></td><td>17.0</td><td>26.8</td><td>A' C E F G H K L</td></tr><tr><td>HU</td><td>68.3</td><td><strong>74.8</strong></td><td>*74.0</td><td>64.9</td><td><strong>72.1</strong></td><td>*70.8</td><td>73.7</td><td>23.5</td><td><strong>83.7</strong></td><td>A B C D F G H K L N</td></tr><tr><td>IT</td><td>39.2</td><td>28.2</td><td><strong>39.9</strong></td><td>39.5</td><td>29.8</td><td><strong>43.6</strong></td><td><strong>33.7</strong></td><td>27.4</td><td>27.2</td><td>A B C H J L</td></tr><tr><td>LT**</td><td>30.5</td><td><strong>34.1</strong></td><td>28.4</td><td>27.3</td><td><strong>31.7</strong></td><td>25.3</td><td>20.7</td><td>8.6</td><td><strong>21.5</strong></td><td>A' C D E F G H I J K L M</td></tr><tr><td>MT*</td><td>8.2</td><td>6.9</td><td><strong>14.4</strong></td><td>12.3</td><td>9.4</td><td><strong>16.3</strong></td><td>7.7</td><td><strong>8.1</strong></td><td>7.2</td><td>A C F G H J L M</td></tr><tr><td>PL</td><td>72.6</td><td><strong>75.1</strong></td><td>69.1</td><td>71.8</td><td>75.5</td><td>*72.7</td><td>70.0</td><td>70.4</td><td><strong>73.6</strong></td><td>A B C H L</td></tr><tr><td>PT</td><td>65.5</td><td><strong>69.6</strong></td><td>67.3</td><td>67.4</td><td><strong>71.4</strong></td><td>70.9</td><td>65.2</td><td>64.7</td><td><strong>67.5</strong></td><td>All features</td></tr><tr><td>RO*</td><td>55.0</td><td><strong>86.3</strong></td><td>*77.8</td><td>65.4</td><td><strong>87.0</strong></td><td>*83.6</td><td>61.8</td><td><strong>86.3</strong></td><td>86.0</td><td>A C D E F G H I J K</td></tr><tr><td>SL</td><td>13.9</td><td>42.9</td><td><strong>43.2</strong></td><td>17.6</td><td>45.7</td><td><strong>46.6</strong></td><td>17.3</td><td><strong>47.7</strong></td><td>40.8</td><td>A B C F G H I K N</td></tr><tr><td>SV</td><td>10.4</td><td>30.1</td><td><strong>30.4</strong></td><td>10.1</td><td>34.3</td><td>*31.5</td><td>6.9</td><td><strong>25.0</strong></td><td>24.7</td><td>All features except N</td></tr><tr><td>TR</td><td>11.3</td><td>53.8</td><td><strong>55.4</strong></td><td>18.1</td><td>53.9</td><td>55.3</td><td>19.3</td><td>58.1</td><td><strong>60.1</strong></td><td>A B C F G H I K</td></tr><tr><td>AVG</td><td>46.2</td><td>56.5</td><td><strong>56.7</strong></td><td>48.5</td><td>58.1</td><td><strong>59.2</strong></td><td>52.0</td><td>60.3</td><td><strong>64.5</strong></td><td></td></tr></tbody></table>
task, and with (iii) ATILF-LLF 1.<fnref n="12" /> The table shows that results are very hetero-
geneous across languages. We can hypothesize that multiple factors come into
play, such as the size of corpora, the availability and the quality of annotations,
the most common VMWE categories in train and test sets, the percentage of un-
known VMWEs in test sets. For example, Figure 2 illustrates the impact of this
last trait, showing an approximative linear negative correlation between VMWE-
based F-score and the proportion of unknown VMWE occurrences in test sets.<fnref n="13" />

Because the datasets have very varying sizes across languages, we provide
in the last row of the table the weighted average F-scores, with each language F-
score weighted by the size of the test set (or of the training set in cross-validation).

**Comparison with the best results at the shared task:** Although the ATILF-LLF 2 benefited from more design time, it is interesting to compare its results to the best results obtained at the shared task for each language. When considering the weighted average results (last row of Table 2), it can be seen that the VMWE-based results are almost as high for ATILF-LLF 2 as for the Best of ST (56.5 versus 56.7), and are ahead for 9 languages out of 18. For token-based results, our system is a bit less effective: while still ahead for 10 languages out of 18, it is on average 1.1 point lower (58.1 versus 59.2). This can be viewed as a particularity of our system: while the token-based results are generally higher than the VMWE-based ones (for the baseline, or for other participating systems, cf. Savary et al. 2017), the gap is less pronounced in our case.

**Comparison with the baseline:** The baseline system is a string matching-based system that uses a lemma-based VMWE dictionary extracted from the training set and identifies as VMWEs all matching strings in the test set.

The matching procedure is very simple: a VMWE is identified inside a sentence if all of its components (lemmas if available, otherwise word forms) occur in the sentence, provided that the order of the components corresponds to an order observed in the dictionary and that the distances between them do not exceed the maximal observed distances in the training dataset.

<footnote n="12">¹²More precisely, for the results on the test sets (columns 2 to 7), the **Best of ST** columns reflect the performance of ATILF-LLF 1 for the non starred values (cf. no star means we ranked first). F-scores of ATILF-LLF 1 for starred values are as following: Hungarian=70%, Romanian=75% for VMWE-based and German=41%, French=60%, Hungarian=67.4%, Polish=70.5%, Romanian=79.1% and Swedish=30% for token-based.</footnote>

<footnote n="13">¹³We also checked for the correlation between the F-score and the training set size, and obtained a positive correlation, but less marked, in particular some languages like Czech and Turkish reach relatively low scores given the size of training data, which is better explained considering the unknown VMWE rate.</footnote>
Regarding VMWE-based evaluation, ATILF-LLF 2 outperforms the baseline in all experimental settings but four (VMWE-based evaluation on test set for Hebrew and VMWE-based cross-validation for Hebrew and Italian): on average, we obtain about 10- and 12.5-point F-score difference when evaluating on the test set and in cross-validation respectively. Yet, the baseline consistently beats our system on Hebrew. This might be explained by several characteristics of this language preventing the system to generalize well to morpho-syntactic variants: (i) small training set and (ii) no companion linguistic information (no POS, no lemmas, no syntactic parses).

**Comparison between ATILF-LLF 2 and ATILF-LLF 1:** The ATILF-LLF 1 system participated in the shared task and reached the best VMWE-based scores for almost all languages (cf. the two starred results out of 18 in column Best of ST, for Hungarian and Romanian). It can be seen that ATILF-LLF 2 shows comparable performance on the same test sets (see in particular the weighted average performance shown in the last row: 56.5 versus 56.7). It is worth noticing though that there is great variation between results on test sets and results in cross-validation. As the latter are more representative, let us focus on them (columns 8 to 10). Despite a few languages showing a drop in performance (in particular Bulgarian, Farsi and Slovene), ATILF-LLF 2 beats ATILF-LLF 1 for 10 languages out of 18, and the average result (last row of Table 2) has improved (4.2-point gain). Again, even though ATILF-LLF 2 benefited from more design time, this is a good result considering that (i) ATILF-LLF 1 did identification only, and the introduction of the categorization task led us to multiply the number of transitions (e.g. 5 MERGE AS C transitions instead of 1), (ii) the expressive power was increased to some cases of embeddings and (iii) the overall architecture is more elegant since hard-coded procedures included in the rush of the shared task have been replaced by features.<fnref n="14" />

**Categorization results:** Table 3 details the categorization results for the basic categories over all languages but Farsi (cf. the Farsi dataset does not comprise VMWE category information).<fnref n="15" /> The table allows us to compare the performance of our system with best-performing shared task systems (for the systems having the optional categorization predictions, note that our former system ATILF-LLF 1 is excluded given that it does not categorize VMWEs). It can be seen

<footnote n="14">¹⁴It is worth noting that feature groups for each language were very close for both systems (ATILF-LLF 1, 2). However, we transformed the dictionary-based hard-coded feature groups into dynamic ones.</footnote>

<footnote n="15">¹⁵We do not include the category OTH because of its marginal presence in test and train datasets for all languages but Turkish (for which our F-score is 51.5 and the Shared task best F-score is 54.6).</footnote>
<table><caption>Table 3: VMWE categorization: detailed results for the four basic categories over all the languages except Farsi. For each category, we display the proportion of the given category in test set, the F-scores $F_1$ for ATILF-LLF 2, and the best performing shared task systems (Best of ST), among systems having provided categorization information.</caption><thead><tr><th rowspan="2">Languages</th><th colspan="3">LVC</th><th colspan="3">IReflV</th><th colspan="3">VPC</th><th colspan="3">ID</th></tr><tr><th>%</th><th>F<sub>1</sub></th><th>In test set</th><th>%</th><th>F<sub>1</sub></th><th>In test set</th><th>%</th><th>F<sub>1</sub></th><th>In test set</th><th>%</th><th>F<sub>1</sub></th><th>In test set</th><th>%</th></tr></thead><tbody><tr><td>BG</td><td>16</td><td>28.6</td><td>63</td><td>68.3</td><td>46.6</td><td></td><td></td><td></td><td></td><td></td><td>21</td><td>19.4</td><td></td></tr><tr><td>CS</td><td>20</td><td>53.3</td><td>40.1</td><td>68</td><td>80.7</td><td>73.3</td><td></td><td></td><td></td><td></td><td>11</td><td>33.6</td><td>22.0</td></tr><tr><td>DE</td><td>8</td><td>4.7</td><td>2.3</td><td>4</td><td>8.9</td><td>16.0</td><td>45</td><td>58.3</td><td>43.3</td><td>43</td><td>29.1</td><td>16.4</td><td></td></tr><tr><td>EL</td><td>67</td><td>44.4</td><td>33.2</td><td></td><td></td><td></td><td>3</td><td>52.2</td><td>36.4</td><td>25</td><td>27.1</td><td>15.4</td><td></td></tr><tr><td>ES</td><td>21</td><td>49.0</td><td>35.1</td><td>44</td><td>72.0</td><td>40.4</td><td></td><td></td><td></td><td></td><td>33</td><td>39.0</td><td>13.8</td></tr><tr><td>FR</td><td>54</td><td>40.0</td><td>42.8</td><td>21</td><td>78.6</td><td>68.3</td><td></td><td></td><td></td><td></td><td>24</td><td>75.0</td><td>60.8</td></tr><tr><td>HE</td><td>25</td><td>31.4</td><td></td><td></td><td></td><td></td><td>37</td><td>27.2</td><td></td><td>6</td><td>16.2</td><td></td><td></td></tr><tr><td>HU</td><td>29</td><td>50.9</td><td>41.5</td><td></td><td></td><td></td><td>71</td><td>80.3</td><td>77.2</td><td></td><td></td><td></td><td></td></tr><tr><td>IT</td><td>17</td><td>17.5</td><td>12.9</td><td>30</td><td>30.3</td><td>9.3</td><td>2</td><td>50.0</td><td>14.3</td><td>50</td><td>25.7</td><td>20.3</td><td></td></tr><tr><td>LT</td><td>42</td><td>56.3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>58</td><td>14.9</td><td></td><td></td></tr><tr><td>MT</td><td>52</td><td>6.6</td><td>5.8</td><td></td><td></td><td></td><td></td><td></td><td></td><td>52</td><td>7.7</td><td>2.1</td><td></td></tr><tr><td>PL</td><td>34</td><td>62.9</td><td>39.1</td><td>53</td><td>87.3</td><td>80.2</td><td></td><td></td><td></td><td>13</td><td>51.5</td><td></td><td></td></tr><tr><td>PT</td><td>66</td><td>68.0</td><td></td><td>16</td><td>68.9</td><td></td><td></td><td></td><td></td><td>18</td><td>65.3</td><td></td><td></td></tr><tr><td>RO</td><td>27</td><td>87.4</td><td>86.3</td><td>58</td><td>86.3</td><td>79.1</td><td></td><td></td><td></td><td>15</td><td>76.7</td><td>65.6</td><td></td></tr><tr><td>SL</td><td>9</td><td>7.4</td><td>8.3</td><td>51</td><td>45.7</td><td>40.8</td><td>22</td><td>47.5</td><td>34.5</td><td>18</td><td>09.1</td><td>3.9</td><td></td></tr><tr><td>SV</td><td>6</td><td>16.7</td><td>21.1</td><td>6</td><td>08.7</td><td></td><td>66</td><td>33.6</td><td>30.2</td><td>21</td><td>13.3</td><td>3.8</td><td></td></tr><tr><td>TR</td><td>40</td><td>57.6</td><td>59.1</td><td></td><td></td><td></td><td></td><td></td><td></td><td>11</td><td>49.3</td><td>49.8</td><td></td></tr></tbody></table>

that our system reaches high performance on categorization too, although performance varies greatly across categories. Although the general trend is higher performance for IReflV, then LVC, then ID, Figure 3 shows that this pattern is not systematic. For instance, results are relatively low for Czech given its high IReflV proportion. On the contrary, results for Portuguese are high despite a high LVC ratio.
<figure><img src="image_44.png" /><figcaption>Figure 2: Correlation between the ATILF-LLF 2 identification results for each language (F-score, on the x axis) and the percentage of occurrences of test VMWEs unknown in the train set (y axis).</figcaption></figure>

<figure><img src="image_45.png" /><figcaption>Figure 3: A graph ranking all languages except Farsi according to their F-scores. In each bar, the proportions of VMWE categories in the test set are shown using symbols.</figcaption></figure>
**5 Related work**

A popular VMWE identification method is to use a sequence labeling approach, with IOB-based tagsets. For instance, Diab & Bhutada (2009) apply a sequential SVM to identify verb-noun idiomatic combinations in English. Note also that three (out of seven) systems participating in the PARSEME shared task used such approach (Boroş et al. 2017; Maldonado et al. 2017; Klyueva et al. 2017). Such an approach was also investigated for MWE identification in general (including verbal expressions) ranging from continuous expressions (Blunsom & Baldwin 2006) to gappy ones (Schneider et al. 2014). Recently, neural networks have been successfully integrated into this framework (Legrand & Collobert 2016; Klyueva et al. 2017).

VMWE identification can naturally take advantage of previously predicted syntactic parses. Some systems use them as soft constraints. For instance, the sequence labeling systems of the shared task and our system use them as source of features in their statistical tagging models. There also exist approaches using syntactic parses as hard constraints. For example, Baptista et al. (2015) apply hand-crafted identification rules on them. Fazly et al. (2009) and Nagy T. & Vincze (2014) propose a two-pass identification process consisting of candidate extraction followed by binary classification. In particular, candidate extraction takes advantage of predicted syntactic parses, through the use of linguistic patterns.

A joint syntactic analysis and VMWE identification approach using off-the-shelf parsers is another interesting alternative that has shown to help VMWE identification such as light-verb constructions (Eryiğit et al. 2011; Vincze & Csirik 2010). Some parsers integrate mechanisms into the parsing algorithm to identify MWEs on top of predicting the syntactic structure, like in Wehrli (2014) and Constant & Nivre (2016), our system being a simplified version of the latter.

**6 Conclusion and future work**

This article presents a simple transition-based system devoted to VMWE identification and categorization. In particular, it offers a simple mechanism to handle discontinuity and embedding, which is a crucial point for VMWEs. Results on the PARSEME Shared Task datasets show that our system is quite robust across languages, and achieves very competitive results. Its linear time complexity is also an asset.
As future work, we would like to apply more sophisticated syntax-based features, as well as more advanced machine-learning techniques like neural networks. We also plan to investigate the use of a dynamic oracle (Goldberg & Nivre 2012).

## Acknowledgements

This work was partially funded by the French National Research Agency (PARSEME-FR ANR-14-CERA-0001).

## Abbreviations

<table><tr><td>ID</td><td>Idiom</td><td>MWT</td><td>multiword token</td></tr><tr><td>IOB</td><td>inside outside beginning</td><td>OTH</td><td>other verbal MWE</td></tr><tr><td>IREFLV</td><td>inherently reflexive verb</td><td>VMWE</td><td>verbal multiword expression</td></tr><tr><td>LVC</td><td>light-verb construction</td><td>VPC</td><td>verb-particle construction</td></tr><tr><td>MWE</td><td>multiword expression</td><td></td><td></td></tr></table>

## References

Al Saied, Hazem, Matthieu Constant & Marie Candito. 2017. The ATILF-LLF system for parseme shared task: A transition-based verbal multiword expression tagger. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 127–132. Association for Computational Linguistics. DOI:10.18653/v1/W17-1717

Baptista, Jorge, Graça Fernandes, Rui Talhadas, Francisco Dias & Nuno Mamede. 2015. Implementing European Portuguese verbal idioms in a natural language processing system. In Gloria Corpas Pastor (ed.), *Proceedings of europhras 2015*, 102–115.

Blunsom, Phil & Timothy Baldwin. 2006. Multilingual deep lexical acquisition for HPSGs via supertagging. In *Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing*, 164–171. Sydney, Australia: Association for Computational Linguistics. DOI:10.3115/1610075.1610101

Boroş, Tiberiu, Sonia Pipa, Verginica Barbu Mititelu & Dan Tufiş. 2017. A data-driven approach to verbal multiword expression detection. PARSEME shared task system description paper. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 121–126. Association for Computational Linguistics. DOI:10.18653/v1/W17-1716
Constant, Matthieu & Joakim Nivre. 2016. A transition-based system for joint lexical and syntactic analysis. In *Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers)*, 161–171. Association for Computational Linguistics. http://www.aclweb.org/anthology/P16-1016.

Diab, Mona & Pravin Bhutada. 2009. Verb noun construction MWE token classification. In *Proceedings of the Workshop on Multiword Expressions: Identification, Interpretation, Disambiguation and Applications*, 17–22. Association for Computational Linguistics. http://www.aclweb.org/anthology/W/W09/W09-2903.

Eryiğit, Gülşen, Tugay İlbay & Ozan Arkan Can. 2011. Multiword expressions in statistical dependency parsing. In *Proceedings of IWPT Workshop on Statistical Parsing of Morphologically-Rich Languages* (SPMRL 2011), 45–55. http://dl.acm.org/citation.cfm?id=2206359.2206365. October 6, 2011.

Fazly, Afsaneh, Paul Cook & Suzanne Stevenson. 2009. Unsupervised type and token identification of idiomatic expressions. *Computational Linguistics* 35(1). 61–103. http://aclweb.org/anthology/J09-1005.

Goldberg, Yoav & Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In *Proceedings of the 24th International Conference on Computational Linguistics (COLING)*, 959–976.

Goldberg, Yoav & Joakim Nivre. 2013. Training deterministic parsers with non-deterministic oracles. *Transactions of the Association for Computational Linguistics* 1. 403–414.

Klyueva, Natalia, Antoine Doucet & Milan Straka. 2017. Neural networks for multi-word expression detection. In *Proceedings of the 13th Workshop on Multi-word Expressions* (MWE ’17), 60–65. Association for Computational Linguistics. April 4, 2017. DOI:10.18653/v1/W17-1707

Legrand, Joël & Ronan Collobert. 2016. Phrase representations for multiword expressions. In *Proceedings of the 12th Workshop on Multiword Expressions* (MWE ’16), 67–71. Association for Computational Linguistics. http://anthology.aclweb.org/W16-1810.

Maldonado, Alfredo, Lifeng Han, Erwan Moreau, Ashjan Alsulaimani, Koel Dutta Chowdhury, Carl Vogel & Qun Liu. 2017. Detection of verbal multi-word expressions via conditional random fields with syntactic dependency features and semantic re-ranking. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE ’17), 114–120. Association for Computational Linguistics. DOI:10.18653/v1/W17-1715

Nagy T., István & Veronika Vincze. 2014. VPCTagger: Detecting verb-particle constructions with syntax-based methods. In *Proceedings of the 10th Workshop*
on *Multiword Expressions* (MWE '14), 17–25. Association for Computational Linguistics. http://www.aclweb.org/anthology/W14-0803.

Nivre, Joakim. 2004. Incrementality in deterministic dependency parsing. In Frank Keller, Stephen Clark, Matthew Crocker & Mark Steedman (eds.), *Proceedings of the ACL Workshop on Incremental Parsing: Bringing Engineering and Cognition together*, 50–57. Association for Computational Linguistics.

Nivre, Joakim. 2014. *Transition-based parsing with multiword expressions*. IC1207 COST PARSEME 2nd general meeting. Athens, Greece. https://typo.uni-konstanz.de/PARSEME/images/Meeting/2014-03-11-Athens-meeting/PosterAbstracts/WG3-Nivre-athens-poster.pdf.

Savary, Agata, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova & Antoine Doucet. 2017. The PARSEME Shared Task on automatic identification of verbal multiword expressions. In *Proceedings of the 13th Workshop on Multiword Expressions* (MWE '17), 31–47. Association for Computational Linguistics. DOI:10.18653/v1/W17-1704

Schneider, Nathan, Emily Danchik, Chris Dyer & Noah A. Smith. 2014. Discriminative lexical semantic segmentation with gaps: Running the MWE gamut. *Transactions of the Association for Computational Linguistics* 2(1). 193–206. https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/281.

Seretan, Violeta. 2011. *Syntax-based collocation extraction* (Text, Speech and Language Technology). Dordrecht, Heidelberg, London, New York: Springer.

Vincze, Veronika & János Csirik. 2010. Hungarian corpus of light verb constructions. In *Proceedings of the 23rd International Conference on Computational Linguistics* (COLING '10), 1110–1118. Association for Computational Linguistics. http://www.aclweb.org/anthology/C10-1125.

Wehrli, Eric. 2014. The relevance of collocations for parsing. In *Proceedings of the 10th Workshop on Multiword Expressions* (MWE '14), 26–32. Association for Computational Linguistics. 26–27 April, 2014.

Zhang, Yue & Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In *Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies*, 188–193. Association for Computational Linguistics. http://www.aclweb.org/anthology/P11-2033.
# Chapter 8

# Identifying verbal multiword expressions with POS tagging and parsing techniques

Katalin Ilona Simkó

University of Szeged

Viktória Kovács

University of Szeged

Veronika Vincze

University of Szeged

MTA-SZTE Research Group on Artificial Intelligence

The chapter describes an extended version (USzeged+) of our previous system (USzeged) submitted to PARSEME's Shared Task on automatic identification of verbal multiword expressions. USzeged+ exploits POS tagging and dependency parsing to identify single- and multi-token verbal MWEs in text. USzeged competed on nine of the eighteen languages, where USzeged+ aims to identify the VMWEs in all eighteen languages of the shared task and contains fixes for deficiencies of the previously submitted system. Our chapter describes how our system works and gives a detailed error analysis.

## 1 Introduction

Multiword expressions (MWEs) are frequent elements of all natural languages. They are made up of more than one lexeme, but their meaning is not predictable from the meaning of their components. There are different types of MWEs such

<figure><img src="image_46.png" /></figure>
as stereotyped similes (*as white as snow*), collocations (*strong tea*), or idioms (*to kick the bucket*). This chapter deals with verbal MWEs (VMWEs) where the head element of the MWE is a verb, for example verb-particle constructions (*look after*), or light-verb constructions (*take a shower*).

This chapter describes our system for verbal MWE recognition. It was built for the PARSEME Shared Task 1.0 (Savary et al. 2017), USzeged and its extension, USzeged+. Both systems use POS tagging and dependency parsing and are capable of identifying single- and multi-token verbal MWEs. They are language-independent: USzeged was submitted for nine of the eighteen languages of the Shared Task, while for this extended version, USzeged+, we present results for all eighteen languages.

In this chapter, we first describe the original USzeged system and give our results submitted to the Shared Task with detailed error analysis. This part of the chapter builds heavily on our workshop paper (Simkó et al. 2017). Then, we describe the details of the updated, USzeged+ version and give the results we achieved with this new system. Last, we give a comparison of the results achieved using our approach in the original, USzeged system and the new USzeged+ one in an experiment using the available Hungarian data.

## 2 USzeged - The original system

The USzeged system was built for the shared task on automatic identification of verbal multiword expressions organized as part of the 2017 MWE workshop (Savary et al. 2017).<fnref n="1" /> The shared task’s aim is to identify verbal MWEs in multiple languages. In total, eighteen languages are covered that were annotated using guidelines taking universal and language-specific phenomena into account.

The guidelines identify five different types of verbal MWEs: idioms (ID), light-verb constructions (LVC), verb-particle constructions (VPC), inherently reflexive verbs (IRefIV) and “other” (OTH). Their identification in natural language processing is difficult because they are often discontinuous and non-compositional, the categories are heterogeneous and the structures show high syntactic variability.

The precise definitions of MWE, VMWE and the VMWE types can be found in Savary et al. (2018 [this volume]), as well as details on the different languages’ databases used.

Our team created the Hungarian shared task database and VMWE annotation. Our system is mostly based on our experiences with the Hungarian data in this

<footnote n="1">¹http://multiword.sourceforge.net/sharedtask2017</footnote>
annotation phase. Our goal was to create a simple system capable of handling
MWE identification in multiple languages.

## 2.1 System description

The USzeged system exploits the syntactic relations within MWEs, i.e. it directly connects MWEs and parsing, an approach described in many sources (Constant & Nivre 2016; Nasr et al. 2015; Candito & Constant 2014; Green et al. 2011; 2013; Wehrli et al. 2010; Waszczuk et al. 2016) and one of the basic ideas behind the work done by the PARSEME group.<fnref n="2" /> The core of our system is directly based on the work described in Vincze et al. (2013): using dependency parsing to identify MWEs. That system uses complex dependency relations specific to the given syntactic relation and MWE type. We note that a high number of the languages of the shared task are morphologically rich and have free word order, which entails that syntactically flexible MWEs might not be adjacent. Hence, a syntax-based approach seems a better fit for the task than sequence labeling or similar strategies.

The USzeged system uses only the MWE type as a merged dependency label, i.e. no clue is encoded to the syntactic relation between two parts of the MWE. Moreover, it also treats single-token MWEs. As multiple languages had single-token MWEs as well as multi-token ones that are dealt with in dependency parsing, we expanded the approach using POS tagging. Frequent single-token MWEs are, for example, German and Hungarian VPCs: when the particle directly precedes the verb, German and Hungarian spelling rules require that they are spelled as one word, however, it still remains a construction made up of two lexemes with non-compositional meaning (e.g. (HU) **kinyír** (ki+nyír) ‘out+cut’ ⇒ ‘kill’ or (DE) **aufmachen** (auf+machen) ‘up+do’ ⇒ ‘open’).

MWEs have specific morphological, syntactic and semantic properties. Our approach treats multi-token MWEs on the level of syntax – similarly to the mwe dependency relation in the Universal Dependency grammar (Nivre 2015) – and single-token MWEs on the level of morphology.

The USzeged system works in four steps, and the main MWE identification happens during POS tagging and dependency parsing of the text. Our system relies on the POS tagging and dependency annotations provided by the organizers of the shared task in the companion CoNLL files and the verbal MWE annotation of the texts and is completely language-independent given those inputs.

<footnote n="2">²http://typo.uni-konstanz.de/parseme/</footnote>
In the first step, we prepared the training file from the above mentioned inputs. We merged the training MWE annotation into its morphological and dependency annotation for single- and multi-token MWEs, respectively. The POS tag of single-token MWEs got replaced with their MWE type, while for the multi-token MWEs the dependency graphs' label changed: the label of the dependent node in the tree was replaced with a label denoting the MWE type.

Figure 1, Figure 2 and Figure 3 show the single-token MWE's change in POS tag and multi-token MWE dependency relabeling for VPCs and LVCs in a Hungarian example.

<table><caption>Figure 1: Adding the VPC and LVC single-token MWE POS tags to (HU) **bekezdés** (be+kezdés) 'in+starting' ⇒ 'paragraph' and (HU) **határozathozatal** (határozat hozatal) 'decision+bringing' ⇒ 'decision-making'.</caption><thead><tr><th></th><th>original label</th><th>relabeled</th><th>(HU)</th></tr></thead><tbody><tr><td><b>bekezdés</b><br>in+starting, 'paragraph'</td><td>NOUN</td><td>VPC</td><td>(HU)</td></tr><tr><td><b>határozathozatal</b><br>decision+bringing, 'decision-making'</td><td>NOUN</td><td>LVC</td><td>(HU)</td></tr></tbody></table>

<figure><img src="image_47.png" /><figcaption>Figure 2: Adding the VPC multi-token MWEs label to the dependency graph in (HU) Péter fontos feladatokat *lát* el. 'Peter important tasks sees away' ⇒ 'Peter takes care of important tasks'.</figcaption></figure>
<figure><img src="image_48.png" /><figcaption>Figure 3: Adding the ID multi-token MWE label to the dependency graph in (HU) Péter **vetette** rá az **első** követ. ‘Peter cast the first stone on him’.</figcaption></figure>

For multi-token MWEs our approach is based on our hypothesis that the dependent MWE elements will be directly connected to the other MWE element(s). We do not change the structure of the dependency relations in the tree, but change the dependency label of the dependent MWE element to the MWE type, therefore making the MWE element retraceable from the dependency annotation of the sentence. For example *lát* and *el* in Figure 2 make up a VPC (**ellát** ‘take care’), so the dependency relation label of the dependent element, *el* changes from the general syntactic label **PREVERB** to the MWE label **VPC**, with this **VPC** label now connecting the two elements of the MWE.

For MWEs of more than two tokens, the conversion replaces the dependency labels of all MWE elements that depend on the head. In Figure 3, the head of the idiom (**az első követ veti** ‘casts the first stone’) is the verb, *vetette* (cast.Sg3.Past). All other elements’ dependency labels are changed to **ID**.

The second step is training the parser: we used the Bohnet parser (Bohnet 2010) for both POS tagging and dependency parsing. For the single-token MWEs, we trained the Bohnet parser’s POS tagger module on the MWE-merged corpora and its dependency parser for the multi-token MWEs. The parser would treat the MWE POS tags and dependency labels as any other POS tag and dependency label.

We did the same for each language and created POS tagging and dependency parsing models capable of identifying MWEs for them. For some languages in the
shared task, we had to omit sentences from the training data that were overly long (spanning over 500 tokens in some cases) and therefore caused errors in training due to lack of memory. This affected one French, one Polish, two Italian, five Romanian and nine Turkish sentences.

Third, we ran the POS tagging and dependency parsing models of each language on their respective test corpora. The output contains the MWE POS tags and dependency labels used in that language as well as the standard POS and syntactic ones.

The fourth and last step is to extract the MWE tags and labels from the output of the POS tagger and the dependency parser. The MWE POS tagged words are annotated as single-token MWEs of the type of their POS tag. From the MWE dependency labels, we annotate the words connected by MWE labels of the same type as making up a multi-token MWE of that type (see Figure 4).

<figure><img src="image_49.png" /><figcaption>Figure 4: Steps of the USzeged system.</figcaption></figure>

There are arguments for and against our approach. The system cannot handle multi-token MWEs where the elements are not connected in the tree and replacing the POS tags and dependency labels can have a negative effect on the accuracy of POS tagging and parsing. However, as our end goal is not the POS tagging or dependency parse of the data, we believe that this side effect is negligible since higher-level applications (e.g. machine translation) can profit from more accurate MWE identification. On the other hand, the approach has low technical requirements and it is very easily adaptable to other languages.

## 2.2 Results

We submitted the USzeged system for all languages in the shared task with provided dependency analysis and POS tagging. We attempted to use just the POS tagging component of our system on the languages that only had POS tagging available to give partial results (i.e. identifying only single-token MWEs), but we
found that these languages incidentally had no or very few single-token MWEs
(Farsi 0, Maltese 4, Romanian 44, Slovene 3, Turkish 22), therefore we had no
access to adequate training data and did not submit results for these languages.

Our results on the nine languages are reported in Simkó et al. (2017). Our sys-
tem was submitted for German, Greek, Spanish, French, Hungarian, Italian, Pol-
ish, Portuguese, and Swedish. For the evaluation, we employed the metrics used
for the evaluation of the shared task (Savary et al. 2017).

The F-scores show great differences between languages, but so did they for the
other systems submitted. Compared to the other, mostly closed-track systems,
the USzeged system ranked close to or at the top on German, Hungarian, and
Swedish. For the other languages (except for Polish and Portuguese, where ours
is the worst performing system), we ranked in the mid-range.

## 2.3 Error analysis

After receiving the gold annotation for the test corpora, we investigated the
strengths and weaknesses of our system.

Our error analysis showed that the USzeged system performs by far best on
single-token MWEs, which in this dataset are mostly made up of the verb-particle
construction category, correctly identifying around 60% of VPCs, but only about
40% of other types on average. It is probably due to the fact that single-token
MWEs are identified by POS tagging techniques, which are known to obtain
more accurate results in most languages than dependency parsing.

German, Hungarian, and Swedish were also the languages with the highest
proportions of the VPC type of verbal MWEs in the shared task, which also corre-
lates with why our system performed best on them. Romance languages contain
almost no VPCs and the remaining ones have much less also. In this way, the
frequency of VPCs strongly influences our results on the given language.

For French and Italian, our system also performed worse on IReflVs. In gen-
eral, we had some trouble identifying longer IDs and LVCs and MWEs including
prepositions. A further source of error was when there was no syntactic edge
in between members of a specific MWE, for instance, in German, the copula
*sein* ‘be’ was often indirectly connected to the other words of the MWE (e.g. *im
Rennen sein* ‘in race be’ ⇒ ‘to compete’), hence our method was not able to rec-
ognize it as part of the MWE. As our system does not restructure the syntactic
trees, if the elements of a multi-token VMWE are not connected (i.e. they do not
form a graph) in their dependency annotation, we cannot identify the full MWE,
however, we can still identify tokens of it correctly if at least two tokens within
the MWE are attached.
## 3 The extended system - USzeged+

The primary aim of our extension was to be able to use our system for the languages in the shared task without any available POS and dependency data. We achieved this by parsing the annotated set in a preprocessing step. For the languages with gold POS and dependency data already available, we did not use this extra step (see Figure 5).

<figure><img src="image_50.png" /><figcaption>Figure 5: Steps of the USzeged+ system.</figcaption></figure>

We used data for the remaining languages from the Universal Dependencies Project release 2.0 (Nivre et al. 2016) to train the Bohnet parser for POS-tagging and dependency parsing and parsed the VMWE annotated shared task's training sets. We should note that for some languages, the VMWE corpus and the Universal Dependencies corpus are overlapping. This influences our dependency parse to some degree as the training data might partially include the test data, but as our end goal here is not the full dependency parse of the texts (moreover, we already use gold dependency annotations for the languages which have it directly available), we feel that this factor is negligible. Henceforward, we exploited the very same processes as before: we merged the parsed data with the VMWE annotations and once again, trained the Bohnet parser on the VMWE merged data. We then parsed the test sets for the shared task and extracted the MWE POS-tagged and MWE dependency labeled words and phrases.

### 3.1 Results

Table 1 shows the USzeged+ results for all shared task languages. The languages covered by USzeged can be found in the upper part of the table, and the ones covered by USzeged+ are in the lower part. The “upper” languages of this table show differences to the results presented in (Simkó et al. 2017). This is due to two main factors: the Bohnet parser was updated between our USzeged and USzeged+
versions of the system and we also corrected some bugs in our conversion tool. The basic working principles of our system are the same as described above.

<table><caption>Table 1: USzeged+ results: Languages covered by the previous system also are on top.</caption><thead><tr><th></th><th>P-MWE</th><th>R-MWE</th><th>F1-MWE</th><th>P-token</th><th>R-token</th><th>F1-token</th></tr></thead><tbody><tr><td>DE</td><td>31.16</td><td>40.20</td><td>35.11</td><td>40.65</td><td>43.05</td><td>41.82</td></tr><tr><td>EL</td><td>37.01</td><td>30.20</td><td>33.26</td><td>49.14</td><td>32.65</td><td>39.23</td></tr><tr><td>ES</td><td>25.67</td><td>52.00</td><td>34.37</td><td>32.13</td><td>55.20</td><td>40.62</td></tr><tr><td>FR</td><td>31.23</td><td>31.60</td><td>31.41</td><td>43.57</td><td>39.44</td><td>41.40</td></tr><tr><td>HU</td><td>62.02</td><td>71.34</td><td>66.36</td><td>58.45</td><td>69.08</td><td>63.32</td></tr><tr><td>IT</td><td>9.21</td><td>6.80</td><td>7.83</td><td>33.29</td><td>18.70</td><td>23.94</td></tr><tr><td>PL</td><td>35.96</td><td>59.40</td><td>44.80</td><td>41.33</td><td>63.68</td><td>50.13</td></tr><tr><td>PT</td><td>33.29</td><td>52.80</td><td>40.84</td><td>40.76</td><td>58.96</td><td>48.20</td></tr><tr><td>SV</td><td>14.96</td><td>22.88</td><td>18.09</td><td>20.55</td><td>28.21</td><td>23.77</td></tr><tr><td>BG</td><td>53.26</td><td>43.13</td><td>47.66</td><td>77.79</td><td>49.25</td><td>60.32</td></tr><tr><td>CS</td><td>44.95</td><td>57.93</td><td>50.62</td><td>57.60</td><td>64.46</td><td>60.84</td></tr><tr><td>FA</td><td>69.58</td><td>46.20</td><td>55.53</td><td>85.78</td><td>53.14</td><td>65.63</td></tr><tr><td>HE</td><td>41.18</td><td>8.40</td><td>13.95</td><td>55.22</td><td>8.63</td><td>14.93</td></tr><tr><td>LT</td><td>33.33</td><td>7.00</td><td>11.57</td><td>40.48</td><td>6.97</td><td>11.89</td></tr><tr><td>MT</td><td>0.00</td><td>0.00</td><td>0.00</td><td>4.65</td><td>0.31</td><td>0.59</td></tr><tr><td>RO</td><td>46.29</td><td>67.40</td><td>54.89</td><td>53.01</td><td>71.68</td><td>60.95</td></tr><tr><td>SL</td><td>59.49</td><td>18.80</td><td>28.57</td><td>66.46</td><td>18.77</td><td>29.27</td></tr><tr><td>TR</td><td>39.34</td><td>37.92</td><td>38.62</td><td>42.07</td><td>39.49</td><td>40.74</td></tr></tbody></table>

Using gold or parsed POS and dependency data as the starting phase does not have a significant impact on the results (as we will show in another experiment in §4), with the exception of Maltese. As Maltese currently has no available Universal Dependencies treebank, we used cross-language training to train our parser. As a Semitic language, Maltese is basically related to Arabic but spelt with Latin characters and about half of its vocabulary originates from Italian. Thus, we selected the available Italian Universal Dependencies treebank to train the parser and parse the VMWE train data. This had a very bad effect on our results: no full MWE could be correctly identified in the VMWE test set. Hence, for Maltese, a more suitable solution is still to be found for our approach. For all other languages – where the parser for the VMWE train data was trained on the same
language – the final results are much more comparable to those of the languages with gold trees.

Besides Maltese, one of the languages where our system performed poorly is Italian. We investigated the Italian training corpus and found that its annotation has different underlying principles than most of the other corpora. Namely, it allows sentences to have multiple roots (which is prohibited in other dependency theories), hence it confuses the parser's training to a high degree and therefore very few valuable results (i.e. MWE annotations) can be converted from the parsed sentences. Finally, Swedish results are probably due to the small size of the training corpora.

Table 2 and Table 3 show our results in F-score for the different MWE types; crossed out cells indicate that the type was not present for the given language.

Overall, the USzeged+ system performs best on inherently reflexive verbs (IReflV). IReflVs contain irreflexive pronouns, which show little variability, thus they can be relatively easily recognized by the system. However, the system performs worst on idioms and the “other” category due to their bigger variability and the longer MWEs in these types. Light-verb constructions and verb-particle constructions show varied results depending on the variability of the category in the given language. VPCs could be easily recognized in Hungarian (an F-score over 80), while LVC identification was most successful in Romanian (an F-score of 70).

There are also differences in the annotations of the languages: for instance, Farsi contains only VMWEs for the “other” category, which makes it hard to make any comparisons with the other languages on the effective identification of VMWE categories.

The results also show that while our system has very similar average results on the other language group of the shared task (interestingly even the “other” category, which is probably due to Farsi), results are much lower on Romance languages on average. This is most probably due to the issues on the Italian dependency data (see above), which resulted in poor performance for almost all of the VMWE categories in Italian.

We did some error analysis based on the languages we can speak. This revealed that as for LVCs, our system usually marks as false positives those verb-object pairs where the verb is an otherwise frequent light verb in the given language (e.g. (PT) *ter* ‘to have’). Also, participle forms of LVCs were often missed in (FR) *études menées* ‘studies conducted’. As for VPCs, many compositional instances of verbs with particles were falsely marked in German and in Hungarian, like (DE) *anheben* ‘hang up’. The same is true for IReflVs: compositional ones like (PT) *encantar-se* ‘enchant’ were sometimes falsely identified as VMWEs. A fur-
<table><caption>Table 2: USzeged+ MWE-level F-score results for the different MWE types.</caption><thead><tr><th></th><th>all</th><th>VPC</th><th>LVC</th><th>ID</th><th>IRefV</th><th>OTH</th></tr></thead><tbody><tr><td>BG</td><td>47.66</td><td>-</td><td>20.44</td><td>18.79</td><td>60.70</td><td>-</td></tr><tr><td>CS</td><td>50.62</td><td>-</td><td>26.96</td><td>2.51</td><td>61.71</td><td>0.00</td></tr><tr><td>DE</td><td>35.11</td><td>54.55</td><td>10.39</td><td>16.16</td><td>17.50</td><td>-</td></tr><tr><td>EL</td><td>33.26</td><td>56.00</td><td>36.68</td><td>9.52</td><td>-</td><td>8.00</td></tr><tr><td>ES</td><td>34.37</td><td>-</td><td>32.40</td><td>9.70</td><td>43.04</td><td>0.00</td></tr><tr><td>FA</td><td>55.53</td><td>-</td><td>-</td><td>-</td><td>-</td><td>55.53</td></tr><tr><td>FR</td><td>31.41</td><td>-</td><td>27.67</td><td>16.48</td><td>52.94</td><td>0.00</td></tr><tr><td>HE</td><td>13.95</td><td>9.35</td><td>24.20</td><td>0.00</td><td>-</td><td>13.76</td></tr><tr><td>HU</td><td>66.36</td><td>80.59</td><td>35.47</td><td>-</td><td>-</td><td>-</td></tr><tr><td>IT</td><td>7.83</td><td>22.22</td><td>5.13</td><td>5.05</td><td>0.00</td><td>0.00</td></tr><tr><td>LT</td><td>11.57</td><td>-</td><td>23.53</td><td>0.00</td><td>-</td><td>-</td></tr><tr><td>MT</td><td>0.00</td><td>-</td><td>0.00</td><td>0.00</td><td>-</td><td>0.00</td></tr><tr><td>PL</td><td>44.80</td><td>-</td><td>25.38</td><td>0.00</td><td>65.63</td><td>-</td></tr><tr><td>PT</td><td>40.84</td><td>-</td><td>45.89</td><td>11.57</td><td>40.99</td><td>-</td></tr><tr><td>RO</td><td>54.89</td><td>-</td><td>70.59</td><td>20.51</td><td>54.92</td><td>-</td></tr><tr><td>SL</td><td>28.57</td><td>0.00</td><td>14.29</td><td>1.90</td><td>45.71</td><td>0.00</td></tr><tr><td>SV</td><td>18.09</td><td>22.11</td><td>8.70</td><td>0.00</td><td>2.90</td><td>0.00</td></tr><tr><td>TR</td><td>38.62</td><td>-</td><td>39.77</td><td>32.71</td><td>-</td><td>34.34</td></tr><tr><td>Average</td><td>34.08</td><td>34.97</td><td>26.32</td><td>9.06</td><td>40.54</td><td>10.15</td></tr></tbody></table>

ther source of errors could also be some inconsistencies in the data: in a few cases, annotators missed to mark some clear examples of VMWEs in the test data, which resulted again in false positives. Finally, the German corpus contained some English sentences, e.g. [...] *if Proporz were to be taken out of the Austrian economy, actual unemployment would be ... higher?* In this sentence, *be* and *higher* are marked as an instance of VPC. The word *be* is a typical particle in German, while last words of the sentences are often verbs in German due to word order reasons. Probably this is the reason why the system gave this analysis.

As our system uses different methods to assign single- and multi-token MWE labels, we also investigated our results for these separately. We found that most languages only contain no or very few single-token MWEs, with the exception of German and Hungarian. Approximately 12% of VMWEs are single-token in the
<table><caption>Table 3: USzeged+ system's token-level F-score results for the different MWE types.</caption><thead><tr><th></th><th>all</th><th>VPC</th><th>LVC</th><th>ID</th><th>IRefV</th><th>OTH</th></tr></thead><tbody><tr><td>BG</td><td>60.32</td><td>-</td><td>32.63</td><td>24.30</td><td>74.05</td><td>-</td></tr><tr><td>CS</td><td>60.84</td><td>-</td><td>31.78</td><td>17.87</td><td>72.44</td><td>0.00</td></tr><tr><td>DE</td><td>41.82</td><td>56.14</td><td>12.58</td><td>32.41</td><td>24.85</td><td>-</td></tr><tr><td>EL</td><td>39.23</td><td>56.00</td><td>40.65</td><td>19.97</td><td>-</td><td>4.04</td></tr><tr><td>ES</td><td>40.62</td><td>-</td><td>35.86</td><td>22.47</td><td>44.27</td><td>0.00</td></tr><tr><td>FA</td><td>65.63</td><td>-</td><td>-</td><td>-</td><td>-</td><td>65.63</td></tr><tr><td>FR</td><td>41.40</td><td>-</td><td>30.17</td><td>36.11</td><td>52.94</td><td>0.00</td></tr><tr><td>HE</td><td>14.93</td><td>13.82</td><td>23.21</td><td>10.62</td><td>-</td><td>10.83</td></tr><tr><td>HU</td><td>63.32</td><td>80.59</td><td>40.82</td><td>-</td><td>-</td><td>-</td></tr><tr><td>IT</td><td>23.94</td><td>27.78</td><td>12.60</td><td>23.78</td><td>0.00</td><td>0.00</td></tr><tr><td>LT</td><td>11.89</td><td>-</td><td>23.30</td><td>3.28</td><td>-</td><td>-</td></tr><tr><td>MT</td><td>0.59</td><td>-</td><td>0.71</td><td>0.33</td><td>-</td><td>0.00</td></tr><tr><td>PL</td><td>50.13</td><td>-</td><td>28.18</td><td>13.19</td><td>69.44</td><td>-</td></tr><tr><td>PT</td><td>48.20</td><td>-</td><td>50.73</td><td>27.80</td><td>42.07</td><td>-</td></tr><tr><td>RO</td><td>60.95</td><td>-</td><td>73.03</td><td>48.88</td><td>55.75</td><td>-</td></tr><tr><td>SL</td><td>29.27</td><td>1.70</td><td>15.87</td><td>9.36</td><td>46.29</td><td>0.00</td></tr><tr><td>SV</td><td>23.77</td><td>26.80</td><td>8.79</td><td>2.26</td><td>2.90</td><td>0.00</td></tr><tr><td>TR</td><td>40.74</td><td>-</td><td>41.32</td><td>34.43</td><td>-</td><td>36.36</td></tr><tr><td>Average</td><td>39.87</td><td>37.55</td><td>29.54</td><td>20.44</td><td>44.09</td><td>10.62</td></tr></tbody></table>

German data, while they make up of 40% of VMWEs in the Hungarian data. Table 4 shows our system's accuracy on single- and multi-token MWEs for German and Hungarian.

<table><caption>Table 4: Accuracy on single- and multi-token MWEs for German and Hungarian.</caption><thead><tr><th></th><th>single-token</th><th>multi-token</th><th>overall</th></tr></thead><tbody><tr><td>DE</td><td>64</td><td>36</td><td>46</td></tr><tr><td>HU</td><td>83</td><td>43.3</td><td>68.9</td></tr></tbody></table>

These results confirm that our system achieves better results on single-token MWEs than on multi-token ones.
## 4 Gold or parsed?

In this last section, we describe a small experiment comparing our new addition to the system: the parsed POS and dependency data. We compare our results on Hungarian using the gold POS and dependency data with an experimental setup mirroring that of the languages without this gold data. We used the Hungarian Universal Dependencies treebank to train the Bohnet parser for POS tagging and dependency parsing and exploited these trained models to parse the VMWE train sentences.

Table 5 shows the results of this experiment; the results for HU-GOLD are the same as the ones for Hungarian in the above tables. The results show that gold and parsed methods in our system can provide very comparable results. Interestingly, in both MWE-level and token-level results, the parsed method provides much higher precision but lower recall than the gold method.

For MWE types, LVCs are causing the main difference in the two systems. Many VPCs in the Hungarian data are single-token, so our system deals with them on the level of POS tagging, which is not affected by gold or parsed dependency trees.

Overall, both options achieved approximately the same results in the automatic VMWE recognition task.

<table><caption>Table 5: Gold and parsed results for Hungarian.</caption><thead><tr><th></th><th>HU-GOLD</th><th>HU-PARSED</th></tr></thead><tbody><tr><td>P-MWE</td><td>62.02</td><td>74.94</td></tr><tr><td>R-MWE</td><td>71.34</td><td>62.93</td></tr><tr><td>F-MWE</td><td>66.36</td><td>68.41</td></tr><tr><td>P-token</td><td>58.45</td><td>74.74</td></tr><tr><td>R-token</td><td>69.08</td><td>53.85</td></tr><tr><td>F-token</td><td>63.32</td><td>62.60</td></tr><tr><td>F-VPC-MWE</td><td>80.59</td><td>78.59</td></tr><tr><td>F-LVC-MWE</td><td>35.47</td><td>25.41</td></tr><tr><td>F-VPC-token</td><td>79.05</td><td>77.66</td></tr><tr><td>F-LVC-token</td><td>40.82</td><td>25.71</td></tr></tbody></table>