CoAM: Corpus of All-Type Multiword Expressions
Authors: Yusuke Ide, Joshua Tanner, Adam Nohejl, Jacob Hoffman, Justin Vasselli, Hidetaka Kamigaito, Taro Watanabe
Accepted at: acl25

CoAM: Corpus of All-Type Multiword Expressions

Yusuke Ide1    Joshua Tanner2    Adam Nohejl1    Jacob Hoffman2    Justin Vasselli1
Hidetaka Kamigaito1    Taro Watanabe1
1Nara Institute of Science and Technology    2Resolve Research
{ide.yusuke.ja6, nohejl.adam.mt3, vasselli.justin_ray.vk4, kamigaito.h, taro}@is.naist.jp, {josh, jake}@rslv.dev

Abstract

Multiword expressions (MWEs) refer to idiomatic sequences of multiple words. MWE identification, i.e., detecting MWEs in text, can play a key role in downstream tasks such as machine translation, but existing datasets for the task are inconsistently annotated, limited to a single type of MWE, or limited in size.

To enable reliable and comprehensive evaluation, we created CoAM: Corpus of All-Type Multiword Expressions, a dataset of 1.3K sentences constructed through a multi-step process to enhance data quality consisting of human annotation, human review, and automated consistency checking. Additionally, for the first time in a dataset for MWE identification, CoAM’s MWEs are tagged with MWE types, such as NOUN and VERB, enabling fine-grained error analysis.1 Annotations for CoAM were collected using a new interface created with our interface generator, which allows easy and flexible annotation of MWEs in any form.2 Through experiments using CoAM, we find that a fine-tuned large language model outperforms MWEasWSD, which achieved the state-of-the-art performance on the DiMSUM dataset. Furthermore, analysis using our MWE type tagged data reveals that VERB MWEs are easier than NOUN MWEs to identify across approaches.

1 Introduction

Vocabulary plays a critical role in the comprehension of natural language. While often simplified as a collection of single words, vocabulary also includes idiomatic sequences known as multiword expressions (MWEs) (Baldwin and Kim, 2010), which form an important part of language knowledge (Jackendoff, 1995). We focus specifically on sequences whose meaning or grammatical structure cannot be derived from their constituent words, as they can impede text analysis or comprehension; that is, we exclude transparent collocations. For example, the bold sequences below constitute MWEs.

(a) ACL stands for Association for Computational Linguistics.

(b) He has been under the weather lately.

MWE identification (MWEI), the process of automatically tagging MWEs in text (Constant et al., 2017), is valuable for tasks where word-by-word text processing is insufficient, such as machine translation (Li et al., 2024) and lexical complexity assessment (Kochmar et al., 2020). Briakou et al. (2024) investigated step-by-step translation using a large language model (LLM) and demonstrated that identifying MWEs in the initial step improves the translation quality. Additionally, it is possible to build reading assistance systems to automatically gloss text by combining MWEI with definition modeling (Bevilacqua et al., 2020); as Huang et al. (2022) discusses, definition modeling will help laypeople comprehend specialized terminology, which includes many MWEs.

Despite the importance of MWEI, existing datasets for the task are inconsistently annotated, limited to a single type of MWE, or limited in size, as described in Section 2. This hinders reliable and comprehensive evaluations of MWEI systems.

In this paper, we introduce the Corpus of All-type Multiword expressions (CoAM), a dataset of 1.3K sentences for MWEI. “Types” refer to MWE categories assigned based on the part of speech of the MWE, with “all-type” signifying the inclusion of all such categories of MWEs. To ensure annotation quality, we assigned two annotators and a reviewer to each sentence and checked all annotations for consistency. Additionally, MWEs in the CoAM test set are tagged with their types, such as NOUN and VERB, facilitating fine-grained error analysis of MWEI systems. This enables us to address questions such as: “Are verbal MWEs—

1The dataset is available at https://huggingface.co/datasets/yusuke196/CoAM.
2The source code of the interface generator is available at https://github.com/Yusuke196/CAIGen.
the focus of the long-running PARSEME project (Savary et al., 2017)—more difficult to identify compared to other categories of MWEs?"

Annotations for CoAM were collected using a checkbox-based annotation interface created using our new interface generator, CAIGen: Checkbox-based Annotation Interface Generator. It allows easy and flexible annotation of MWEs in any form, including discontinuous MWEs such as pick . . . up in pick me up at the station, which are common in English and have historically been a challenge for MWEI systems (Rohanian et al., 2019).

Using CoAM, we evaluate two distinct MWEI approaches. The first approach, MWEasWSD (MaW, Tanner and Hoffman, 2023), combines a rule-based pipeline and a trainable bi-encoder model, achieving state-of-the-art performance on the DiMSUM dataset (Schneider et al., 2016). The second is LLM fine-tuning for MWEI, inspired by the effectiveness of similar approaches in the task of named entity recognition (NER, Zhou et al., 2024). We use CoAM to train and evaluate LLMs from the Llama (Dubey et al., 2024) and Qwen (Qwen Team, 2024) model families.

The results reveal that a fine-tuned Qwen model with 72B parameters greatly outperforms MaW, demonstrating the effectiveness of LLM fine-tuning. On the other hand, all approaches suffer from low recall (e.g., 50.7% for Qwen-72B). Further analysis shows that fine-tuned LLMs struggle more with detecting NOUN and CLAUSE MWEs than with detecting VERB MWEs. MWEs not contained in WordNet (Miller, 1995), e.g., real estate, were found particularly difficult to identify, presumably because they are less widely recognized as MWEs.

2 Related Work

Datasets Previous studies have presented several datasets for MWEI and idiom\(^3\) identification. The DiMSUM (Schneider et al., 2016) dataset consists of 5,799 sentences annotated with MWEs, including, but not limited to, verbal MWEs, noun MWEs, phatics, and multi-word (MW) proper nouns. Although DiMSUM has been used in multiple MWEI studies, e.g., Kirilin et al. (2016) and Liu et al. (2021), there exist inconsistencies in the annotation, which hinder proper evaluation. Tanner and Hoffman (2023) found that over 80% of the false positives of their system were actually caused by DiMSUM’s inconsistent annotation. Next, the PARSEME corpus (Savary et al., 2017; Walsh et al., 2018) is a high-quality dataset for the identification of verbal MWEs. It has been continuously updated, and the latest version (1.3) contains over 455,000 sentences across 26 languages. However, their focus is limited to verbal MWEs. Lastly, ID10M (Tedeschi et al., 2022) is an idiom detection dataset consisting of automatically created training data in 10 languages and a manually curated evaluation benchmark of four languages. Their evaluation dataset was created with the help of professional annotators, but it contains only 200 sentences per language. They used Wiktionary as the source of idioms, and MWEs not in Wiktionary were skipped in the annotation. Additionally, they did not annotate discontinuous idioms. Among non-English datasets for MWE identification, Sequoia Corpus (Candito et al., 2020), a French corpus annotated for MWEs and named entities, stands out for containing 3,440 MWEs.

Tagging Schemes ID10M uses the BIO scheme. DiMSUM uses the more flexible 6-tag scheme, which allows discontinuous MWEs, but cannot handle overlapping MWEs (see Appendix D.2 for an example of overlapping MWEs). In contrast, the parseme-tsv format (Savary et al., 2017) accepts MWEs in any form, which leads us to adopt an equivalent data format.

Other Tasks Whereas these studies addressed MWE/idiom identification as a sequence tagging task, others worked on a related but different task, idiom usage recognition. It is a binary classification of word sequences in context as idiomatic/figurative or literal. The detection task in Muzny and Zettlemoyer (2013), SemEval-2013 Task 5b (Korkontzelos et al., 2013), SemEval-2022 Task 2 Subtask A (Tayyar Madabushi et al., 2022), and the MAGPIE corpus (Haagsma et al., 2020) are set up for this task. Our task setting is more realistic and challenging than these, requiring systems to identify all MWEs within a given sentence.

3 Task Formulation

We formulate the MWEI task as token-level sequence tagging, where the inputs are tokenized sentences and each token can belong to multiple MWEs. Given the token sequence of each sentence, \( x_1, \ldots, x_n \), the task is to output a list of

\footnotetext{3 According to Tedeschi et al. (2022), idioms are a subset of MWEs.}
MWEs where each MWE is represented as a list of token indices. The \( i \)-th MWE in the sentence is represented with \([idx(t_{i,1}), idx(t_{i,2}), \ldots]\) where \( t_{i,j} \) is the \( j \)-th token of the MWE and \( idx(t) \) is \( t \)'s index. Our annotation interface allows annotations in this scheme, as illustrated in Figure 1.

4 Construction of CoAM

4.1 Data Selection and Preprocessing

In selecting the sources of our dataset, we prioritized sources aimed at general audiences so that they are in standard English and mostly free of grammatical errors. We also included both written text and transcribed speech. Consequently, we utilized the following four data sources (see Appendix A.1 for their details). News is news text written by professional writers sourced from EMM NewsBrief, which was introduced by Glavaš and Štajner (2013).\footnote{The dataset also contains data from WikiNews and Wikipedia, but we preferred EMM NewsBrief, which contains more MWEs according to our preliminary analysis based on the annotation by Kochmar et al. (2020).} Commentary is commentaries on news from the WMT23 Shared Task monolingual training data (Kocmi et al., 2023). TED is a collection of TED talk transcriptions from two sources. (1) NAIST is the dataset by Neubig et al. (2014); (2) IWSLT is the subset of IWSLT 2017 Shared Task (Cettolo et al., 2017). UD is a collection of single sentences sourced from weblogs, reviews, question-answers, newsgroups, and emails in the English Web Treebank. It is part of the Universal Dependencies (Zeman et al., 2018) and the English PARSEME corpus (Walsh et al., 2018). We did not include any sentences from UD in the test set, because UD contains user-generated content with frequent grammatical errors. All the other sources were used both for training and test splits.

For all sources, we took the first 10 (or all, when there are less than 10) sentences from each article or talk and presented them in original order. Each sentence was tokenized using spaCy (we use version 3.7.1 and the en_core_web_trf model throughout this paper), with the exception of the UD sentences, which were already tokenized. Note that what we call words are tokens given in this manner.

4.2 Annotation Guidelines

The construction of a reliable dataset requires clear guidelines for annotators in order to minimize misannotation. However, we found no such guidelines for all-type MWE annotation, which led us to create new guidelines.

We define MWEs as idiomatic sequences that satisfy the following three conditions, based on the definition by the often-cited Baldwin and Kim (2010) and the PARSEME annotation guidelines.\footnote{https://parsemefr.lis-lab.fr/parseme-st-guidelines/1.3/}

(a) **It consists of at least two words that are always realized by the same lexemes.** This condition means that *his in put yourself in his shoes* is not part of the MWE, as other words like *Michael’s* can replace it.

(b) **It displays semantic, lexical, or syntactic idiomaticity.** Semantic idiomaticity occurs when the meaning of an expression cannot be derived from its constituent words. It is the most important type of idiomaticity because it accounts for the majority of MWEs being classified as idiomatic. See additional discussion on semantic idiomaticity in relation to non-compositionality in Appendix A.2. Note that transparent collocations such as *stuck at* are not MWEs in our definition because they are not semantically, lexically, or syntactically idiomatic. The meaning or grammatical structure of transparent collocations can be understood by their constituents, and thus they can be processed word-by-word.

(c) **It is not a proper noun, i.e., a specific name of a person, facility, and so on.** Previous MWE studies either classified MW proper nouns as MWEs (Schneider et al., 2016) or excluded them as non-idiomatic (Tayyar Madabushi et al., 2022). We opted for the latter, as proper nouns are linked to encyclopedic knowledge rather than lexicographic knowledge (Navigli and Ponzetto, 2012), to which typical MWEs belong to.

The full MWE definition is given in Appendix D. The guidelines were updated whenever an issue was found, e.g., an ambiguous description.

4.3 Annotation Interface

For flexible and efficient annotation, we developed a novel annotation interface generator, CAIGen. CAIGen builds a checkbox-based interface in Google Sheets\footnote{https://developers.google.com/sheets}, allowing annotators to perform annotations of MWEs and other kinds of spans simply by checking checkboxes as shown in Figure 1. In the interface generation process, for each sentence, CAIGen first writes the sentence ID, the
Figure 1: An example sentence presented in our interface by CAIGen. Checks in the checkboxes are instantly reflected in the bordered zone at the upper right, showing which MWEs are currently marked. Note that the number of rows is reduced here for brevity; the actual interfaces we used have nine rows per sentence.

<table>
  <tr>
    <th></th>
    <th>Flexible annotation</th>
    <th>Easy collaboration</th>
    <th>Easy-to-use interface</th>
    <th>Customizable interface</th>
  </tr>
  <tr>
    <td>brat<sup>7</sup></td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
    <td>x</td>
  </tr>
  <tr>
    <td>FLAT<sup>8</sup></td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparison of annotation tools.

sentence itself, and a bordered box to show the result of annotations. Then, it arranges each token from left to right, wrapping the line when its length hits a pre-set limit.

Its main advantages over other comparable annotation tools are summarized in Table 1. The first tool, brat<sup>7</sup> (Stenetorp et al., 2012), has been used for annotation of a wide range of tasks such as sentiment analysis (Pontiki et al., 2016) and NER (Tabassum et al., 2020). FLAT<sup>8</sup> was used for the annotation of the PARSEME corpus (Savary et al., 2017). The advantages of CAIGen over these are as follows. First, the generated interface supports annotations of spans of any form, including discontinuous or overlapping spans (see Appendix D.2 for examples), making it more flexible than the previous tools. Second, CAIGen alleviates researchers’ overhead of managing servers and annotator accounts by delegating to Google, while the other tools require either (1) distributing data slices to each annotator and having them run applications locally or (2) running and managing the application on a remote server. Third, CAIGen provides an easy-to-use spreadsheet-based interface, which is simple and familiar to technical and non-technical annotators alike. Lastly, our interface is highly customizable because CAIGen builds it using Google Apps Script, a well-documented programming language based on JavaScript; for example, additional annotations of span types could be easily collected.

4.4 Annotation

The annotation was done by two annotators hired through a company and five authors. The hired annotators comprise one native and one non-native English speaker, both with at least six years of experience as translators. They received feedback from the authors after annotating the first 50 or so sentences to correct misunderstandings about the guidelines. We paid 154 JPY (roughly 1 USD) per sentence as a reward to the company. Meanwhile, the author annotators consist of three native and two non-native speakers, all with a language-related degree and sufficient English proficiency to perform the task. For reliable annotation, we assigned each sentence to two annotators (one hired annotator and one author), ensuring that at least one was a native English speaker, as the task requires a rich English vocabulary.

All annotators were instructed to carefully read the guidelines and perform annotation using the checkbox-based interface. In the authors’ annotation, we marked unclear sentences to remove them later because it is hard to determine whether a span is an MWE in such sentences.

4.5 Inter-Annotator Agreement

With two annotators being assigned to each sentence, we measured the inter-annotator agreement (IAA) by the MWE-based (exact-match) F1 score. The IAA was found to be only 37.3%, which could be attributed to annotators failing to annotate valid MWEs. We thus suggest that MWE datasets should be constructed with at least two independent annotators to reduce missed MWEs.

Nonetheless, it should be noted that this IAA is comparable with other MWEI datasets, and that it does not reflect the quality of CoAM. Our IAA is computed in a strict way, which we chose for its
clear interpretability. CoAM’s IAA is comparable to those of other datasets when the same, more lenient methods are used. PARSEME 1.1 (Ramisch et al., 2018) computes F1 for exact span matches as we do, but they only report “the highest scores among all possible annotator pairs” (see Table 1, ibid.). The highest score for English PARSEME is 52.9%, similar to the highest score computed in the same way for CoAM: 52.2%. Meanwhile, DiMSUM was annotated by a single annotator, and thus no actual agreement was reported. Although Schneider et al. (2016) estimates DiMSUM’s IAA to range from 60% to 75%, this is based on a very small subset (66 sentences) reannotated by the first author and computed as a more lenient F1 score based on partial span matches. Thus, while our reported IAA value is low, we emphasize that our IAA is in line with comparable MWEI datasets. Additionally, we addressed the annotation disagreements through the combination of human review and consistency checking as described below.

4.6 Review
To solve the disagreement between the two annotators and correct any other problematic annotations, two authors—both native English speakers—reviewed all the annotations. We presented them with the annotations in a special interface for reviewing, where the tags given by only one annotator were highlighted, and we asked them to mark inappropriate tags to be deleted and newly found MWEs to be added. When the review was complete, we updated our dataset according to the marks added by reviewers.

4.7 Consistency Check
One of the primary issues with the annotations in the DiMSUM dataset is their inconsistency—that is, a number of MWEs are annotated in one location and not another, despite equivalent constituents and semantics being present in both places. For example, see how give_try is labeled in only one of the sentences in Table 3.

In order to quantify this issue in both DiMSUM and CoAM—and to eliminate it from CoAM—we used a partially automated approach to find inconsistencies in both the entirety of CoAM and 1.3K sentences (the size of CoAM) randomly sampled from DiMSUM. We started by initializing an empty set \( M \), then iterated through all given sentences and added all labeled MWEs from them to \( M \). Next, we used a simple rule-based MWEI pipeline, repurposed from Tanner and Hoffman (2023), to find constituent groups in other sentences that could correspond to an MWE in \( M \) but were not already labeled. Finally, an author and native speaker of English reviewed each of these candidate constituent groups to see if they are semantically equivalent to already labeled instances of this MWE—that is, to see if they represent an inconsistency.

We found 118 instances of inconsistencies like this in the random sample of DiMSUM and 147 in CoAM,\footnote{When counting inconsistencies, we always consider positive labels correct, and negative labels inconsistent.} reaffirming the difficulty involved in producing consistent MWE annotations. However, we then added the missing labels to all MWEs found in this way, eliminating these inconsistencies from the final CoAM data.

4.8 MWE Type Tagging
To enable fine-grained error analysis, we automatically tagged all the MWEs in CoAM with MWE types. We then manually corrected the tags in the test set, as described below, to enable a precise analysis of the results by MWE types.

Inspired by the classification by Schneider et al. (2014b), we group MWEs into five types: NOUN, VERB, MODIFIER/CONNECTIVE (MOD/CONN),\footnote{We include adjectival, adverbial, adpositional, and connective MWEs in the category of MOD/CONN to create a category with a sufficient data size.} CLAUSE, and OTHER. The details are described in Table 2. We automatically tag MWE types in each sentence based on the dependency structure. Dependency parses were provided in the data source for UD and obtained using spaCy for News, Commentary, and TED. For each MWE \( m = (w_1, \ldots, w_{|m|}) \), we look for its syntactic head, namely, try to find \( w^* \) that has all the other words in \( m \) as its descendants. If \( m \) has such \( w^* \), we determine its type based on the PoS of \( w^* \); for example, we tag \( m \) as VERB when \( w^* \) is a verb. If \( m \) does not have such \( w^* \), we tag \( m \) as OTHER. For the details, see the algorithm in Appendix A.3.

We then manually checked all 384 MWEs in the test set and corrected the errors in them. Two authors who are native English speakers conducted this, identifying and correcting 30 errors. The error rate of automatic tagging was therefore 7.8%. The most common errors were MOD/CONN automatically tagged as VERB (e.g., according to) and OTHER (e.g., not only) constituting 3.3% and 3.1% of the test data, respectively. We expect this to be
<table>
  <tr>
    <th>Type</th>
    <th>PoS of Head</th>
    <th>Description</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>NOUN</td>
    <td>NOUS, PRON, PROPN</td>
    <td>Noun MWEs or compounds.</td>
    <td><i>the middle of nowhere, red tape</i></td>
  </tr>
  <tr>
    <td>VERB</td>
    <td>VERB, AUX</td>
    <td>Verbal MWEs.</td>
    <td><i>stand for, pick up, be the case</i></td>
  </tr>
  <tr>
    <td>Mod/CONN</td>
    <td>ADJ, ADV, ADP, CCONJ, SCONJ</td>
    <td>Adjectival, adverbial, adpositional, or connective MWEs.</td>
    <td><i>under the weather, of course, in spite of</i></td>
  </tr>
  <tr>
    <td>CLAUSE</td>
    <td>VERB, AUX</td>
    <td>MWEs containing (1) a verb or auxiliary verb and (2) its nominal subject, such as phatics and proverbs.</td>
    <td><i>you’re welcome, the early bird gets the worm, when it comes to</i></td>
  </tr>
  <tr>
    <td>OTHER</td>
    <td>Any</td>
    <td>MWEs whose head is not contained in them or whose PoS is none of the PoS above.</td>
    <td><i>and so on, oh man</i></td>
  </tr>
</table>

Table 2: MWE types in CoAM. The PoS are denoted with UPOS tags. Underlines denote the head of an MWE.

<table>
  <tr>
    <th>Sentence</th>
    <th>Marked</th>
  </tr>
  <tr>
    <td>Having never booked train tickets online before thought I would give it a try and was very surprised at how much I saved.</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Would recomend giving this a try.</td>
    <td>✗</td>
  </tr>
</table>

Table 3: Inconsistent annotation of give_try. Note that the second sentence is reproduced as in DiMSUM, including apparent typos.

representative of the errors in the type tags automatically assigned to the training set. Note, however, that we used only the manually corrected tags on the test set for result analysis in Section 6.2.

4.9 Statistics

Table 4 shows dataset statistics. CoAM has more than 1.3K sentences. The MWE density is 6–7% in both training and test sets. This proportion is much lower than that of DiMSUM (Schneider et al., 2016)—13% overall—most likely because DiMSUM includes proper noun phrases in MWEs. Among the five MWE types, VERB and NOUN are the most frequent ones across data sources. See Appendix A.4 for additional analysis regarding MWE continuity.

5 MWEI Approaches

We use CoAM to evaluate the following MWEI approaches.

5.1 MWEasWSD

MWEasWSD (MaW) is an approach that uses (1) an MWE lexicon—WordNet (Miller, 1995)—and a rule-based pipeline to identify MWE candidates and (2) a trainable model to filter MWE candidates (Tanner and Hoffman, 2023). It achieved state-of-the-art performance on the DiMSUM dataset. They published four of their filtering models, among which we use the bi-encoder (BiEnc) and DCA poly-encoder (DCA). Both models are based on BERT (Devlin et al., 2019), specifically bert-base-uncased. They have been trained with SemCor (Miller et al., 1993), and we further fine-tune each model with the CoAM training set. We also run MaW with the rule-based pipeline only, i.e., without a filtering model.

5.2 LLM Fine-Tuning

Recent studies have shown the effectiveness of LLM fine-tuning for a wide range of tasks, such as NER (Zhou et al., 2024) and grammatical error correction (Kaneko and Okazaki, 2023). Framing their task as sequence transduction, they achieved high performance by providing LLMs with prompts that included instructions on the task.

Inspired by their success, we perform LLM fine-tuning, where the inputs to LLMs are instructions for MWEI (a summary of the annotation guidelines) followed by formatted tokens in a sentence.

6 Experiments

6.1 Setup

LLM Fine-Tuning We use four instruction-tuned LLMs that are available on Hugging Face Hub: Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct (Dubey et al., 2024), Qwen-2.5-7B-Instruct, and Qwen-2.5-72B-Instruct (Qwen Team, 2024). We abbreviate them, e.g., to Llama-8B, omitting the versions. For efficient training and inference, we use QLoRA (Dettmers et al., 2023), performing 4-bit NormalFloat quantization and double quantization. Other hyperparameters are described in Appendix B.1. The computational budgets are in Appendix C.

To find a performant input-output format, we conduct preliminary experiments with three for-
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3"> </th>
    <th rowspan="2">MWE Density (%)</th>
    <th colspan="5">MWE Type Proportion (%)</th>
  </tr>
  <tr>
    <th>Sentences</th>
    <th>Words</th>
    <th>MWEs</th>
    <th>NOUN</th>
    <th>VERB</th>
    <th>MOD/CONN</th>
    <th>CLAUSE</th>
    <th>OTHER</th>
  </tr>
  <tr>
    <td>News</td>
    <td>360</td>
    <td>9,328</td>
    <td>230</td>
    <td>5.5</td>
    <td>32.2</td>
    <td>44.3</td>
    <td>23.0</td>
    <td>0.0</td>
    <td>0.4</td>
  </tr>
  <tr>
    <td>Commentary</td>
    <td>357</td>
    <td>9,310</td>
    <td>269</td>
    <td>7.0</td>
    <td>30.5</td>
    <td>29.7</td>
    <td>33.1</td>
    <td>1.5</td>
    <td>5.2</td>
  </tr>
  <tr>
    <td>TED</td>
    <td>299</td>
    <td>6,592</td>
    <td>210</td>
    <td>7.1</td>
    <td>35.2</td>
    <td>38.6</td>
    <td>21.4</td>
    <td>3.8</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>UD</td>
    <td>285</td>
    <td>5,001</td>
    <td>158</td>
    <td>7.2</td>
    <td>42.4</td>
    <td>39.2</td>
    <td>12.7</td>
    <td>1.3</td>
    <td>4.4</td>
  </tr>
  <tr>
    <td>Train</td>
    <td>780</td>
    <td>16,817</td>
    <td>486</td>
    <td>6.7</td>
    <td>36.2</td>
    <td>38.3</td>
    <td>19.8</td>
    <td>1.4</td>
    <td>4.3</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>521</td>
    <td>13,414</td>
    <td>381</td>
    <td>6.4</td>
    <td>31.8</td>
    <td>36.5</td>
    <td>29.1</td>
    <td>1.8</td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>1,301</td>
    <td>30,231</td>
    <td>867</td>
    <td>6.6</td>
    <td>34.3</td>
    <td>37.5</td>
    <td>23.9</td>
    <td>1.6</td>
    <td>2.8</td>
  </tr>
</table>

Table 4: Statistics of CoAM. MWE density is the percentage of words in MWEs. The test set comprises part of News, Commentary, and TED, while the training set comprises the rest of the data.

<table>
  <tr>
    <th>Role</th>
    <th>Message</th>
  </tr>
  <tr>
    <td>System</td>
    <td>You are a helpful system to identify multiple-word expressions (MWEs).<br>Identify all the MWEs in the given sentence, and output their surface forms and the indices of their components.\n[MWE_DEFINITION]\n\nEach sentence is given as a string of words delimited by '\n'. Respond in TSV format, where the first and second columns contain words and MWE tags, respectively. The MWE tag should be a string of MWE identifiers. When a word belongs to multiple MWEs, the tag should be a concatenation of their numbers delimited by semicolons.\n\nSentence:\nACL\nstands\n</td>
  </tr>
  <tr>
    <td>User</td>
    <td></td>
  </tr>
</table>

Table 5: Example prompt for fine-tuning, based on the tsv_to_tsv format. In our main experiments, [MWE_DEFINITION] will be filled with the long MWE definition described in Appendix B.1.

mats, adjusting the prompt for each format. These experiments are detailed in Appendix B.1. We find tsv_to_tsv (see Table 5) is the only format the models can comply with and thus adopt it.

All prompts contain a placeholder for a definition of MWEs to clarify what sequences should be marked. We report scores using what we refer to as the long definition in Table 6. For scores using a shorter definition, see the ablation study in Appendix B.1.

Evaluation Metrics We use MWE-based precision, recall, and F1 score (Savary et al., 2023). See Appendix B.2 for their exact definitions.

6.2 Results and Analysis

In Table 6a, the left three columns show the overall scores. Fine-tuned Qwen-72B achieves the best F1, surpassing the highest F1 by Rule+DCA of MaW. Additionally, fine-tuned Qwen-72B outperforms all MaW systems in precision and recall. This indicates the effectiveness of fine-tuning LLMs for MWEI—particularly LLMs with a large number of parameters. One explanation for this could be that knowledge about MWEs was acquired by LLMs through pre-training, and we can harness this knowledge for MWEI by fine-tuning.

Meanwhile, all systems, except for the rule-based one, suffer from low recall. Even the best system, fine-tuned Qwen-72B, achieves a recall of only 52.8%, missing almost half of the gold MWEs.

Analysis of Recall Given the low recall across all systems, we conduct further analysis to determine which MWEs the models particularly struggle to identify.

Recall by MWE Type in Table 6a shows that CLAUSE and NOUN MWEs tend to be more difficult to identify than MOD/CONN or VERB MWEs across models.

In Table 6b, the first two columns analyze how much recall changes depending on whether the MWE is seen or unseen, where an MWE in the test set is considered unseen if the multi-set of lemmas of its constituents was never annotated in the training set. We find that seen MWEs are easier to identify than unseen ones across models and systems, which resembles the results of the PARSEME shared task 1.2 (Ramisch et al., 2020).

Recall by Continuity in Table 6b shows that discontinuous MWEs are, across models and systems,
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">F1</th>
    <th colspan="4">Recall by MWE Type</th>
  </tr>
  <tr>
    <th>P</th>
    <th>R</th>
    <th>NOUN (121)</th>
    <th>VERB (139)</th>
    <th>MOD/CONN (111)</th>
    <th>CLAUSE (7)</th>
  </tr>
  <tr>
    <td rowspan="4">FT</td>
    <td>Llama-8B</td>
    <td>24.9±3.7</td>
    <td><b>92.0±2.0</b></td>
    <td>14.4±2.4</td>
    <td>3.3±0.0</td>
    <td>19.4±2.9</td>
    <td>21.6±4.8</td>
    <td>0.0±0.0</td>
  </tr>
  <tr>
    <td>Llama-70B</td>
    <td>38.3±5.2</td>
    <td>69.0±1.7</td>
    <td>26.8±5.1</td>
    <td>19.0±5.2</td>
    <td>39.6±7.5</td>
    <td>21.6±2.7</td>
    <td>0.0±0.0</td>
  </tr>
  <tr>
    <td>Qwen-7B</td>
    <td>48.1±1.0</td>
    <td>60.9±0.8</td>
    <td>39.7±1.0</td>
    <td>28.7±1.3</td>
    <td>52.0±0.8</td>
    <td>39.6±0.9</td>
    <td>0.0±0.0</td>
  </tr>
  <tr>
    <td>Qwen-72B</td>
    <td><b>57.8±1.8</b></td>
    <td>63.8±2.2</td>
    <td><b>52.8±2.1</b></td>
    <td><b>42.1±4.4</b></td>
    <td><b>59.7±1.4</b></td>
    <td><b>57.7±1.8</b></td>
    <td><b>28.6±0.0</b></td>
  </tr>
  <tr>
    <td rowspan="3">MaW</td>
    <td>Rule</td>
    <td>32.4</td>
    <td>27.9</td>
    <td>38.6</td>
    <td><b>33.1</b></td>
    <td><b>45.3</b></td>
    <td><b>39.6</b></td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Rule+BiEnc</td>
    <td>41.4±0.3</td>
    <td>47.9±0.8</td>
    <td>36.5±0.3</td>
    <td>28.9±0.0</td>
    <td>45.3±0.0</td>
    <td>36.9±0.9</td>
    <td>0.0±0.0</td>
  </tr>
  <tr>
    <td>Rule+DCA</td>
    <td>41.9±0.2</td>
    <td>49.0±0.5</td>
    <td>36.7±0.5</td>
    <td>29.8±0.0</td>
    <td>45.3±0.7</td>
    <td>36.6±1.0</td>
    <td>0.0±0.0</td>
  </tr>
</table>

(a)

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="2">Recall by Seen/Unseen</th>
    <th colspan="2">Recall by Continuity</th>
    <th colspan="2">Recall by In WN or Not</th>
  </tr>
  <tr>
    <th>Seen (141)</th>
    <th>Unseen (240)</th>
    <th>Continuous (338)</th>
    <th>Discontinuous (43)</th>
    <th>True (161)</th>
    <th>False (220)</th>
  </tr>
  <tr>
    <td rowspan="4">FT</td>
    <td>Llama-8B</td>
    <td>29.6±5.0</td>
    <td>5.6±0.9</td>
    <td>16.0±2.7</td>
    <td>2.3±0.0</td>
    <td>23.4±3.2</td>
    <td>7.9±1.8</td>
  </tr>
  <tr>
    <td>Llama-70B</td>
    <td>37.1±4.5</td>
    <td>20.7±5.4</td>
    <td>29.0±4.9</td>
    <td>9.3±6.2</td>
    <td>43.9±6.3</td>
    <td>14.2±4.1</td>
  </tr>
  <tr>
    <td>Qwen-7B</td>
    <td>47.3±0.4</td>
    <td>35.3±1.3</td>
    <td>43.9±1.1</td>
    <td>7.0±0.0</td>
    <td>53.8±0.9</td>
    <td>29.4±1.1</td>
  </tr>
  <tr>
    <td>Qwen-72B</td>
    <td><b>61.9±0.8</b></td>
    <td><b>47.4±3.8</b></td>
    <td><b>57.3±1.8</b></td>
    <td><b>17.1±4.8</b></td>
    <td><b>62.7±2.8</b></td>
    <td><b>45.5±2.8</b></td>
  </tr>
  <tr>
    <td rowspan="3">MaW</td>
    <td>Rule</td>
    <td>46.8</td>
    <td>33.8</td>
    <td>41.7</td>
    <td>14.0</td>
    <td><b>91.3</b></td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Rule+BiEnc</td>
    <td>45.2±0.4</td>
    <td>31.4±0.2</td>
    <td>39.6±0.3</td>
    <td>11.6±0.0</td>
    <td>86.3±0.6</td>
    <td>0.0±0.0</td>
  </tr>
  <tr>
    <td>Rule+DCA</td>
    <td>44.0±1.2</td>
    <td>32.4±0.2</td>
    <td>40.3±0.5</td>
    <td>7.8±1.3</td>
    <td>86.7±1.3</td>
    <td>0.0±0.0</td>
  </tr>
</table>

(b)

Table 6: Results by MWEI system and metric/MWE category, as mean percentage scores of three runs with random training seeds. The numbers in parentheses are the MWE counts. ± denotes standard deviation. The bold font denotes the highest score. Rule stands for the rule-based baseline, and WN for WordNet.

<table>
  <tr>
    <th>Result</th>
    <th>MWE</th>
    <th>Context</th>
    <th>Note</th>
  </tr>
  <tr>
    <td>TP</td>
    <td><i>fire up</i></td>
    <td><i>The allegations have fired up the opposition.</i></td>
    <td><b>VERB</b> in WordNet</td>
  </tr>
  <tr>
    <td>TP</td>
    <td><i>at least</i></td>
    <td>… since <i>at least</i> the 1950s.</td>
    <td><b>MOD/CONN</b> in WordNet</td>
  </tr>
  <tr>
    <td>FN</td>
    <td><i>in . . . hands</i></td>
    <td>… concentration of power <i>in his own hands</i>.</td>
    <td><b>MOD/CONN</b> in WordNet</td>
  </tr>
  <tr>
    <td>FN</td>
    <td><i>real estate</i></td>
    <td>… park their toxic <i>real estate assets</i> …</td>
    <td><b>NOUN</b> not in WordNet</td>
  </tr>
  <tr>
    <td>FN</td>
    <td><i>you know</i></td>
    <td><b>You know</b>, it’s very old …</td>
    <td><b>CLAUSE</b> not in WordNet</td>
  </tr>
</table>

Table 7: Examples of true positives (TPs) and false negatives (FNs), i.e., MWEs identified/missed in all three runs, of fine-tuned Qwen-72B.

far more difficult to identify than continuous ones, indicating that discontinuous MWEs remain a key challenge and require further attention.

Recall by In WN or Not in Table 6b reveals that MWEs not contained in WordNet are much more difficult to identify than those in WordNet. It is natural that MaW systems cannot identify the MWEs not in WordNet, as their rule-based pipeline cannot detect these candidates, but interestingly we find that fine-tuned LLMs also struggle to identify MWEs not in WordNet. We hypothesize that this is caused by MWEs in WordNet being more widely recognized as MWEs or idioms and that this recognition is reflected in the training data of the LLMs, enhancing their ability to identify these MWEs.

Table 7 shows examples of correctly identified MWEs and missed MWEs by fine-tuned Qwen-72B. VERB MWEs in WordNet like <i>fire up</i> are relatively easy to identify, with a recall of 79.0% achieved by the model. Meanwhile, the discontinuous MWE, <i>in . . . hands</i>, was not correctly identified. Additionally, the NOUN MWE <i>real estate</i> was not identified; the aforementioned hypothesis could explain this, as <i>real estate</i> is not in WordNet. Accurately identifying those MWEs could be a challenge for future studies.

Ablation Study on LLM Fine-Tuning Given the relatively high performance of fine-tuned Qwen-72B, we investigate how much fine-tuning (FT) contributes to the performance by comparing FT to zero-shot learning (ZSL) and few-shot learning (FSL). In ZSL, we provide the models with the same prompt as FT. In FSL, we sample 5 pairs of
<table>
  <tr>
    <th></th>
    <th>Llama-70B</th>
    <th>Qwen-72B</th>
  </tr>
  <tr>
    <th>FT</th>
    <td>38.3\pm5.2</td>
    <td>57.8\pm1.8</td>
  </tr>
  <tr>
    <th>FSL</th>
    <td>6.1\pm0.0</td>
    <td>12.9\pm0.0</td>
  </tr>
  <tr>
    <th>ZSL</th>
    <td>3.9</td>
    <td>1.4</td>
  </tr>
</table>

Table 8: Ablation results by MWEI method and model, as mean percentage F1 scores of three runs (for FT the randomness arises from training, and for FSL from example selection). \( \pm \) denotes standard deviation.

input (sentence) and output (gold MWE set) from the training set, convert them into the tsv_to_tsv format, and include them in the prompt as examples. In the sampling process, we ensure that the models have a sufficient number of examples to learn the task and format by repeating random sampling until at least two of the example sentences contain one or more MWEs.

Table 8 shows the results. FT greatly outperforms ZSL and FSL, demonstrating the effectiveness of fine-tuning on the CoAM training set.

Comparison considering computational efficiency As shown in Table 6a, fine-tuned Qwen-72B outperforms the best MaW system in all of the F1 score, precision, and recall. However, the LLM takes substantial memory and compute, and MaW is more efficient in this regard. MaW’s encoder model (bert-base-uncased) has only 110M parameters, taking approximately \( 2 \times 110M \times 4 = 880 \)M bytes in total (2 is the number of encoders). 4-bit quantized Qwen-72B takes \( 72,000M \times 0.5 = 36,000 \)M bytes, taking roughly 40 times more space than MaW. On the other hand, the performance of 4-bit quantized Qwen-7B, which takes roughly 4 times more space, is comparable with Rule+DCA.

7 Conclusion

In this paper, we constructed CoAM, a high-quality dataset of 1.3K sentences for MWEI covering all types of MWEs. Using a combination of human review and automated consistency checking, we addressed consistency issues that have been a problem for previous MWE datasets, enabling more accurate evaluation results for future work in MWEI. We used CoAM to evaluate two MWEI approaches: MaW and LLM fine-tuning. Our largest fine-tuned LLM performed the best, but all systems suffered from low recall. Consequently, we argue that MWEI overall remains a challenging task.

Limitations

Dataset Size Because the objective of this work was to maximize dataset quality, we invested heavily in efforts to improve data quality, employing multiple annotators per sentence, manual review, and consistency checking. This focus on quality over quantity resulted in a slightly smaller dataset size (1.3K sentences) compared to some previous works like DiMSUM and PARSEME. However, the evaluation set of CoAM has many more sentences than the gold data of ID10M (Tedeschi et al., 2022), and is large enough for our purpose of reliably evaluating MWEI systems.

CAIGen We observed a usability issue with our annotation interface, CAIGen. It requires annotators to use separate table rows for each annotated MWE. Because annotators may forget to use separate rows, which happened in the construction of CoAM, researchers should urge them to confirm their annotations are as intended after finishing each sentence.

MWEI Approaches In this study, we experimented with an existing approach, MaW, as well as LLM fine-tuning. Future work could explore other previously proposed methods, such as BERT-based tagging with BIO-like labels, as introduced by Taslimipoor et al. (2020).

Ethical Considerations

All sources of CoAM are permitted at minimum for use in research, as described in Appendix A.1. CoAM itself will be released under the condition that the users do not publish the data on the open web to prevent its leakage to the training data of future models. All annotators of CoAM consented to the publication of their annotations. Throughout the dataset construction processes, we found no content harmful to the environment, specific groups of people, or privacy.

Acknowledgments

We are grateful for the insightful comments by Yuki Arase, Shohei Higashiyama, and anonymous reviewers. To polish the content of this paper, we used ChatGPT11 as a writing assistant tool. Financially, this work was supported by JST SPRING Grant Number JPMJSP2140.

11https://chatgpt.com
References

Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. In Handbook of Natural Language Processing, pages 267–292, Boca Raton. CRC Press.

Michele Bevilacqua, Marco Maru, and Roberto Navigli. 2020. Generationary or “how we went beyond word sense inventories and learned to gloss”. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7207–7221, Online. Association for Computational Linguistics.

Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English web treebank. LDC2012T13.

Eleftheria Briakou, Jiaming Luo, Colin Cherry, and Markus Freitag. 2024. Translating step-by-step: Decomposing the translation process for improved translation quality of long-form texts. In Proceedings of the Ninth Conference on Machine Translation, pages 1301–1317, Miami, Florida, USA. Association for Computational Linguistics.

Marie Candito, Mathieu Constant, Carlos Ramisch, Agata Savary, Bruno Guillaume, Yannick Parmentier, and Silvio Ricardo Cordeiro. 2020. A french corpus annotated for multiword expressions and named entities. Journal of Language Modelling, 8(2).

Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Conference on Spoken Language Translation, pages 2–14, Tokyo, Japan. International Workshop on Spoken Language Translation.

Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit3: Web inventory of transcribed and translated talks. In EAMT, pages 261–268.

Mathieu Constant, Gülşen Eryiğit, Johanna Monti, Loneke Van Der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todorascu. 2017. Multiword expression processing: A survey. Computational Linguistics, 43(4):837–892.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauluri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Goran Glavaš and Sanja Štajner. 2013. Event-centered simplification of news stories. In Proceedings of the Student Research Workshop associated with RANLP 2013, pages 71–78, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.

Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020. MAGPIE: A large corpus of potentially idiomatic expressions. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 279–287, Marseille, France. European Language Resources Association.

Jie Huang, Hanyin Shao, Kevin Chen-Chuan Chang, Jinjin Xiong, and Wen-mei Hwu. 2022. Understanding jargon: Combining extraction and generation for definition modeling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3994–4004, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Ray Jackendoff. 1995. The Boundaries of Lexicon. In M. Everaert, E.J. van der Linden, A. Schenk, R. Schreuder, and R. Schreuder, editors, Idioms: Structural and Psychological Perspectives, pages 133–166. Erlbaum, NJ.

Masahiro Kaneko and Naoki Okazaki. 2023. Reducing sequence length by predicting edit spans with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10017–10029, Singapore. Association for Computational Linguistics.

Angelika Kirilin, Felix Krauss, and Yannick Versley. 2016. ICL-HD at SemEval-2016 task 10: Improving the detection of minimal semantic units and their meanings with an ontology and word embeddings. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 937–945, San Diego, California. Association for Computational Linguistics.

Ekaterina Kochmar, Sian Gooding, and Matthew Shardlow. 2020. Detecting multiword expression type helps lexical complexity assessment. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4426–4435, Marseille, France. European Language Resources Association.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. 2023. Findings of the 2023
conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, pages 1–42, Singapore. Association for Computational Linguistics.

Ioannis Korkontzelos, Torsten Zesch, Fabio Massimo Zanzotto, and Chris Biemann. 2013. SemEval-2013 task 5: Evaluating phrasal semantics. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 39–47, Atlanta, Georgia, USA. Association for Computational Linguistics.

Shuang Li, Jiangjie Chen, Siyu Yuan, Xinyi Wu, Hao Yang, Shimin Tao, and Yanghua Xiao. 2024. Translate meanings, not just words: Idiomkb’s role in optimizing idiomatic translation with language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAII’24. AAAI Press.

Nelson F. Liu, Daniel Hershcovich, Michael Kranzlein, and Nathan Schneider. 2021. Lexical semantic recognition. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 49–56, Online. Association for Computational Linguistics.

George A. Miller. 1995. Wordnet: A lexical database for english. Communications of the ACM, 38(1):39–41.

George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21–24, 1993.

Grace Muzny and Luke Zettlemoyer. 2013. Automatic idiom identification in Wiktionary. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1417–1421, Seattle, Washington, USA. Association for Computational Linguistics.

Roberto Navigli and Simone Paolo Ponzetto. 2012. Balabelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250.

Graham Neubig, Katsuhiro Sudoh, Yusuke Oda, Kevin Duh, Hajime Tsukuda, and Masakki Nagata. 2014. The NAIST-NTT TED talk treebank. In Proceedings of the 11th International Workshop on Spoken Language Translation: Papers, pages 265–270, Lake Tahoe, California.

Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, Ion Androutsopoulos, Núria Bel, and Gülşen Eryiğit. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In International Workshop on Semantic Evaluation.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Carlos Ramisch, Silvio Ricardo Cordeiro, Agata Savary, Veronika Vincze, Verginica Barbu Mititelu, Archna Bhatia, Maja Buljan, Marie Candito, Polona Gantar, Voula Giouli, Tunga Güngör, Abdelati Hawwari, Uoxa Inurrieta, Jolanta Kovalevskaitė, Simon Krek, Timm Lichte, Chaya Liebeskind, Johanna Monti, Carla Parra Escartín, Behrang QasemiZadeh, Renata Ramisch, Nathan Schneider, Ivelina Stoyanova, Ashwini Vaidya, and Abigail Walsh. 2018. Edition 1.1 of the PARSEME shared task on automatic identification of verbal multiword expressions. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 222–240, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Carlos Ramisch, Agata Savary, Bruno Guillaume, Jakub Waszcuk, Marie Candito, Ashwini Vaidya, Verginica Barbu Mititelu, Archna Bhatia, Uoxa Inurrieta, Voula Giouli, Tunga Güngör, Menghan Jiang, Timm Lichte, Chaya Liebeskind, Johanna Monti, Renata Ramisch, Sara Stymne, Abigail Walsh, and Hongzhi Xu. 2020. Edition 1.2 of the PARSEME shared task on semi-supervised identification of verbal multiword expressions. In Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons, pages 107–118, online. Association for Computational Linguistics.

Omid Rohanian, Shiva Taslimipoor, Samaneh Kouchaki, Le An Ha, and Ruslan Mitkov. 2019. Bridging the gap: Attending to discontinuity in identification of multiword expressions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2692–2698, Minneapolis, Minnesota. Association for Computational Linguistics.

Ivan A. Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions : a pain in the neck for nlp.

Agata Savary, Cherifa Ben Khelil, Carlos Ramisch, Voula Giouli, Verginica Barbu Mititelu, Najet Hadji Mohamed, Cvetana Krstev, Chaya Liebeskind, Hongzhi Xu, Sara Stymne, Tunga Güngör, Thomas Pickard, Bruno Guillaume, Eduard Bejček, Archna Bhatia, Marie Candito, Polona Gantar, Uoxa Inurrieta, Albert Gatt, Jolanta Kovalevskaitė, Timm Lichte, Nikola Ljubešić, Johanna Monti, Carla Parra Escartín, Mehrnoush Shamstard, Ivelina Stoyanova, Veronika Vincze, and Abigail Walsh. 2023. PARSEME corpus release 1.3. In Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023), pages 24–35, Dubrovnik, Croatia. Association for Computational Linguistics.

Agata Savary, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova, and Antoine Doucet. 2017. The
PARSEME shared task on automatic identification of verbal multiword expressions. In Proceedings of the 13th Workshop on Multiword Expressions (MWE 2017), pages 31–47, Valencia, Spain. Association for Computational Linguistics.

Nathan Schneider, Emily Danchik, Chris Dyer, and Noah A. Smith. 2014a. Discriminative lexical semantic segmentation with gaps: Running the MWE gamut. Transactions of the Association for Computational Linguistics, 2:193–206.

Nathan Schneider, Dirk Hovy, Anders Johannsen, and Marine Carpuat. 2016. SemEval-2016 task 10: Detecting minimal semantic units and their meanings (DiMSUM). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 546–559, San Diego, California. Association for Computational Linguistics.

Nathan Schneider, Spencer Onuffer, Nora Kazour, Emily Danchik, Michael T Mordowane, Henrietta Conrad, and Noah A Smith. 2014b. Comprehensive annotation of multiword expressions in a social web corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, pages 455–461. European Language Resources Association (ELRA).

Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. Brat: a web-based tool for nlp-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 102–107.

Jeniya Tabassum, Mounica Maddela, Wei Xu, and Alan Ritter. 2020. Code and named entity recognition in stackoverflow. In Annual Meeting of the Association for Computational Linguistics.

Joshua Tanner and Jacob Hoffman. 2023. MWE as WSD: Solving multiword expression identification with word sense disambiguation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 181–193, Singapore. Association for Computational Linguistics.

Shiva Taslimipoor, Sara Bahaaedini, and Ekaterina Kochmar. 2020. MTLB-STRUCT @parseme 2020: Capturing unseen multiword expressions using multi-task learning and pre-trained masked language models. In Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons, pages 142–148, online. Association for Computational Linguistics.

Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022. SemEval-2022 task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 107–121, Seattle, United States. Association for Computational Linguistics.

Simone Tedeschi, Federico Martelli, and RobertoNavigli. 2022. ID10M: Idiom identification in 10 languages. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2715–2726, Seattle, United States. Association for Computational Linguistics.

Abigail Walsh, Claire Bonial, Kristina Geeraert, John P. McCrae, Nathan Schneider, and Clarissa Somers. 2018. Constructing an annotated corpus of verbal MWEs for English. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 193–200, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ni-anwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. Ontonotes release 5.0. LDC2013T19.

Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo Paetzold, Lucia Specia, Sanja Štajner, Anais Tack, and Marcos Zampieri. 2018. A report on the complex word identification shared task 2018. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 66–78, New Orleans, Louisiana. Association for Computational Linguistics.

Daniel Zeman, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21, Brussels, Belgium. Association for Computational Linguistics.

Wenxuan Zhou, Sheng Zhang, Yu Gu, Muahao Chen, and Hoifung Poon. 2024. UniversalNER: Targeted distillation from large language models for open named entity recognition. In The Twelfth International Conference on Learning Representations.
<table>
  <tr>
    <th>Source</th>
    <th>Paper</th>
    <th>URL</th>
    <th>License</th>
    <th>Note</th>
  </tr>
  <tr>
    <td>News</td>
    <td>Gjavas and Štajner (2013); Yimam et al. (2018)</td>
    <td>https://takelab.fer.hr/data/evsimplify/ https://sites.google.com/view/cwsharedtask2018/datasets</td>
    <td>Creative Commons Attribution-NonCommercial-ShareAlike 3.0</td>
    <td>We used the News_Train data from the CWI Shared Task 2018 datasets.</td>
  </tr>
  <tr>
    <td>Commentary</td>
    <td>Kocmi et al. (2023)</td>
    <td>https://data.statmt.org/news-commentary/v18.1/training-monolingual/</td>
    <td>Can be freely used for research purposes.</td>
    <td>We used the English part of v18.1.</td>
  </tr>
  <tr>
    <td>TED (NAIST-NTT)</td>
    <td>Neubig et al. (2014)</td>
    <td>https://ahcweb01.naist.jp/old/resource/tedtrebank/</td>
    <td>Creative Commons ShareAlike-Attribution-NonCommercial</td>
    <td></td>
  </tr>
  <tr>
    <td>TED (IWSLT)</td>
    <td>Cettolo et al. (2012, 2017)</td>
    <td>https://wit3.fbk.eu/2017-01 https://wit3.fbk.eu/2017-01-b</td>
    <td>Creative Commons Attribution-NonCommercial-NoDerivs 3.0</td>
    <td></td>
  </tr>
  <tr>
    <td>UD</td>
    <td>Bies et al. (2012); Walsh et al. (2018)</td>
    <td>https://gitlab.com/parseme/parseme_corpus_en</td>
    <td>Creative Commons ShareAlike 4.0</td>
    <td>We used the portion of PARSEME corpus from the English Web Treebank.</td>
  </tr>
</table>

Table 9: Data sources of CoAM.

A  Construction of CoAM

A.1  Data Sources

See Table 9.

A.2  Notes on Idiomaticity and (Non-)Compositionality

To judge whether a sequence is an MWE, we focus on (semantic) idiomaticity instead of non-compositionality, although non-compositionality has been considered an inherent property of MWEs in previous studies, such as Tedeschi et al. (2022). To discuss the difference between the two notions, let us consider the expression spill the beans. It is deemed compositional because it can be analyzed as being made up of spill in a “reveal” sense and the beans in a “secret(s)” sense, resulting in the overall compositional reading of “reveal the secret(s)” (Sag et al., 2002). Meanwhile, the “secret(s)” sense is unique to the expression and cannot be derived from the word beans, making the whole expression semantically idiomatic. We argue that spill the beans should be identified as an MWE because the meaning of beans depends on the whole expression, and that any idiomatic sequences of this kind should be identified even if they are compositional.

A.3  Algorithm for MWE Type Tagging

We tag MWEs in CoAM by the algorithm shown in Algorithm 1.

A.4  Statistics Regarding MWE Continuity

Discontinuous MWEs are common in English, and previous work has stressed the importance of identifying them (Schneider et al., 2014a; Rohanian et al., 2019). Table 10 shows the ratio of discontinuous MWEs by MWE type. VERB MWEs have the highest proportion of discontinuous expressions, while MOD/CONN, NOUN, and OTHER MWEs reach lower values.
Algorithm 1 Algorithm of MWE Type Tagging

Input: MWE m
Output: Type t
1: for all \( w_i \in m \) do
2:     if \( w_i = w^* \) then
3:         if PoS of \( w_i \in \{ \text{NOUN}, \text{PRON}, \text{PROPN} \} \) then
4:             if children of \( w_i \) include a relative clause modifier that is verb then
5:                 \( t \leftarrow \text{VERB} \)                ▷ E.g., . . . *the price he pays* . . .
6:             end if
7:             \( t \leftarrow \text{NOUN} \)                ▷ E.g., . . . *born in the middle of nowhere* . . .
8:         else if PoS of \( w_i \in \{ \text{VERB}, \text{AUX} \} \) then
9:             if nominal subject of \( w_i \in m \) then
10:                 \( t \leftarrow \text{CLAUSE} \)           ▷ E.g., . . . *when it comes to climate change* . . .
11:             else
12:                 \( t \leftarrow \text{VERB} \)                ▷ E.g., . . . *the expression stands for* . . .
13:             end if
14:         else if PoS of \( w_i \in \{ \text{ADP}, \text{ADJ}, \text{ADV}, \text{CCONJ}, \text{SCONJ} \} \) then
15:             \( t \leftarrow \text{MOD/CONN} \)              ▷ E.g., . . . *for at least two decades* . . .
16:         else
17:             \( t \leftarrow \text{OTHER} \) (other PoS)      ▷ E.g., . . . *de facto target gauge* . . .
18:         end if
19:     else
20:         \( t \leftarrow \text{OTHER} \) (head not in MWE)   ▷ E.g., . . . *and so on* . . .
21:     end if
22: end for
23: return t

<table>
  <tr>
    <th>MWE Type</th>
    <th>Discontinuous (%)</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>Noun</td>
    <td>2.0</td>
    <td>*a suicide car bomber and Taliban militants*</td>
  </tr>
  <tr>
    <td>Verb</td>
    <td>21.8</td>
    <td>*turned four aircraft into cruise missiles*</td>
  </tr>
  <tr>
    <td>Mod/Conn</td>
    <td>4.8</td>
    <td>*concentration of power in his own hands*</td>
  </tr>
  <tr>
    <td>Clause</td>
    <td>0.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Other</td>
    <td>20.8</td>
    <td>*no more than 2.7%*</td>
  </tr>
</table>

Table 10: Ratio of discontinuous MWEs in CoAM by MWE type.
<table>
  <tr>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td rowspan="4">LoRA</td>
    <td>r</td>
    <td>64</td>
  </tr>
  <tr>
    <td>\( \alpha \)</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Target modules</td>
    <td>All linear layers</td>
  </tr>
  <tr>
    <td rowspan="5">Training</td>
    <td>Epoch</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Effective batch size</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>1e-4 for Llama-70B<br>2e-4 for other models</td>
  </tr>
  <tr>
    <td>Learning rate scheduler</td>
    <td>constant</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>paged_adamw_8bit (\( \beta_2 = 0.999 \))</td>
  </tr>
  <tr>
    <td>Max grad norm</td>
    <td>0.3</td>
  </tr>
</table>

Table 11: Hyperparameters for fine-tuning. The epoch was determined based on preliminary experiments, while other parameters are based on Dettmers et al. (2023).

<table>
  <tr>
    <th>Name</th>
    <th>Example (Input → Output)</th>
  </tr>
  <tr>
    <td>dict_to_dict_list</td>
    <td>{1: 'ACL', 2: 'stands', 3: 'for', ...} → [{'surface': 'stands for', 'indices': [2, 3]}]</td>
  </tr>
  <tr>
    <td>str_to_str_number_span</td>
    <td>ACL stands for Association for Computational Linguists . → ACL <i>stands for</i> Association for Computational Linguists .</td>
  </tr>
  <tr>
    <td>tsv_to_tsv</td>
    <td>ACL\nstands\nfor\n→ ACL\t1\nstands\t1\nfor\t1\n</td>
  </tr>
</table>

Table 12: Input-output formats for fine-tuning. Bold fonts denote the format employed for the main experiments. Suppose the sentence given as the example is <b>ACL stands for Association for Computational Linguistics</b>.

B Experimental Setup

B.1 LLM fine-tuning

Hyperparameters Table 11 shows hyperparameters used for training and evaluation. At inference, we perform greedy decoding.

Input-Output Format To investigate the optimal input-output format, we perform preliminary experiments. We prepare three input-output formats described in Table 12. All the output formats allow us to represent MWEs in any form, including discontinuous or overlapping MWEs. We train and evaluate Llama-8B and Qwen-7B with CoAM, using each input-output format. We use the hyperparameters described in Table 11 and the long MWE definition (same as the main experiments).

As a result, tsv_to_tsv turned out to be the only format the models can comply with. Employing other formats results in the violation of the format despite carefully written instructions. With dict_to_dict_list, the models produce wrong indices in the outputs. With str_to_str_number_span, the models delete spaces before punctuations.

MWE Definition To validate the efficacy of the long MWE definition, we perform an ablation study comparing the long definition to the short definition. They are both a summary of the full definition described in Appendix D. As shown in Table 13, the long definition has 162 words while the short is further contracted to 57 words. We perform experiments with Llama-8B and Qwen-7B, using the tsv_to_tsv format and the hyperparameters described in Table 11 (same as the main experiments).

Table 14 presents the result. For both models, the long definition achieves higher F1 scores than the short by more than 12 points, indicating the effectiveness of a detailed definition.

B.2 Evaluation Metrics

We use MWE-based (exact-match) precision, recall, and F1 score (Savary et al., 2023). Let \( G \) be the set of gold MWEs and \( H \) the set of predicted MWEs (hypothesis),

\[
\text{Recall} = |G \cap H| / |G| \tag{1}
\]
\[
\text{Precision} = |G \cap H| / |H| \tag{2}
\]

where each MWE is represented with the ID of the sentence and the IDs of its constituent tokens. F1 is the harmonic mean of precision and recall.
<table>
  <tr>
    <th>Name</th>
    <th>Content</th>
  </tr>
  <tr>
    <td rowspan="3">Long</td>
    <td>Here, an MWE is defined as a sequence that satisfies the following three conditions.<br>
1. It consists of multiple words that are always realized by the same lexemes. Such words cannot be replaced without distorting the meaning of the expression or violating language conventions.<br>
2. It displays semantic, lexical, or syntactic idiomaticity. Semantic idiomaticity occurs when the meaning of an expression cannot be explicitly derived from its components. In other words, a semantically idiomatic takes on a meaning that is unique to that combination of words. Lexical idiomaticity occurs when one or more components of an expression are not used as stand-alone words in standard English. Syntactic idiomaticity occurs when the grammar of an expression cannot be derived directly from that of its components. For example, semantically idiomatic MWEs include "break up", the lexically idiomatic include "to and fro", and the syntactically idiomatic include "long time no see".<br>
3. It is not a multi-word named entity, i.e., a specific name of a person, facility, etc.</td>
  </tr>
  <tr>
    <td>Here, an MWE is defined as a sequence that satisfies the following three conditions.<br>
1. It consists of multiple words that are always realized by the same lexemes.<br>
2. It is idiomatic, that is, its meaning cannot be explicitly derived from its components.<br>
3. It is not a multi-word named entity, i.e., a specific name of a person, facility, etc.</td>
  </tr>
</table>

Table 13: MWE definitions to be included in prompts for fine-tuning.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">F1</th>
    <th colspan="4">Recall by MWE Type</th>
  </tr>
  <tr>
    <th>P</th>
    <th>R (121)</th>
    <th>Noun (139)</th>
    <th>Verb (111)</th>
    <th>Mod/Conn (7)</th>
    <th>Clause</th>
  </tr>
  <tr>
    <td rowspan="2">Long</td>
    <td>Llama-8B</td>
    <td>24.9\(_{\pm3.7}\)</td>
    <td>92.0\(_{\pm2.0}\)</td>
    <td>14.4\(_{\pm2.4}\)</td>
    <td>3.3\(_{\pm0.0}\)</td>
    <td>19.4\(_{\pm2.9}\)</td>
    <td>21.6\(_{\pm4.8}\)</td>
    <td>0.0\(_{\pm0.0}\)</td>
  </tr>
  <tr>
    <td>Qwen-7B</td>
    <td>48.1\(_{\pm1.0}\)</td>
    <td>60.9\(_{\pm0.8}\)</td>
    <td>39.7\(_{\pm1.0}\)</td>
    <td>28.7\(_{\pm1.3}\)</td>
    <td>52.0\(_{\pm0.8}\)</td>
    <td>39.6\(_{\pm0.9}\)</td>
    <td>0.0\(_{\pm0.0}\)</td>
  </tr>
  <tr>
    <td rowspan="2">Short</td>
    <td>Llama-8B</td>
    <td>21.0\(_{\pm3.1}\)</td>
    <td>89.9\(_{\pm4.7}\)</td>
    <td>11.9\(_{\pm2.0}\)</td>
    <td>3.3\(_{\pm0.0}\)</td>
    <td>16.3\(_{\pm2.3}\)</td>
    <td>16.8\(_{\pm5.0}\)</td>
    <td>0.0\(_{\pm0.0}\)</td>
  </tr>
  <tr>
    <td>Qwen-7B</td>
    <td>36.7\(_{\pm1.3}\)</td>
    <td>65.7\(_{\pm0.7}\)</td>
    <td>25.5\(_{\pm1.3}\)</td>
    <td>15.4\(_{\pm1.7}\)</td>
    <td>32.6\(_{\pm2.2}\)</td>
    <td>29.7\(_{\pm0.0}\)</td>
    <td>0.0\(_{\pm0.0}\)</td>
  </tr>
</table>

Table 14: Ablation results, as mean percentage scores of three runs with random seeds. The numbers in parentheses are the MWE counts. \( \pm \) denotes standard deviation.

C Computational Budgets

For experiments for MaW (Rule+BiEnc and Rule+DCA), we use a single NVIDIA RTX 2080Ti GPU with 11GB RAM. Each run of training and testing takes around 8 minutes. Consequently, the total GPU hours for MaW are estimated to be 0.8 hours.

For LLM fine-tuning, we use NVIDIA RTX 6000 GPU with 48GB RAM for smaller models (Llama-8B and Qwen-7B) and NVIDIA A100 PCIe with 80GB RAM for larger models (Llama-70B and Qwen-72B). Each run of training and testing takes around 66 minutes for the smaller models and 588 minutes for the larger models. Thus, the total GPU hours for fine-tuning experiments of smaller and larger models are estimated to be 6.6 hours and 58.8 hours, respectively.
D  Excerpt of Annotation Guidelines

D.1  Definition of MWEs

In our definition, MWEs are idiomatic sequences that (a) consist of multiple words, (b) display semantic, lexical, or syntactic idiomaticity, and (c) are not proper nouns.

a. An MWE consists of at least two words that are always realized by the same lexemes.12 For example, the MWE "break up" is always realized by (1) "break" or its conjugated form such as "broke" and (2) "up". Such words cannot be replaced without distorting the meaning of the expression or violating the language conventions.

b. An MWE displays semantic, lexical, or syntactic idiomaticity. The semantically idiomatic MWEs include "break up", the lexically idiomatic include "to and fro", and the syntactically idiomatic include "long time no see".

• Semantic idiomaticity occurs when the meaning of an expression cannot be derived from its components. That is, you cannot necessarily infer the meaning of the expression even if you know all the senses of its components. In other words, a semantically idiomatic expression takes on a meaning that is unique to that combination of words.
The inferability differs from one expression to another. The meaning of idiomatic MWEs such as "kick the bucket" cannot be inferred from its components at all. The meaning of institutionalized phrases like "traffic light" is inferable to some degree. Yet "traffic light" is semantically idiomatic because it does not mean any type of light related to traffic but a specific type of light.
• Lexical idiomaticity occurs when one or more components of an expression are not used as stand-alone words in standard English. Examples include "bide one's time": "bide" rarely appears by itself in today’s standard English.
• Syntactic idiomaticity occurs when the grammatical structure of an expression cannot be derived directly from that of its components. It constitutes expressions whose grammar seems to go against standard English grammatical conventions. This includes expressions such as "all of a sudden," where "sudden" appears in its archaic noun form.

c. An MWE is not a proper noun, such as the name of a specific person, organization, and so forth.
In this project, do not annotate proper nouns unless they are part of an MWE (like Pavlov in "Pavlov's dog"). Proper nouns usually start with a capital letter, while exceptions like "iPhone 15" exist. Below is the full list of proper nouns in our definition.13

<table>
  <tr>
    <th>Type</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>People’s names</td>
    <td>Shohei Ohtani</td>
  </tr>
  <tr>
    <td>Nationalities or religious or political groups</td>
    <td>African American, Sunni Muslims</td>
  </tr>
  <tr>
    <td>Facilities</td>
    <td>Narita International Airport, Taipei 101</td>
  </tr>
  <tr>
    <td>Organizations</td>
    <td>Procter & Gamble, Kyoto University</td>
  </tr>
  <tr>
    <td>Geopolitical entities (GPE)</td>
    <td>Los Angeles, the United Kingdom</td>
  </tr>
  <tr>
    <td>Non-GPE locations</td>
    <td>Mount Fuji, Amazon River</td>
  </tr>
  <tr>
    <td>Products</td>
    <td>Toyota Prius, Samsung Galaxy</td>
  </tr>
  <tr>
    <td>Named events</td>
    <td>World War II, Olympic Games Tokyo 2020</td>
  </tr>
  <tr>
    <td>Works of art</td>
    <td>Norwegian Wood, Bohemian Rhapsody</td>
  </tr>
  <tr>
    <td>Named legal documents</td>
    <td>The Magna Carta</td>
  </tr>
  <tr>
    <td>Named languages</td>
    <td>Middle English, American Sign Language</td>
  </tr>
</table>

D.2  Notes

MWEs containing replaceable words

MWEs are not necessarily made of continuous words. Some MWEs contain open slots, that is, words that may be replaced with a large or open class of words. In annotation, we do not include open slots in MWEs, as illustrated by the following examples:

12Lexeme is a set of words related through inflection. Words belonging to the same lexeme share a common lemma.
13The 11 types derive from the named entity types of OntoNotes Release 5.0 (Weischel et al., 2013).
• You took me by surprise!
• Pick me up at the station.

<table>
  <tr>
    <th>MWE</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th>
  </tr>
  <tr>
    <td>Pick up</td><td>✔</td><td></td><td>✔</td><td></td><td></td><td></td>
  </tr>
</table>

Note that if a word is replaceable with only a very small number of alternatives, we consider it as part of the MWE. For example, we count the following as different MWEs:

• Their food leaves a lot to be desired.
• The work leaves much to be desired.

<table>
  <tr>
    <th>MWE</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th>
  </tr>
  <tr>
    <td>leaves much to be desired</td><td></td><td></td><td>✔</td><td>✔</td><td>✔</td><td>✔</td><td>✔</td>
  </tr>
</table>

Overlapping MWES
There is a chance that multiple MWES share the same word. For example, letting in the following sentence belongs to both letting in and letting out.

<table>
  <tr>
    <th>MWE</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th>
  </tr>
  <tr>
    <td>letting in</td><td></td><td></td><td>✔</td><td></td><td>✔</td><td></td><td></td>
  </tr>
  <tr>
    <td>letting out</td><td></td><td></td><td>✔</td><td></td><td></td><td></td><td>✔</td>
  </tr>
</table>

Rearranged MWES
There is a chance that the order of component words in an MWE is different from its canonical form, but we still count such rearranged sequences as MWES. In the following example, rearrangement happens in the MWE break <one’s> heart. As a side note, my here is not annotated because it is replaceable.

<table>
  <tr>
    <th>MWE</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th>
  </tr>
  <tr>
    <td>heart broken</td><td>✔</td><td></td><td>✔</td><td></td><td></td>
  </tr>
</table>

Hyphens
On our annotation interface, hyphens are treated as words. When you encounter a hyphen in an MWE, mark the hyphen. Meanwhile, note that some hyphenated words, e.g., twenty-one are not MWES and should not be annotated.

<table>
  <tr>
    <th>MWE</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th>
  </tr>
  <tr>
    <td>next-door neighbor</td><td></td><td></td><td>✔</td><td>✔</td><td>✔</td><td></td><td></td>
  </tr>
</table>

Reviewer paper summaries: The paper presents a new corpus of Multi-Word Expressions in English, along with a discussion of the limitations of previous examples of this type of corpus

This paper describes the construction and annotation of a Corpus of All-Type Multiword Expressions. Source sentences were collected from a variety of English datasets including news, Commentary, Tedtalks, and the English Web Treebank. Then an annotation guideline was developed for all-type MWEs annotation. These guidelines were used to by annotators and authors (of the paper, I presume) to annotate the corpus for all-type MWEs, using a specially developed annotation interface based on google sheets. Then two methods of MWEs detection were evaluated on the annotated corpus: MWEasWSD (Rule-based then trainable filter), and LLM-finetuning. The authors found that LLM-finetuning outperformed MWEasWSD.
Summary of strengths: The new corpus is larger and more carefully constructed than previous efforts to accomplish the same task.

More types of MWEs are annotated, comparing to known datasets.
The annotation efforts (but also see weaknesses).
The evaluation efforts, providing further evidence that LLMs (especially larger ones) are good at a variety of NLP tasks.
Summary of weaknesses: Frankly it's not clear in this era that MWEs in English are a compelling research area.  The paper writes up some interesting experiments with getting LLMs to identify MWEs, but does not describe any use of the MWEs themselves in downstream tasks.  The paper would be greatly improved by demonstrating how this dataset can improve other tasks, perhaps MT, NLG, or the like.

My problem with the paper is that MWEs themselves are ambiguous. This has been shown in a low IAA of 37.3%. I think that before we want to detect MWEs automatically, we need to address this low IAA first. The authors implied that DiMSUM’s inconsistent annotation was one of the motivations for a new annotation effort, but the IAA in DiMSUM is reported to be in the region of 60-65%. Although the authors listed this low IAA in the limitation, I am not convinced that the implicit aim to provide a more consistently annotated dataset is supported. (In other words, how is this corpus better than DiMSUM, a five times larger one?)
The relative small size of the corpus is also a weakness, compared to other MWE datasets.
Comments, suggestions and typos: In section 4.8, the tagging mechanism described only addresses 3 of the types of MWEs listed.  Was there an error checking step after applying those tags?  Regardless, the process described seems unlikely to correctly label all possible MWEs.  For example, I would expect "tongue in cheek" to be a "modifier", but the head of that phrase would be a noun in this process.

If this is too narrow for an ACL style conference, perhaps some venue such as SyntaxFest would be more interested in this work.

In table 4, why there is no annotation stats for UD?
Line 608: “Timothy Baldwin and Su Nam Kim. 2010. Multiword…”
Should include the publisher of the book.
Overall assessment scores: 2.5

3