Introduction
Idiomatic expressions pose a major challenge for a wide range of applications in natural language processing BIBREF0. These include machine translation BIBREF1, BIBREF2, semantic parsing BIBREF3, sentiment analysis BIBREF4, and word sense disambiguation BIBREF5. Idioms show significant syntactic and morphological variability (e.g. beans being spilled for spill the beans), which makes them hard to find automatically. Moreover, their non-compositional nature makes idioms really hard to interpret, because their meaning is often very different from the meanings of the words that make them up. Hence, successful systems need not only be able to recognise idiomatic expressions in text or dialogue, but they also need to give a proper interpretation to them. As a matter of fact, current language technology performs badly on idiom understanding, a phenomenon that perhaps has not received enough attention.
Nearly all current language technology used in NLP applications is based on supervised machine learning. This requires large amounts of labelled data. In the case of idiom interpretation, however, only small datasets are available. These contain just a couple of thousand idiom instances, covering only about fifty different types of idiomatic expressions. In fact, existing annotated corpora tend to cover only a small set of idiom types, comprising just a few syntactic patterns (e.g., verb-object combinations), of which a limited number of instances are extracted from a large corpus.
This is not surprising as preparing and compiling such corpora involves a large amount of manual extraction work, especially if one wants to allow for form variation in the idiomatic expressions (for example, extracting cooking all the books for cook the books). This work involves both the crafting of syntactic patterns to match potential idiomatic expressions and the filtering of false extractions (non-instances of the target expression e.g. due to wrong parses), and increases with the amount of idiom types included in the corpus (which, in the worst case, means an exponential increase in false extractions). Thus, building a large corpus of idioms, especially one that covers many types in many syntactic constructions, is costly. If a high-precision, high-recall system can be developed for the task of extracting the annotation candidates, this cost will be greatly reduced, making the construction of a large corpus much more feasible.
The variability of idioms has been a significant topic of interest among researchers of idioms. For example, BIBREF6 investigates the internal and external modification of a set of idioms in a large English corpus, whereas BIBREF7, quantifies and classifies the variation of a set of idioms in a large corpus of Dutch, setting up a useful taxonomy of variation types. Both find that, although idiomatic expressions mainly occur in their dictionary form, there is a significant minority of idiom instances that occur in non-dictionary variants. Additionally, BIBREF8 show that idiom variants retain their idiomatic meaning more often and are processed more easily than previously assumed. This emphasises the need for corpora covering idiomatic expressions to include these variants, and for tools to be robust in dealing with them.
As such, the aim of this article is to describe methods and provide tools for constructing larger corpora annotated with a wider range of idiom types than currently in existence due to the reduced amount of manual labour required. In this way we hope to stimulate further research in this area. In contrast to previous approaches, we want to catch as many idiomatic expressions as possible, and we achieve this by casting a wide net, that is, we consider the widest range of possible idiom variants first and then filter out any bycatch in a way that requires the least manual effort.
We expect research will benefit from having larger corpora by improving evaluation quality, by allowing for the training of better supervised systems, and by providing additional linguistic insight into idiomatic expressions. A reliable method for extracting idiomatic expressions is not only needed for building an annotated corpus, but can also be used as part of an automatic idiom processing pipeline. In such a pipeline, extracting potentially idiomatic expressions can be seen as a first step before idiom disambiguation, and the combination of the two modules then functions as an complete idiom extraction system.
The main research question that we aim to answer in this article is whether dictionary-based extraction of potentially idiomatic expressions is robust and reliable enough to facilitate the creation of wide-coverage sense-annotated idiom corpora.
By answering this question we make several contributions to research on multiword expressions, in particular that of idiom extraction. Firstly, we provide an overview of existing research on annotating idiomatic expressions in corpora, showing that current corpora cover only small sets of idiomatic types (Section SECREF3). Secondly, we quantify the coverage and reliability of a set of idiom dictionaries, demonstrating that there is little overlap between resources (Section SECREF4). Thirdly, we develop and release an evaluation corpus for extracting potentially idiomatic expressions from text (Section SECREF5). Finally, various extraction systems and combinations thereof are implemented, made available to the research community, and evaluated empirically (Section SECREF6).
New Terminology: Potentially Idiomatic Expression (PIE)
The ambiguity of phrases like wake up and smell the coffee poses a terminological problem. Usually, these phrases are called idiomatic expressions, which is suitable when they are used in an idiomatic sense, but not so much when they are used in a literal sense. Therefore, we propose a new term: potentially idiomatic expressions, or PIEs for short. The term potentially idiomatic expression refers to those expressions which can have an idiomatic meaning, regardless of whether they actually have that meaning in a given context. So, see the light is a PIE in both `After another explanation, I finally saw the light' and `I saw the light of the sun through the trees', while it is an idiomatic expression in the first context, and a literal phrase in the latter context.
The processing of PIEs involves three main challenges: the discovery of (new) PIE types, the extraction of instances of known PIE types in text, and the disambiguation of PIE instances in context. Here, we propose calling the discovery task simply PIE discovery, the extraction task simply PIE extraction, and the disambiguation task PIE disambiguation. Note that these terms contrast with the terms used in existing research. There, the discovery task is called type-based idiom detection and the disambiguation task is called token-based idiom detection (cf. BIBREF10, BIBREF11), although this usage is not always consistent. Because these terms are very similar, they are potentially confusing, and that is why we propose novel terminology.
Other terminology comes from literature on multiword expressions (MWEs) more generally, i.e. not specific to idioms. Here, the task of finding new MWE types is called MWE discovery and finding instances of known MWE types is called MWE identification BIBREF12. Note, however, that MWE identification generally consists of finding only the idiomatic usages of these types (e.g. BIBREF13). This means that MWE identification consists of both the extraction and disambiguation tasks, performed jointly. In this work, we propose to split this into two separate tasks, and we are concerned only with the PIE extraction part, leaving PIE disambiguation as a separate problem.
Related Work
This section is structured so as to reflect the dual contribution of the present work. First, we discuss existing resources annotated for idiomatic expressions. Second, we discuss existing approaches to the automatic extraction of idioms.
Related Work ::: Annotated Corpora and Annotation Schemes for Idioms
There are four sizeable sense-annotated PIE corpora for English: the VNC-Tokens Dataset BIBREF9, the Gigaword dataset BIBREF14, the IDIX Corpus BIBREF10, and the SemEval-2013 Task 5 dataset BIBREF15. An overview of these corpora is presented in Table TABREF7.
Related Work ::: Annotated Corpora and Annotation Schemes for Idioms ::: VNC-Tokens
The VNC-Tokens dataset contains 53 different PIE types. BIBREF9 extract up to 100 instances from the British National Corpus for each type, for a total of 2,984 instances. These types are based on a pre-existing list of verb-noun combinations and were filtered for frequency and whether two idiom dictionaries both listed them. Instances were extracted automatically, by parsing the corpus and selecting all sentences with the right verb and noun in a direct-object relation. It is unclear whether the extracted sentences were manually checked, but no false extractions are mentioned in the paper or present in the dataset.
All extracted PIE instances were annotated for sense as either idiomatic, literal or unclear. This is a self-explanatory annotation scheme, but BIBREF9 note that senses are not binary, but can form a continuum. For example, the idiomaticity of have a word in `You have my word' is different from both the literal sense in `The French have a word for this' and the figurative sense in `My manager asked to have a word'. They instructed annotators to choose idiomatic or literal even in ambiguous middle-of-the-continuum cases, and restrict the unclear label only to cases where there is not enough context to disambiguate the meaning of the PIE.
Related Work ::: Annotated Corpora and Annotation Schemes for Idioms ::: Gigaword
BIBREF14 present a corpus of 17 PIE types, for which they extracted all instances from the Gigaword corpus BIBREF18, yielding a total of 3,964 instances. BIBREF14 extracted these instances semi-automatically by manually defining all inflectional variants of the verb in the PIE and matching these in the corpus. They did not allow for inflectional variations in non-verb words, nor did they allow intervening words. They annotated these potential idioms as either literal or figurative, excluding ambiguous and unclear instances from the dataset.
Related Work ::: Annotated Corpora and Annotation Schemes for Idioms ::: IDIX
BIBREF10 build on the methodology of BIBREF14, but annotate a larger set of idioms (52 types) and extract all occurrences from the BNC rather than the Gigaword corpus, for a total of 4,022 instances including false extractions. BIBREF10 use a more complex semi-automatic extraction method, which involves parsing the corpus, manually defining the dependency patterns that match the PIE, and extracting all sentences containing those patterns from the corpus. This allows for larger form variations, including intervening words and inflectional variation of all words. In some cases, this yields many non-PIE extractions, as for recharge one's batteries in Example SECREF10. These were not filtered out before annotation, but rather filtered out as part of the annotation process, by having false extraction as an additional annotation label.
For sense annotation, they use an extensive tagset, distinguishing literal, non-literal, both, meta-linguistic, embedded, and undecided labels. Here, the both label (Example SECREF10) is used for cases where both senses are present, often as a form of deliberate word play. The meta-linguistic label (Example SECREF10) applies to cases where the PIE instance is used as a linguistic item to discuss, not as part of a sentence. The embedded label (Example SECREF10) applies to cases where the PIE is embedded in a larger figurative context, which makes it impossible to say whether a literal or figurative sense is more applicable. The undecided label is used for unclear and undecidable cases. They take into account the fact that a PIE can have multiple figurative senses, and enumerate these separately as part of the annotation.
. These high-performance, rugged tools are claimed to offer the best value for money on the market for the enthusiastic d-i-yer and tradesman, and for the first time offer the possibility of a battery recharging time of just a quarter of an hour. (from IDIX corpus, ID #314)
. Left holding the baby, single mothers find it hard to fend for themselves. (from BIBREF10, p.642)
. It has long been recognised that expressions such as to pull someone's leg, to have a bee in one's bonnet, to kick the bucket, to cook someone's goose, to be off one's rocker, round the bend, up the creek, etc. are semantically peculiar. (from BIBREF10, p.642)
. You're like a restless bird in a cage. When you get out of the cage, you'll fly very high. (from BIBREF10, p.642)
The both, meta-linguistic, and embedded labels are useful and linguistically interesting distinctions, although they occur very rarely (0.69%, 0.15%, and an unknown %, respectively). As such, we include these cases in our tagset (see Section SECREF5), but group them under a single label, other, to reduce annotation complexity. We also follow BIBREF10 in that we combine both the PIE/non-PIE annotation and the sense annotation in a single task.
Related Work ::: Annotated Corpora and Annotation Schemes for Idioms ::: SemEval-2013 Task 5b
BIBREF15 created a dataset for SemEval-2013 Task 5b, a task on detecting semantic compositionality in context. They selected 65 PIE types from Wiktionary, and extracted instances from the ukWaC corpus BIBREF17, for a total of 4,350 instances. It is unclear how they extracted the instances, and how much variation was allowed for, although there is some inflectional variation in the dataset. An unspecified amount of manual filtering was done on the extracted instances.
The extracted PIE instances were labelled as literal, idiomatic, both, or undecidable. Interestingly, they crowdsourced the sense annotations using CrowdFlower, with high agreement (90%–94% pairwise). Undecidable cases and instances on which annotators disagreed were removed from the dataset.
Related Work ::: Annotated Corpora and Annotation Schemes for Idioms ::: General Multiword Expression Corpora
In addition to the aforementioned idiom corpora, there are also corpora focused on multiword expressions (MWEs) in a more general sense. As idioms are a subcategory of MWEs, these corpora also include some idioms. The most important of these are the PARSEME corpus BIBREF19 and the DiMSUM corpus BIBREF20.
DiMSUM provides annotations of over 5,000 MWEs in approximately 90K tokens of English text, consisting of reviews, tweets and TED talks. However, they do not categorise the MWEs into specific types, meaning we cannot easily quantify the number of idioms in the corpus. In contrast to the corpus-specific sense labels seen in other corpora, DiMSUM annotates MWEs with WordNet supersenses, which provide a broad category of meaning for each MWE.
Similarly, the PARSEME corpus consists of over 62K MWEs in almost 275K tokens of text across 18 different languages (with the notable exception of English). The main differences with DiMSUM, except for scale and multilingualism, are that it only includes verbal MWEs, and that subcategorisation is performed, including a specific category for idioms. Idioms make up almost a quarter of all verbal MWEs in the corpus, although the proportion varies wildly between languages. In both corpora, MWE annotation was done in an unrestricted manner, i.e. there was not a predefined set of expressions to which annotation was restricted.
Related Work ::: Annotated Corpora and Annotation Schemes for Idioms ::: Overview
In sum, there is large variation in corpus creation methods, regarding PIE definition, extraction method, annotation schemes, base corpus, and PIE type inventory. Depending on the goal of the corpus, the amount of deviation that is allowed from the PIE's dictionary form to the instances can be very little BIBREF14, to quite a lot BIBREF10. The number of PIE types covered by each corpus is limited, ranging from 17 to 65 types, often limited to one or more syntactic patterns. The extraction of PIE instances is usually done in a semi-automatic manner, by manually defining patterns in a text or parse tree, and doing some manual filtering afterwards. This works well, but an extension to a large number of PIE types (e.g. several hundreds) would also require a large increase in the amount of manual effort involved. Considering the sense annotations done on the PIE corpora, there is significant variation, with BIBREF9 using only three tags, whereas BIBREF10 use six. Outside of PIE-specific corpora there are MWE corpora, which provide a different perspective. A major difference there is that annotation is not restricted to a pre-specified set of expressions, which has not been done for PIEs specifically.
Related Work ::: Extracting Idioms from Corpora
There are two main approaches to idiom extraction. The first approach aims to distinguish idioms from other multiword phrases, where the main purpose is to expand idiom inventories with rare or novel expressions BIBREF21, BIBREF22, BIBREF23, BIBREF24. The second approach aims to extract all occurrences of a known idiomatic expression in a text. In this paper, we focus on the latter approach. We rely on idiom dictionaries to provide a list of PIE types, and build a system that extracts all instances of those PIE types from a corpus. High-quality idiom dictionaries exist for most well-resourced languages, but their reliability and coverage is not known. As such, we quantify the coverage of dictionaries in Section SECREF4.
There is, to the best of our knowledge, no existing work that focuses on dictionary-based PIE extraction. However, there is closely-related work by BIBREF25, who present a system for the dictionary-based extraction of verb-noun combinations (VNCs) in English and Spanish. In their case, the VNCs can be any kind of multiword expression, which they subdivide into literal expressions, collocations, light verb constructions, metaphoric expressions, and idioms. They extract 173 English VNCs and 150 Spanish VNCs and annotate these with both their lexico-semantic MWE type and the amount of morphosyntactic variation they exhibit. BIBREF25 then compare a word sequence-based method, a chunking-based method, and a parse-based method for VNC extraction. Each method relies on the morpho-syntactic information in order to limit false extractions. Precision is evaluated manually on a sample of the extracted VNCs, and recall is estimated by calculating the overlap between the output of the three methods. Evaluation shows that the methods are highly complementary both in recall, since they extract different VNCs, and in precision, since combining the extractors yields fewer false extractions.
Whereas BIBREF25 focus on both idiomatic and literal uses of the set of expressions, like in this paper, BIBREF26 tackle only half of that task, namely extracting only literal uses of a given set of VMWEs in Polish. This complicates the task, since it combines extracting all occurrences of the VMWEs and then distinguishing literal from idiomatic uses. Interestingly, they also experiment with models of varying complexity, i.e. just words, part-of-speech tags, and syntactic structures. Their results are hard to put into perspective however, since the frequency of literal VMWEs in their corpus is very rare, whereas corpora containing PIEs tend to show a more balanced distribution.
Other similar work to ours also focuses on MWEs more generally, or on different subtypes of MWEs. In addition, these tend to combine both extraction and disambiguation in that they aim to extract only idiomatically used instances of the MWE, without extracting literally used instances or non-instances. Within this line of work, BIBREF27 focuses on verb-particle constructions, BIBREF28 on verbal MWEs (including idioms), and BIBREF29 on verbal MWEs (especially non-canonical variants).
Both BIBREF28 and BIBREF29 rely on a pre-defined set of expressions, whereas BIBREF27 also extracts unseen expressions, although based on a pre-defined set of particles and within the vary narrow syntactic frame of verb-particle constructions. The work of BIBREF27 is most similar to ours in that it builds an unsupervised system using existing NLP tools (PoS taggers, chunkers, parsers) and finds that a combination of systems using those tools performs best, as we find in Section SECREF69. BIBREF28 and BIBREF29, by contrast, use supervised classifiers which require training data, not just for the task in general, but specific to the set of expressions used in the task.
Although our approach is similar to that of BIBREF25, both in the range of methods used and in the goal of extracting certain multiword expressions regardless of morphosyntactic variation, there are two main differences. First, we use dictionaries, but extract entries automatically and do not manually annotate their type and variability. As a result, our methods rely only on the surface form of the expression taken from the dictionary. Second, we evaluate precision and recall in a more rigorous way, by using an evaluation corpus exhaustively annotated for PIEs. In addition, we do not put any restriction on the syntactic type of the expressions to be extracted, which BIBREF27, BIBREF28, BIBREF25, and BIBREF29 all do.
Coverage of Idiom Inventories ::: Background
Since our goal is developing a dictionary-based system for extracting potentially idiomatic expressions, we need to devise a proper method for evaluating such a system. This is not straightforward, even though the final goal of such a system is simple: it should extract all potentially idiomatic expressions from a corpus and nothing else, regardless of their sense and the form they are used in. The type of system proposed here hence has two aspects that can be evaluated: the dictionary that it is using as a resource for idiomatic expression, and the extractor component that finds idioms in a corpus.
The difficulty here is that there is no undisputed and unambiguous definition of what counts as an idiom BIBREF30, as is the case with multiword expressions in general BIBREF12. Of course, a complete set of idiomatic expressions for English (or any other language) is impossible to get due to the broad and ever-changing nature of language. This incompleteness is exacerbated by the ambiguity problem: if we had a clear definition of idiom we could make an attempt of evaluating idiom dictionaries on their accuracy, but it is practically impossible to come up with a definition of idiom that leaves no room for ambiguity. This ambiguity, among others, creates a large grey area between clearly non-idiomatic phrases on the one hand (e.g. buy a house), and clear potentially idiomatic phrases on the other hand (e.g. buy the farm). As a consequence, we cannot empirically evaluate the coverage of the dictionaries. Instead, in this work, we will quantify the divergence between various idiom dictionaries and corpora, with regard to their idiom inventories. If they show large discrepancies, we take that to mean that either there is little agreement on definitions of idiom or the category is so broad that a single resource can only cover a small proportion. Conversely, if there is large agreement, we assume that idiom resources are largely reliable, and that there is consensus around what is, and what is not, an idiomatic expression.
We use different idiom resources and assume that the combined set of resources yields an approximation of the true set of idioms in English. A large divergence between the idiom inventories of these resources would then suggest a low recall for a single resource, since many other idioms are present in the other resources. Conversely, if the idiom inventories largely overlap, that indicates that a single resource can already yield decent coverage of idioms in the English language. The results of the dictionary comparisons are in Section SECREF36.
Coverage of Idiom Inventories ::: Selected Idiom Resources (Data and Method)
We evaluate the quality of three idiom dictionaries by comparing them to each other and to three idiom corpora. Before we report on the comparison we first describe why we select and how we prepare these resources. We investigate the following six idiom resources:
Wiktionary;
the Oxford Dictionary of English Idioms (ODEI, BIBREF31);
UsingEnglish.com (UE);
the Sporleder corpus BIBREF10;
the VNC dataset BIBREF9;
and the SemEval-2013 Task 5 dataset BIBREF15.
These dictionaries were selected because they are available in digital format. Wiktionary and UsingEnglish have the added benefit of being freely available. However, they are both crowdsourced, which means they lack professional editing. In contrast, ODEI is a traditional dictionary, created and edited by lexicographers, but it has the downside of not being freely available.
For Wiktionary, we extracted all idioms from the category `English Idioms' from the English version of Wiktionary. We took the titles of all pages containing a dictionary entry and considered these idioms. Since we focus on multiword idiomatic expressions, we filtered out all single-word entries in this category. More specifically, since Wiktionary is a constantly changing resource, we used the 8,482 idioms retrieved on 10-03-2017, 15:30. We used a similar extraction method for UE, a web page containing freely available resources for ESL learners, including a list of idioms. We extracted all idioms which have publicly available definitions, which numbered 3,727 on 10-03-2017, 15:30. Again, single-word entries and duplicates were filtered out. Concerning ODEI, all idioms from the e-book version were extracted, amounting to 5,911 idioms scraped on 13-03-2017, 10:34. Here we performed an extra processing step to expand idioms containing content in parentheses, such as a tough (or hard) nut (to crack). Using a set of simple expansion rules and some hand-crafted exceptions, we automatically generated all variants for this idiom, with good, but not perfect accuracy. For the example above, the generated variants are: {a tough nut, a tough nut to crack, a hard nut, a hard nut to crack}. The idioms in the VNC dataset are in the form verb_noun, e.g. blow_top, so they were manually expanded to a regular dictionary form, e.g. blow one's top before comparison.
Coverage of Idiom Inventories ::: Method
In many cases, using simple string-match to check overlap in idioms does not work, as exact comparison of idioms misses equivalent idioms that differ only slightly in dictionary form. Differences between resources are caused by, for example:
inflectional variation (crossing the Rubicon — cross the Rubicon);
variation in scope (as easy as ABC — easy as ABC);
determiner variation (put the damper on — put a damper on);
spelling variation (mind your p's and q's — mind your ps and qs);
order variation (call off the dogs — call the dogs off);
and different conventions for placeholder words (recharge your batteries — recharge one's batteries), where both your and one's can generalise to any possessive personal pronoun.
These minor variations do not fundamentally change the nature of the idiom, and we should count these types of variation as belonging to the same idiom (see also BIBREF32, who devise a measure to quantify different types of variation allowed by specific MWEs). So, to get a good estimate of the true overlap between idiom resources, these variations need to be accounted for, which we do in our flexible matching approach.
There is one other case of variation not listed above, namely lexical variation (e.g. rub someone up the wrong way - stroke someone the wrong way). We do not abstract over this, since we consider lexical variation to be a more fundamental change to the nature of the idiom. That is, a lexical variant is an indicator of the coverage of the dictionary, where the other variations are due to different stylistic conventions and do not indicate actual coverage. In addition, it is easy to abstract over the other types of variation in an NLP application, but this is not the case for lexical variation.
The overlap counts are estimated by abstracting over all variations except lexical variation in a semi-automatic manner, using heuristics and manual checking. Potentially overlapping idioms are selected using the following set of heuristics: whether an idiom from one resource is a substring (including gaps) of an idiom in the other resource, whether the words of an idiom form a subset of the words of an idiom in the other resource, and whether there is an idiom in the other resource which has a Levenshtein ratio of over 0.8. The Levenshtein ratio is an indicator of the Levenshtein distance between the two idioms relative to their length. These potential matches are then judged manually on whether they are really forms of the same idiom or not.
Coverage of Idiom Inventories ::: Results
The results of using exact string matching to quantify the overlap between the dictionaries is illustrated in Figure FIGREF37.
Overlap between the three dictionaries is low. A possible explanation for this lies with the different nature of the dictionaries. Oxford is a traditional dictionary, created and edited by professional lexicographers, whereas Wiktionary is a crowdsourced dictionary open to everyone, and UsingEnglish is similar, but focused on ESL-learners. It is likely that these different origins result in different idiom inventories. Similarly, we would expect that the overlap between a pair of traditional dictionaries, such as the ODEI and the Penguin Dictionary of English Idioms BIBREF33 would be significantly higher. It should also be noted, however, that comparisons between more similar dictionaries also found relatively little overlap (BIBREF34; BIBREF35). A counterpoint is provided by BIBREF36, who quantifies coverage of verb-particle constructions in three different dictionaries and finds large overlap – perhaps because verb-particle are a more restricted class.
As noted previously, using exact string matching is a very limited approach to calculating overlap. Therefore, we used heuristics and manual checking to get more precise numbers, as shown in Table TABREF39, which also includes the three corpora in addition to the three dictionaries. As the manual checking only involved judging similar idioms found in pairs of resources, we cannot calculate three-way overlap as in Figure FIGREF37. The counts of the pair-wise overlap between dictionaries differ significantly between the two methods, which serves to illustrate the limitations of using only exact string matching and the necessity of using more advanced methods and manual effort.
Several insights can be gained from the data in Table TABREF39. The relation between Wiktionary and the SemEval corpus is obvious (cf. Section SECREF12), given the 96.92% coverage. For the other dictionary-corpus pairs, the coverage increases proportionally with the size of the dictionary, except in the case of UsingEnglish and the Sporleder corpus. The proportional increase indicates no clear qualitative differences between the dictionaries, i.e. one does not have a significantly higher percentage of non-idioms than the other, when compared to the corpora.
Generally, overlap between dictionaries and corpora is low: the two biggest, ODEI and Wiktionary have only around 30% overlap, while the dictionaries also cover no more than approximately 70% of the idioms used in the various corpora. Overlap between the three corpora is also extremely low, at below 5%. This is unsurprising, since a new dataset is more interesting and useful when it covers a different set of idioms than used in an existing dataset, and thus is likely constructed with this goal in mind.
Corpus Annotation
In order to evaluate the PIE extraction methods developed in this work (Section SECREF6), we exhaustively annotate an evaluation corpus with all instances of a pre-defined set of PIEs. As part of this, we come up with a workable definition of PIEs, and measure the reliability of PIE annotation by inter-annotator agreement.
Assuming that we have a set of idioms, the main problem of defining what is and what is not a potentially idiomatic expression is caused by variation. In principle, potentially idiomatic expression is an instance of a phrase that, when seen without context, could have either an idiomatic or a literal meaning. This is clearest for the dictionary form of the idiom, as in Example SECREF5. Literal uses generally allow all kinds of variation, but not all of these variations allow a figurative interpretation, e.g. Example SECREF5. However, how much variation an idiom can undergo while retaining its figurative interpretation is different for each expression, and judgements of this might vary from one speaker to the other. An example of this is spill the bean, a variant of spill the beans, in Example SECREF5 judged by BIBREF21 as being highly questionable. However, even here a corpus example can be found containing the same variant used in a figurative sense (Example SECREF5).
As such, we assume that we cannot know a priori which variants of an expression allow a figurative reading, and are thus a potentially idiomatic expression. Therefore we consider every possible morpho-syntactic variation of an idiom a PIE, regardless of whether it actually allows a figurative reading. We believe the boundaries of this variation can only be determined based on corpus evidence, and even then they are likely variable.
Note that a similar question is tackled by BIBREF26, when they establish the boundary between a `literal reading of a VMWE' and a `coincidental co-occurrence'. BIBREF26's answer is similar to ours, in that they count something as a literal reading of a VMWE if it `the same or equivalent dependencies hold between [the expression]'s components as in its canonical form'.
. John kicked the bucket last night.
. * The bucket, John kicked last night.
. ?? Azin spilled the bean. (from BIBREF21)
. Alba reveals Fantastic Four 2 details The Invisible Woman actress spills the bean on super sequel (from ukWaC)
Corpus Annotation ::: Evaluating the Extraction Methods
Evaluating the extraction methods is easier than evaluating dictionary coverage, since the goal of the extraction component is more clearly delimited: given a set of PIEs from one or more dictionaries, extract all occurrences of those PIEs from a corpus. Thus, rather than dealing with the undefined set of all PIEs, we can work with a clearly defined and finite set of PIEs from a dictionary.
Because we have a clearly defined set of PIEs, we can exhaustively annotate a corpus for PIEs, and use that annotated corpus for automatic evaluation of extraction methods using recall and precision. This allows us to facilitate and speed up annotation by pre-extracting sentences possibly containing a PIE. After the corpus is annotated, the precision and recall can be easily estimated by comparing the extracted PIE instances to those marked in the corpus. The details of the corpus selection, dictionary selection, extraction heuristic and annotation procedure are presented in Section SECREF46, and the details and results of the various extraction methods are presented in Section SECREF6.
Corpus Annotation ::: Base Corpus and Idiom Selection
As a base corpus, we use the XML version of the British National Corpus BIBREF37, because of its size, variety, and wide availability. The BNC is pre-segmented into s-units, which we take to be sentences, w-units, which we take to be words, and c-units, punctuation. We then extract the text of all w-units and c-units. We keep the sentence segmentation, resulting in a set of plain text sentences. All sentences are included, except for sentences containing <gap> elements, which are filtered out. These <gap> elements indicate places where material from the original has been left out, e.g. for anonymisation purposes. Since this can result in incomplete sentences that cannot be parsed correctly, we filter out sentences containing these gaps.
We use only the written part of the BNC. From this, we extract a set of documents with the aim of having as much genre variation as possible. To achieve this, we select the first document in each genre, as defined by the classCode attribute (e.g. nonAc, commerce, letters). The resulting set of 46 documents makes up our base corpus. Note that these documents vary greatly in size, which means the resulting corpus is varied, but not balanced in terms of size (Table TABREF43). The documents are split across a development and test set, as specified at the end of Section SECREF46. We exclude documents with IDs starting with A0 from all annotation and evaluation procedures, as these were used during development of the extraction tool and annotation guidelines.
As for the set of potentially idiomatic expressions, we use the intersection of the three dictionaries, Wiktionary, Oxford, and UsingEnglish. Based on the assumption that, if all three resources include a certain idiom, it must unquestionably be an idiom, we choose the intersection (also see Figure FIGREF37). This serves to exclude questionable entries, like at all, which is in Wiktionary. The final set of idioms used for these experiments consists of 591 different multiword expressions. Although we aim for wide coverage, this is a necessary trade-off to ensure quality. At the same time, it leaves us with a set of idiom types that is approximately ten times larger than present in existing corpora. The set of 591 idioms includes idioms with a large variety of syntactic patterns, of which the most frequent ones are shown in Table TABREF44. The statistics show that the types most prevalent in existing corpora, verb-noun and preposition-noun combinations, are indeed the most frequent ones, but that there is a sizeable minority of types that do not fall into those categories, including coordinated adjectives, coordinated nouns, and nouns with prepositional phrases. This serves to emphasise the necessity of not restricting corpora to a small set of syntactic patterns.
Corpus Annotation ::: Extraction of PIE Candidates
To annotate the corpus completely manually would require annotators to read the whole corpus, and cross-reference each sentence to a list of almost 600 PIEs, to check whether one of those PIEs occurs in a sentence. We do not consider this a feasible annotation settings, due to both the difficulty of recognising literal usages of idioms and the time cost needed to find enough PIEs, given their low overall frequency. As such, we use a pre-extraction step to present candidates for annotation to the human annotators.
Given the corpus and the set of PIEs, we heuristically extract the PIE candidates as follows: given an idiomatic expression, extract every sentence which contains all the defining words of the idiom, in any form. This ensures that all possibly matching sentences get extracted, while greatly pruning the amount of sentences for annotators to look at. In addition, it allows us to present the heuristically matched PIE type and corresponding words to the annotators, which makes it much easier to judge whether something is a PIE or not. This also means that annotators never have to go through the full list of PIEs during the annotation process.
Initially, the heuristic simply extracted any sentence containing all the required words, where a word is any of the inflectional variants of the words in the PIE, except for determiners and punctuation. This method produced large amounts of noise, that is, a set of PIE candidates with only a very low percentage of actual PIEs. This was caused by the presence of some highly frequent PIEs with very little defining lexical content, such as on the make, and in the running. For example, with the original method, every sentence containing the preposition on, and any inflectional form of the verb make was extracted, resulting in a huge number of non-PIE candidates.
To limit the amount of noise, two restrictions were imposed. The first restrictions disallows word order variation for PIEs which do not contain a verb. The rationale behind this is that word order variation is only possible with PIEs like spill the beans (e.g. the beans were spilled), and not with PIEs like in the running (*the running in??). The second restriction is that we limit the number of words that can be inserted between the words of a PIE, but only for PIEs like on the make, and in the running, i.e. PIEs which only contain prepositions, determiners and a single noun. The number of intervening words was limited to three tokens, allowing for some variation, as in Example SECREF45, but preventing sentences like Example SECREF45 from being extracted. This restriction could result in the loss of some PIE candidates with a large number of intervening words. However, the savings in annotation time clearly outweigh the small loss in recall in this situation.
. Either at New Year or before July you can anticipate a change in the everyday running of your life. (in the running - BNC - document CBC - sentence 458)
. [..] if [he] hung around near the goal or in the box for that matter instead of running all over the show [..] (in the running - BNC - document J1C - sentence 1341)
Corpus Annotation ::: Annotation Procedure
The manual annotation procedure consists of three different phases (pilot, double annotation, single annotation), followed by an adjudication step to resolve conflicting annotations. Two things are annotated: whether something is a PIE or not, and if it is a PIE, which sense the PIE is used in. In the first phase (0-100-*), we randomly select hundred of the 2,239 PIE candidates which are then annotated by three annotators. All annotators have a good command of English, are computational linguists, and familiar with the subject. The annotators include the first and last author of this paper.
The annotators were provided with a short set of guidelines, of which the main rule-of-thumb for labelling a phrase as a PIE is as follows: any phrase is a PIE when it contains all the words, with the same part-of-speech, and in the same grammatical relations as in the dictionary form of the PIE, ignoring determiners.
For sense annotation, annotators were to mark a PIE as idiomatic if it had a sense listed in one of the idiom dictionaries, and as literal if it had a meaning that is a regular composition of its component words. For cases which were undecidable due to lack of context, the ?-label was used. The other-label was used as a container label for all cases in which neither the literal or idiomatic sense was correct (e.g. meta-linguistic uses and embeddings in metaphorical frames, see also Section SECREF10).
The first phase of annotation serves to bring to light any inconsistencies between annotators and fill in any gaps in the annotation guidelines. The resulting annotations already show a reasonably high agreement of 0.74 Fleiss' Kappa. Table TABREF48 shows annotation details and agreement statistics for all three phases. The annotation tasks suffixed by -PIE indicate agreement on PIE/non-PIE annotation and the tasks suffixed by -sense indicate agreement on sense annotation for PIEs.
In the second phase of annotation (100-600-* & 600-1100-*), another 1000 of the 2239 PIE candidates are selected to be annotated by two pairs of annotators. This shows very high agreement, as shown in Table TABREF48. This is probably due to the improvement in guidelines and the discussion following the pilot round of annotation. The exception to this are the somewhat lower scores for the 600-1100-sense annotation task. Adjudication revealed that this is due almost exclusively because of a different interpretation of the literal and idiomatic senses of a single PIE type: on the ground. Excluding this PIE type, Fleiss' Kappa increases from 0.63 to 0.77.
Because of the high agreement on PIE annotation, we deem it sufficient for the remainder (1108 candidates) to be annotated by only the primary annotator in the third phase of annotation (1100-2239-*). The reliability of the single annotation can be checked by comparing the distribution of labels to the multi-annotated parts. This shows that it falls clearly within the ranges of the other parts, both in the proportion of PIEs and idiomatic senses (see Table TABREF49). The single-annotated part has 49.0% PIEs, which is only 4 percentage points above the 44.7% PIEs in the multi-annotated parts. The proportion of idioms is just 2 percentage points higher, with 55.9% versus 53.9.%.
Although inter-annotator agreement was high, there was still a significant number of cases in the triple and double annotated PIE candidate sets where not all annotators agreed. These cases were adjudicated through discussion by all annotators, until they were in agreement. In addition, all PIE candidates which initially received the ?-label (unclear or undecidable) for sense or PIE were resolved in the same manner. In the adjudication procedure, annotators were provided with additional context on each side of the idiom, in contrast to the single sentence provided during the initial annotation. The main reason to do adjudication, rather than simply discarding all candidates for which there was disagreement, was that we expected exactly those cases for which there are conflicting annotations to be the most interesting ones, since having non-standard properties would cause the annotations to diverge. Examples of such interesting non-standard cases are at sea as part of a larger satirical frame in Example SECREF46 and cut the mustard in Example SECREF46 where it is used in a headline as wordplay on a Cluedo character.
. The bovine heroine has connections with Cowpeace International, and deals with a huge treacle slick at sea. (at sea - BNC - document CBC - sentence 13550)
. Why not cut the Mustard? [..] WADDINGTON Games's proposal to axe Reverend Green from the board game Cluedo is a bad one. (cut the mustard - BNC - document CBC - sentence 14548)
We split the corpus at the document level. The corpus consists of 45 documents from the BNC, and we split it in such a way that the development set has 1,112 candidates across 22 documents and the test set has 1,127 candidates from 23 documents. Note that this means that the development and test set contain different genres. This ensures that we do not optimise our systems on genre-specific aspects of the data.
Dictionary-based PIE Extraction
We propose and implement four different extraction methods, of differing complexities: exact string match, fuzzy string match, inflectional string match, and parser-based extraction. Because of the absence of existing work on this task, we compare these methods to each other, where the more basic methods function as baselines. More complex methods serve to shine light on the difficulty of the PIE extraction task; if simple methods already work sufficiently well, the task is not as hard as expected, and vice versa. Below, each of the extraction methods is presented and discussed in detail.
Dictionary-based PIE Extraction ::: String-based Extraction Methods ::: Exact String Match
This is, very simply, extracting all instances of the exact dictionary form of the PIE, from the tokenized text of the corpus. Word boundaries are taken into account, so at sea does not match `that seawater'. As a result, all inflectional and other variants of the PIE are ignored.
Dictionary-based PIE Extraction ::: String-based Extraction Methods ::: Fuzzy String Match
Fuzzy string match is a rough way of dealing with morphological inflection of the words in a PIE. We match all words in the PIE, taking into account word boundaries, and allow for up to 3 additional letters at the end of each word. These 3 additional characters serve to cover inflectional suffixes.
Dictionary-based PIE Extraction ::: String-based Extraction Methods ::: Inflectional String Match
In inflectional string match, we aim to match all inflected variations of a PIE. This is done by generating all morphological variants of the words in a PIE, generating all combinations of those words, and then using exact string match as described earlier.
Generating morphological variations consists of three steps: part-of-speech tagging, morphological analysis, and morphological reinflection. Since inflectional variation only applies to verbs and nouns, we use the Spacy part-of-speech tagger to detect the verbs and nouns. Then, we apply the morphological analyser morpha to get the base, uninflected form of the word, and then use the morphological generation tool morphg to get all possible inflections of the word. Both tools are part of the Morph morphological processing suite BIBREF38. Note that the Morph tools depend on the part-of-speech tag in the input, so that a wrong PoS may lead to an incorrect set of morphological variants.
For a PIE like spill the beans, this results in the following set of variants: $\lbrace $spill the bean, spills the bean, spilled the bean, spilling the bean, spill the beans, spills the beans, spilled the beans, spilling the beans$\rbrace $. Since we generate up to 2 variants for each noun, and up to 4 variants for each verb, the number of variants for PIEs containing multiple verbs and nouns can get quite large. On average, 8 additional variants are generated for each potentially idiomatic expression.
Dictionary-based PIE Extraction ::: String-based Extraction Methods ::: Additional Steps
For all string match-based methods, ways to improve performance are implemented, to make them as competitive as possible. Rather than doing exact string matching, we also allow words to be separated by something other than spaces, e.g. nuts-and-bolts for nuts and bolts. Additionally, there is an option to take into account case distinctions. With the case-sensitive option, case is preserved in the idiom lists, e.g. coals to Newcastle, and the string matching is done in a case-sensitive manner. This increases precision, e.g. by avoiding PIEs as part of proper names, but also comes at a cost of recall, e.g. for sentence-initial PIEs. Thirdly, there is the option to allow for a certain number of intervening words between each pair of words in the PIE. This should improve recall, at the cost of precision. For example, this would yield the true positive make a huge mountain out of a molehill for make a mountain out of a molehill, but also false positives like have a smoke and go for have a go.
A third shared property of the string-based methods is the processing of placeholders in PIEs. PIEs containing possessive pronoun placeholders, such as one's and someone's are expanded. That is, we remove the original PIE, and add copies of the PIE where the placeholder is replaced by one of the possessive personal pronouns. For example, a thorn in someone's side is replaced by a thorn in {my, your, his, ...} side. In the case of someone's, we also add a wildcard for any possessively used word, i.e. a thorn in —'s side, to match e.g. a thorn in Google's side. Similarly, we make sure that PIE entries containing —, such as the mother of all —, will match any word for — during extraction. We do the same for someone, for which we substitute objective pronouns. For one, this is not possible, since it is too hard to distinguish from the one used as a number.
Dictionary-based PIE Extraction ::: Parser-Based Extraction Methods
Parser-based extraction is potentially the widest-coverage extraction method, with the capacity to extract both morphological and syntactic variants of the PIE. This should be robust against the most common modifications of the PIE, e.g. through word insertions (spill all the beans), passivisation (the beans were spilled), and abstract over articles (spill beans).
In this method, PIEs are extracted using the assumption that any sentence which contains the lemmata of the words in the PIE, in the same dependency relations as in the PIE, contains an instance of the PIE type in question. More concretely, this means that the parse of the sentence should contain the parse tree of the PIE as a subtree. This is illustrated in Figure FIGREF57, which shows the parse tree for the PIE lose the plot, parsed without context. Note that this is a subtree of the parse tree for the sentence `you might just lose the plot completely', which is shown in Figure FIGREF58. Since the sentence parse contains the parse of the PIE, we can conclude that the sentence contains an instance of that PIE and extract the span of the PIE instance.
All PIEs are parsed in isolation, based on the assumption that all PIEs can be parsed, since they are almost always well-formed phrases. However, not all PIEs will be parsed correctly, especially since there is no context to resolve ambiguity. Errors tend to occur at the part-of-speech level, where, for example, verb-object combinations like jump ship and touch wood are erroneously tagged as noun-noun compounds. An analysis of the impact of parser error on PIE extraction performance is presented in Section SECREF73. Initially, we use the Spacy parser for parsing both the PIEs and the sentences.
Next, the sentence is parsed, and the lemma of the top node of the parsed PIE is matched against the lemmata of the sentence parse. If a match is found, the parse tree of the PIE is matched against the subtree of the matching sentence parse node. If the whole PIE parse tree matches, the span ranging from the first PIE token to the last is extracted. This span can thus include words that are not directly part of the PIE's dictionary form, in order to account for insertions like ships were jumped for jump ship, or have a big heart for have a heart.
During the matching, articles (a/an/the) are ignored, and passivisation is accounted for with a special rule. In addition, a number of special cases are dealt with. These are PIEs containing someone('s), something('s), one's, or —. These words are used in PIEs as placeholders for a generic possessor (someone's/something's/one's), generic object (someone/something), or any word of the right PoS (—).
For someone's, and something's, we match any possessive pronoun, or (proper) noun + possessive marker. For one's, only possessive pronouns are matched, since this is a placeholder for reflexive possessors. For someone and something, any non-possessive pronoun or (proper) noun is matched.
For — wildcards, any word can be matched, as long as it has the right relation to the right head. An additional challenge with these wildcards is that PIEs containing them cannot be parsed, e.g. too — for words is not parseable. This is dealt with by substituting the — by a PoS-ambiguous word, such as fine, or back.
Two optional features are added to the parser-based method with the goal of making it more robust to parser errors: generalising over dependency relation labels, and generalising over dependency relation direction. We expect this to increase recall at the cost of precision. In the first no labels setting, we match parts of the parse tree which have the same head lemma and the same dependent lemma, regardless of the relation label. An example of this is Figure FIGREF60, which has the wrong relation label between up and ante. If labels are ignored, however, we can still extract the PIE instance in Figure FIGREF61, which has the correct label. In the no directionality setting, relation labels are also ignored, and in addition the directionality of the relation is ignored, that is, we allow for the reversal of heads and dependents. This benefits performance in a case like Figure FIGREF62, which has stock as the head of laughing in a compound relation, whereas the parse of the PIE (Figure FIGREF63) has laughing as the head of stock in a dobj relation.
Note that similar settings were implemented by BIBREF26, who detect literal uses of VMWEs using a parser-based method with either full labelled dependencies, unlabelled dependencies, or directionless unlabelled dependencies (which they call BagOfDeps). They find that recall increases when less restrictions on the dependencies are used, but that this does not hurt precision, as we would expect. However, we cannot draw too many conclusions from these results due to the small size of their evaluation set, which consists of just 72 literal VMWEs in total.
Dictionary-based PIE Extraction ::: Parser-Based Extraction Methods ::: In-Context Parsing
Since the parser-based method parses PIEs without any context, it often finds an incorrect parse, as for jump ship in Figure FIGREF65. As such, we add an option to the method that aims to increase the number of correct parses by parsing the PIE within context, that is, within a sentence. This can greatly help to disambiguate the parse, as in Figure FIGREF66. If the number of correct parses goes up, the recall of the extraction method should also increase. Naturally, it can also be the case that a PIE is parsed correctly without context, and incorrectly with context. However, we expect the gains to outweigh the losses.
The challenge here is thus to collect example sentences containing the PIE. Since the whole point of this work is to extract PIEs from raw text, this provides a catch-22-like situation: we need to extract a sentence containing a PIE in order to extract sentences containing a PIE.
The workaround for this problem is to use the exact string matching method with the dictionary form of the PIE and a very large plain text corpus to gather example sentences. By only considering the exact dictionary form we both simplify the finding of example sentences and the extraction of the PIE's parse from the sentence parse.
In case multiple example sentences are found, the shortest sentence is selected, since we assume it is easiest to parse. This is also the reason we make use of very large corpora, to increase the likelihood of finding a short, simple sentence. The example sentence extraction method is modified in such a way that sentences where the PIE is used meta-linguistically in quotes, e.g. “the well-known English idiom `to spill the beans' has no equivalents in other languages”, are excluded, since they do not provide a natural context for parsing. When no example sentence can be found in the corpus, we back-off to parsing the PIE without context. After a parse has been found for each PIE (i.e. with or without context), the method proceeds identically to the regular parser-based method.
We make use of the combination of two large corpora for the extraction of example sentences: the English Wikipedia, and ukWaC BIBREF17. For the Wikipedia corpus, we use a dump (13-01-2016) of the English-language Wikipedia, and remove all Wikipedia markup. This is done using WikiExtractor. The resulting files still contain some mark-up, which is removed heuristically. The resulting corpus contains mostly clean, raw, untokenized text, numbering approximately 1.78 billion tokens.
As for ukWaC, all XML-markup was removed, and the corpus is converted to a one-sentence-per-line format. UkWaC is tokenized, which makes it difficult for a simple string match method to find PIEs containing punctuation, for example day in, day out. Therefore, all spaces before commas, apostrophes, and sentence-final punctuation are removed. The resulting corpus contains approximately 2.05 billion tokens, making for a total of 3.83 billion tokens in the combined ukWaC and Wikipedia corpus.
Dictionary-based PIE Extraction ::: Results
In order to determine which of the methods described previously produces the highest quality extraction of potentially idiomatic expressions, we evaluate them, in various settings, on the corpus described in Section SECREF5.
For parser-based extraction, systems with and without in-context parsing, ignoring labels, and ignoring directionality are tested. For the three string-based extraction methods, varying numbers of intervening words and case sensitivity are evaluated. Evaluation is done using the development set, consisting of 22 documents and 1112 PIE candidates, and the test set, which consists of 23 documents and 1127 PIE candidates. For each method the best set of parameters and/or options is determined using the development set, after which the best variant by F1-score of each method is evaluated on the test set.
Since these documents in the corpus are exhaustively annotated for PIEs (see Section SECREF40), we can calculate true and false positives, and false negatives, and thus precision, recall and F1-score. The exact spans are ignored, because the spans annotated in the evaluation corpus are not completely reliable. These were automatically generated during candidate extraction, as described in Section SECREF45. Rather, we count an extraction as a true positive if it finds the correct PIE type in the correct sentence.
Note that we judge the system with the highest F1-score to be the best-performing system, since it is a clear and objective criterion. However, when using the system in practice, the best performance depends on the goal. When used as a preprocessing step for PIE disambiguation, the system with the highest F1-score is perhaps the most suitable, but as a corpus building tool, one might want to sacrifice some precision for an increase in recall. This helps to get the most comprehensive annotation of PIEs possible, without overloading the annotators with false extractions (i.e. non-PIEs), by maintaining high precision.
The results for each system on the development set are presented in Tables TABREF70 and TABREF71. Generally, results are in line with expectations: (the best) parse-based methods are better than (the best) string-based methods, and within string-based methods, inflectional matching works best. The same goes for the different settings: case-sensitivity increases precision at the cost of recall, allowing intervening words increases recall at the cost of precision, and the same goes for the no labels and no directionality options for parser-based extraction. Overall, in-context parser-based extraction works best, with an F1 of 88.54%, whereas fuzzy matching does very poorly.
Within string-based methods, exact matching has the highest precision, but low recall. Fuzzy matching increases recall at a disproportionately large precision cost, whereas inflectional matching combines the best of both worlds and has high recall at a small loss in precision. For the parser-based system, it is notable that parsing idioms within context yields a clear overall improvement by greatly improving recall at a small cost in precision.
We evaluate the best variant of each system, as determined by F1-score, on the test set. This gives us an indication of whether the system is robust enough, or was overfitted on the development data. Results on the test set are shown in Table TABREF72. On average, the results are lower than the results on the development set. The string-based methods perform clearly worse, with drops of about 4% F1-score for exact and inflectional match, and a large drop of almost 9% F1-score for fuzzy matching. The parser-based method, on the other hand, is more robust, with a small 0.59% increase in F1-score on the test set.
Dictionary-based PIE Extraction ::: Analysis
Broadly speaking, the PIE extraction systems presented above perform in line with expectations. It is nevertheless useful to see where the best-performing system misses out, and where improvements like in-context parsing help performance.
We analyse the shortcomings of the in-context parser-based system by looking at the false positives and false negatives on the development set. We consider the output of the system with best overall performance, since it will provide the clearest picture. The system extracts 529 PIEs in total, of which 54 are false extractions (false positives), and it misses 69 annotated PIE instances (false negatives). Most false positives stem from the system's failure to capture nuances of PIE annotation. This includes cases where PIEs contain, or are part of, proper nouns (Example SECREF73), PIEs that are part of coordination constructions (Example SECREF73), and incorrect attachments (Example SECREF73). Among these errors, sentences containing proper nouns are an especially frequent problem.
. Drama series include [..] airline security thrills in Cleared For Takeoff and Head Over Heels [..] (in the clear - BNC - document CBC - sentence 5177)
. They prefer silk, satin or lace underwear in tasteful black or ivory. (in the black - BNC - document CBC - sentence 14673)
. [..] `I saw this chap make something out of an ordinary piece of wood — he fashioned it into an exquisite work of art.' (out of the woods - BNC - document ABV - sentence 1300)
The main cause of false negatives are errors made by the parser. In order to correctly extract a PIE from a sentence, both the PIE and the sentence have to be parsed correctly, or at least parsed in the same way. This means a missed extraction can be caused by a wrong parse for the PIE or a wrong parse for the sentence. These two error types form the largest class of false negatives. Since some PIE types are rather frequent, a wrong parse for a single PIE type can potentially lead to a large number of missed extractions.
It is not surprising that the parser makes many mistakes, since idioms often have unusual syntactic constructions (e.g. come a cropper) and contain words where default part-of-speech tags lead to the wrong interpretation (e.g. round is a preposition in round the bend, not a noun or adjective). This is especially true when idioms are parsed without context, and hence, where in-context parsing provides the largest benefit: the number of PIEs which are parsed incorrectly drops, which leads to F1-scores on those types going from 0% to almost 100% (e.g. in light of and ring a bell). Since parser errors are the main contributor to false negatives, hurting recall, we can observe that parsing idioms in context serves to benefit only recall, by 7 percentage points, at only a small loss in precision.
We find that adding context mainly helps for parsing expressions which are structurally relatively simple, but still ambiguous, such as rub shoulders, laughing stock, and round the bend. Compare, for example, the parse trees for laughing stock in isolation and within the extracted context sentence in Figures FIGREF74 and FIGREF75. When parsed in isolation, the relation between the two words is incorrectly labelled as a compound relation, whereas in context it is correctly labelled as a direct object relation. Note however, that for the most difficult PIEs, embedding them in a context does solve the parsing problem: a syntactically odd phrase is hard to phrase (e.g. for the time being), and a syntactically odd phrase in a sentence makes for a syntactically odd sentence that is still hard to parse (e.g. `London for the time being had been abandoned.'). Finding example sentences turned out not to be a problem, since appropriate sentences were found for 559 of 591 PIE types.
An alternative method for reducing parser error is to use a different, better parser. The Spacy parser was mainly chosen for implementation convenience and speed, and there are parsers which have better performance, as measured on established parsing benchmarks. To investigate the effectiveness of this method, we used the Stanford Neural Dependency Parser BIBREF39 to extract PIEs in the regular parsing, in-context parsing and the no labels settings. In all cases, using the Stanford parser yielded worse extraction performance than the Spacy parser. A possible explanation for why a supposedly better parser performs worse here is that parsers are optimised and trained to do well on established benchmarks, which consist of complete sentences, often from news texts. This does not necessarily correlate with parsing performance on short (sentences containing) idiomatic phrases. As such, we cannot assume that better overall parsing performance implies PIE extraction performance.
It should be noted that, when assessing the quality of PIE extraction performance, the parser-based methods are sensitive to specific PIE types. That is, if a single PIE type is parsed incorrectly, then it is highly probable that all instances of that type are missed. If this type is also highly frequent, this means that a small change in actual performance yields a large change in evaluation scores. Our goal is to have a PIE extraction system that is robust across all PIE types, and thus the current evaluation setting does not align exactly with our aim.
Splitting out performance per PIE type reveals whether there is indeed a large variance in performance across types. Table TABREF76 shows the 25 most frequent PIE types in the corpus, and the performance of the in-context-parsing-based system on each. Except two cases (in the black and round the bend), we see that the performance is in the 80–100% range, even showing perfect performance on the majority of types.
For none of the types do we see low precision paired with high recall, which indicates that the parser never matches a highly frequent non-PIE phrase. For the system with the no labels and no-directionality options (per-type numbers not shown here), however, this does occur. For example, ignoring the labels for the parse of the PIE have a go leads to the erroneous matching of many sentences containing a form of have to go, which is highly frequent, thus leading to a large drop in precision.
Although performance is stable across the most frequent types, among the less frequent types it is more spotty. This hurts overall performance, and there are potential gains in mitigating the poor performance on these types, such as for the time being. At the same time, the string matching methods show much more stable performance across types, and some of them do so with very high precision. As such, a combination of two such methods could boost performance significantly. If we use a high-precision string match-based method, such as the exact string match variant with a precision of 97.35%, recall could be improved for the wrongly parsed PIE types, without a significant loss of precision.
We experiment with two such combinations, by simply taking the union of the sets of extracted idioms of both systems, and filtering out duplicates. Results are shown in Table TABREF77. Both combinations show the expected effect: a clear gain in recall at a minimal loss in precision. Compared to the in-context-parsing-based system, the combination with exact string matching yields a gain in recall of over 6%, and the combination with inflectional string matching yields an even bigger gain of almost 8%, at precision losses of 0.6% and 0.8%, respectively. This indicates that the systems are very much complementary in the PIEs they extract. It also means that, when used in practice, combining inflectional string matching and parse-based extraction is the most reliable configuration.
Conclusions and Outlook
We present an in-depth study on the automatic extraction of potentially idiomatic expressions based on dictionaries. The purpose of automatic dictionary-based extraction is, on the one hand, to function as a pre-extraction step in the building of a large idiom-annotated corpus. On the other hand, it can function as part of an idiom extraction system when combined with a disambiguation component. In both cases, the ultimate goal is to improve the processing of idiomatic expressions within NLP. This work consists of three parts: a comparative evaluation of the coverage of idiom dictionaries, the annotation of a PIE corpus, and the development and evaluation of several dictionary-based PIE extraction methods.
In the first part, we present a study of idiom dictionary coverage, which serves to answer the question of whether a single idiom dictionary, or a combination of dictionaries, can provide good coverage of the set of all English idioms. Based on the comparison of dictionaries to each other, we estimate that the overlap between them is limited, varying from 20% to 55%, which indicates a large divergence between the dictionaries. This can be explained by the fact that idioms vary widely by register, genre, language variety, and time period. In our case, it is also likely that the divergence is caused partly by the gap between crowdsourced dictionaries on the one hand, and a dictionary compiled by professional lexicographers on the other. Given these factors, we can conclude that a single dictionary cannot provide even close to complete coverage of English idioms, but that by combining dictionaries from various sources, significant gains can be made. Since `English idioms' are a diffuse and constantly changing set, we have no gold standard to compare to. As such, we conclude that multiple dictionaries should be used when possible, but that we cannot say any anything definitive on the coverage of dictionaries with regard to the complete set of English idioms (which can only be approximated in the first place). A more comprehensive of idiom resources could be made in the future by using more advanced automatic methods for matching, for example by using BIBREF32's (BIBREF32) method for measuring expression variability. This would make it easier to evaluate a larger number of dictionaries, since no manual effort would be required.
In the second part, we experiment with the exhaustive annotation of PIEs in a corpus of documents from the BNC. Using a set of 591 PIE types, much larger and more varied than in existing resources, we show that it is very much possible to establish a working definition of PIE that allows for a large amount of variation, while still being useful for reliable annotation. This resulted in high inter-annotator agreement, ranging from 0.74 to 0.91 Fleiss' Kappa. This means that we can build a resource to evaluate a wide-range idiom extraction system with relatively little effort. The final corpus of PIEs with sense annotations is publicly available consists of 2,239 PIE candidates, of which 1,050 actual PIEs instances, and contains 278 different PIE types.
Finally, several methods for the automatic extraction of PIE instances were developed and evaluated on the annotated PIE corpus. We tested methods of differing complexity, from simple string match to dependency parse-based extraction. Comparison of these methods revealed that the more computationally complex method, parser-based extraction, works best. Parser-based extraction is especially effective in capturing a larger amount of variation, but is less precise than string-based methods, mostly because of parser error. The best overall setting of this method, which parses idioms within context, yielded an F1-score of 89.13% on the test set. Parser error can be partly compensated by combining the parse-based method and the inflectional string match method, which yields an F1-score of 92.01% (on the development set). This aligns well with the findings by BIBREF27, who found that combining simpler and more complex methods improves over just using a simple method case for extracting verb-particle constructions. This level of performance means that we can use the tool in corpus building. This greatly reduces the amount of manual extraction effort involved, while still maintaining a high level of recall. We make the source code for the different systems publicly available. Note that, although used here in the context of PIE extraction, our methods are equally applicable to other phrase extraction tasks, for example the extraction of light-verb constructions, metaphoric constructions, collocations, or any other type of multiword expression (cf. BIBREF27, BIBREF25, BIBREF26). Similarly, our method can be conceived as a blueprint and extended to languages other than English. For this to be possible, for any given new language one would need a list of target expressions and, in the case of the parser-based method, a reliable syntactic parser. If this is not the case, the inflectional matching method can be used, which requires only a morphological analyser and generator. Obviously, for languages that are morphologically richer than English, one would need to develop strategies aimed at controlling non-exact matches, so as to enhance recall without sacrificing precision. Previous work on Italian, for example, has shown the feasibility of achieving such balance through controlled pattern matching BIBREF40. Languages that are typologically very different from English would obviously require a dedicated approach for the matching of PIEs in corpora, but the overall principles of extraction, using language-specific tools, could stay the same.
Currently, no corpora containing annotation of PIEs exist for languages other than English. However, the PARSEME corpus BIBREF19 already contains idioms (only idiomatic readings) for many languages and would only need annotation of literal usages of idioms to make up a set of PIEs. Paired with the Universal Dependencies project BIBREF41, which increasingly provides annotated data as well as processing tools for an ever growing number of languages, this seems an excellent starting point for creating PIE resources in multiple languages.

Question: How big PIE datasets are obtained from dictionaries?
Answer: 46 documents makes up our base corpus

Question: What compleentary PIE extraction methods are used to increase reliability further?
Answer: exact string matching, inflectional string matching

Question: Are PIEs extracted automatically subjected to human evaluation?
Answer: Yes

Question: What dictionaries are used for automatic extraction of PIEs?
Answer: Wiktionary, Oxford Dictionary of English Idioms, UsingEnglish.com (UE), Sporleder corpus, VNC dataset, SemEval-2013 Task 5 dataset