Automatic extraction of Word Combinations from corpora: evaluating methods and benchmarks
Authors: Malvina Nissim, Sara Castagnoli, Francesca Masini, Gianluca E. Lebani, Lucia C. Passaro, Alessandro Lenci (University of Groningen, Università di Bologna, Università di Pisa, 2015)
Topic: Lexical and Semantic Resources and Analysis

Abstract
We report on three experiments aimed at comparing two popular methods for the automatic extraction of Word Combinations from corpora, with a view to evaluate: i) their efficacy in acquiring data to be included in a combinatory resource for Italian; ii) the impact of different types of benchmarks on the evaluation itself.

Introduction
We use the term Word Combinations (WoCs) to encompass both Multiword Expressions, namely WoCs characterised by different degrees of fixedness and idiomaticity, such as idioms, phrasal lexemes, collocations, preferred combinations AUTHOR, and the distributional properties of a word at a more abstract level (argument structure, subcategorization frames, selectional preferences). Currently, apart from purely statistical approaches, the most common methods for the extraction of WoCs involve searching a corpus via sets of patterns and then ranking the extracted candidates according to various association measures (AMs) in order to distinguish meaningful combinations from sequences of words that do not form any kind of relevant unit AUTHOR. Generally, the search is performed for either shallow morphosyntactic (POS) patterns (P-based approach) or syntactic dependency relations (S-based approach) AUTHOR. While P-based approaches have shown to yield satisfactory results for relatively fixed, short and adjacent WoCs, it has been suggested that syntactic dependencies might be more helpful to capture discontinuous and syntactically flexible WoCs AUTHOR. The two methods intuitively seem to be highly complementary rather than competing with one another, and attempts are currently being proposed to put them together AUTHOR. In previous work AUTHOR, we compared the performance of the two methods against two benchmarks (a dictionary and expert judgments), showing that the two methods are indeed complementary and that automatic extraction from corpora adds a high number of WoCs that are not recorded in manually compiled dictionaries. As an extension of that work, in this paper we shift the focus of investigation by addressing the following research questions: What is the effect of different benchmarks when evaluating an extraction method? What do our results tell us about the bechmarks themselves? And, as a byproduct, can experts / laypeople be exploited to populate a lexicographic combinatory resource for Italian?

Conclusion
As for extraction methods per se, we observed that recall against a manually compiled WoC dictionary is good for both EXTra and LexIt, and, especially, that the two systems are complementary. In the human evaluation performed by experts, 40% of WoCs automatically extracted with EXTra and LexIt are deemed valid, and more than half of these are not attested in DiCi. We can thus say that data from corpora proves to be very fruitful, especially if we use the two methods complementarily. As for benchmarks, we observed that the dictionary we have evaluated is not an exhaustive resource, and should be complemented with corpusextracted WoCs. We also observed that expert- and laypeople-based evaluations differ, which raises a number of interesting, albeit puzzling questions. Overall, it seems that the notion of WoC, as well as of idiomaticity, is quite a complex one to grasp for non-linguists: the collection of judgments took quite a long time to be completed (much more than we expected) and evaluators explicitly regarded the task and the instructions as particularly complex. The results of our experiments thus leave us with a sort of methodological conundrum, as both a dictionary-based gold standard and a humanbased evaluation have limitations. Using experts not only makes the evaluation expensive, but also little ecological, as it is standard practice in psycholinguistics and computational linguistics to resort to laypeople judgments. The fact that evaluating WoCs isn’t easy for laypeole may cast some shadows on the concept of WoC itself. This suggests that improving extraction methods must go hand in hand with the theoretical effort of making the very notion of WoC more precise, in order to make it an experimentally solid and testable notion.