SlowGuess/NAACL_2022_OCR huggingface dataset:
Simone Tedeschi $^{1,2}$ , Federico Martelli $^{2}$  and Roberto Navigli $^{2}$
1Babelscape, Italy
$^{2}$ Sapienza NLP Group, Sapienza University of Rome
tedeschi@babelscape.com, martelli@di.uniromal.it,
navigli@diag.uniromal.it
Idioms are phrases which present a figurative meaning that cannot be (completely) derived by looking at the meaning of their individual components. Identifying and understanding idioms in context is a crucial goal and a key challenge in a wide range of Natural Language Understanding tasks. Although efforts have been undertaken in this direction, the automatic identification and understanding of idioms is still a largely underinvestigated area, especially when operating in a multilingual scenario. In this paper, we address such limitations and put forward several new contributions: we propose a novel multilingual Transformer-based system for the identification of idioms; we produce a high-quality automatically-created training dataset in 10 languages, along with a novel manually-curated evaluation benchmark; finally, we carry out a thorough performance analysis and release our evaluation suite at https://github.com/Babelscope/ID10M.
Idioms pertain to a wider family of linguistic phenomena referred to as multi-word expressions (MWEs). Broadly speaking, an MWE can be defined as a combination of two or more words, behaving as a complex lexical unit and showing idiosyncratic properties (Baldwin and Kim, 2010). Over the course of the last few years, several attempts have been made to classify MWEs based on specific dimensions such as polylexicality, fixedness, compositionality and idiomaticity (Sailer and Markantonatou, 2018). According to Sag et al. (2002), MWEs can be divided into lexicalized and institutionalized phrases. While the former show syntactic or semantic idiosyncrasies, e.g. kingdom come and spill the beans, the latter are compositional from a syntactic and semantic perspective, but statistically idiosyncratic, e.g. traffic light and telephone booth.
Among lexicalized phrases, idioms are of particular interest in that their meaning cannot be obtained by compositionally interpreting their word constituents. These include non-compositional phrases, e.g. kick the bucket, and partially compositional phrases, e.g. rain cats and dogs (Nunberg et al., 1994).
Given their complex nature, idioms are hard to be automatically identified and pose a crucial challenge to the entire field of Natural Language Understanding (NLU). Although research in this field has recently achieved great advancements, the current formulation of many tasks tends to overlook the idiomatic usage of language. Instead, idioms ought to be playing an important role in NLU as they are a frequent phenomenon which can be observed in all languages. The correct identification of idioms in context is crucial for tasks such as Word Sense Disambiguation (Bevilacqua et al., 2021) and Entity Linking (Sevgili et al., 2020), but also for many downstream applications. For instance, in Question Answering or dialog, a system must be able to understand "It was a piece of cake" in relation to the question "How was the test?" (Jhamtani et al., 2021; Mishra and Jain, 2016). Similarly, if the idiom kick the bucket is identified, then a Text Summarization system would be able to summarize all its occurrences within a text with "die" (Chu and Wang, 2018; Gambhir and Gupta, 2017). Finally, once an idiom is identified, a Machine Translation system would then be able to avoid its compositional translation, and treat it as a whole (Anastasiou, 2010). Furthermore, idioms are widely studied in linguistics and psycholinguistics (Cacciari and Tabossi, 1988; Gibbs Jr, 1992; Nunberg et al., 1994; Cacciari and Tabossi, 2014; Liu, 2017), hence a system capable of effectively identifying idioms in texts would significantly improve many research areas, far beyond NLU.
Most of the past idiom extraction strategies focused on specific domains and on a limited number
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2715 - 2726
of languages. In our work, we tackle these shortcomings and, taking inspiration from the Named Entity Recognition (NER) task (Yadav and Bethard, 2018), we reformulate the identification of idioms as a sequence labeling task. Specifically, we propose the following new contributions:
- We design a novel multilingual Transformer-based system for the identification of idioms;
- We release a high-quality silver training dataset in 10 languages and a novel manually-curated evaluation benchmark in 4 languages;
- We measure the quality of the data produced and of our system design through an extensive evaluation.
We hope that this work will provide a stepping stone for further studies regarding idiomatic expressions and their applications, and encourage further work on the identification of idioms in multiple languages. We release the produced datasets and software at https://github.com/Babelscape/ID10M.
Systems Over the course of the past two decades, several approaches have been put forward to address the idiom identification task. To this end, two main properties of idioms have been leveraged, namely their syntactic and semantic idiosyncrasies. While the former refers to the peculiar syntactic behaviour of idioms, the latter indicates the linguistic property in which the meaning of an idiomatic expression cannot be completely derived from the meaning of its individual components.
Initial studies regarding idiom identification focused on syntactic idiosyncrasy, concentrating on verb/noun idioms, e.g. shoot the breeze (Fazly and Stevenson, 2006; Cook et al., 2007; Diab and Bhutada, 2009), on verb/particle idioms, e.g. call off (Ramisch et al., 2008) or on idioms satisfying specific restrictions, i.e. subject/verb such as tension mounted and verb/direct-object, e.g. break the ice (Shutova et al., 2010).
Subsequent approaches exploited semantic idiosyncrasies. This property implies that idiomatic expressions often occur in contexts typically unrelated to the meaning of their individual constituents, thus providing a key feature to be exploited in an automatic approach. In particular, Muzny and Zettlemoyer (2013) introduced new lexical and graph
based features that use WordNet $^{1}$  and Wiktionary $^{2}$ , and proposed a simple yet efficient binary Perceptron classifier to distinguish between idiomatic and non-idiomatic expressions by exploiting their components and dictionary definitions. A similar, but unsupervised approach was adopted by Verma and Vuppuluri (2015) which relied on the dictionary definitions of each component of a given idiom.
These latter methods have more recently been superseded by approaches making use of distributional similarity in the form of both static and contextualized word embeddings (Gharbieh et al., 2016; Ehren, 2017; Senaldi et al., 2019; Hashempour and Villavicencio, 2020; Fakharian, 2021; Garcia et al., 2021; Nedumpozhimana and Kelleher, 2021), while keeping the underlying assumption unchanged: the vector representation of the component words should be distant from the vector representation of the context or of the expression as a whole.
Notwithstanding the recent improvements, to the best of our knowledge, the identification of idiomatic expressions in multiple languages is largely under-investigated.
Datasets In the early 2000s, several datasets for idiom identification were created. For instance, Cook et al. (2008) and Sporleder et al. (2010) manually selected a limited number of idioms, and then extracted sentences containing such idioms from the British National Corpus (BNC, Consortium et al., 2007). Similarly, Sporleder and Li (2009) extracted a dataset from the Gigaword corpus (Graff and Christopher, 2003). Street et al. (2010), instead, used multiple annotators to validate sentences from the American National Corpus (ANC, Ide and Macleod, 2001). Additionally, Muzny and Zettlemoyer (2013) created a dataset by applying the aforementioned classifier on Wiktionary entries, more than doubling the number of idiomatic expressions in Wiktionary.
Furthermore, Korkontzelos et al. (2013) introduced Task 5b at SemEval-2013 regarding the detection of semantic compositionality in context. The authors selected idioms from Wiktionary, and extracted instances from the ukWaC corpus (Ferraresi et al., 2008). Schneider et al. (2016), instead, proposed the DiMSUM dataset for Task 10 at SemEval-2016, and extracted annotations from reviews, tweets and TED talks. However, this work
'https://wordnet.princeton.edu/
$^{2}$ https://www.wiktionary.org/
did not categorise MWEs into subtypes, making it difficult to quantify the number of idioms in the corpus.
Finally, Peng et al. (2015) expanded the dataset introduced by Cook et al. (2008) by retrieving further sentences from the BNC corpus, while more recently Gong et al. (2017) introduced a small-scale dataset derived from Google Books<sup>3</sup> for English and Chinese.
Unfortunately, almost all the aforementioned approaches focused on English. The first concrete attempt to scale to multiple languages was made by Madabushi et al. (2021) who also proposed a SemEval-2022 task on idiom identification. Nevertheless, their datasets are limited in size and they only cover three languages, namely English, Portuguese and Galician.
3 ID10M
In what follows, we first describe the creation process of our training datasets (Section 3.1) and the manually-curated test sets (Section 3.2). Then, we introduce our new task formulation and illustrate the architecture of our idiom identification system (Section 3.3).
3.1 Silver-Standard Data Creation
Automatic Annotation In order to create our training data, we exploit Wiktionary<sup>4</sup> as the main source, as it provides access to a large number of MWEs along with usage examples in multiple languages. However, since such examples are provided for a limited number of MWEs, we search for further textual contexts in a large external source, namely WikiMatrix<sup>5</sup> (Schwenk et al., 2021), a multilingual corpus that covers 83 languages and contains parallel sentences extracted from Wikipedia<sup>6</sup>.
We perform data extraction as follows. Let  $E_{l}$  be the set of MWEs available in Wiktionary in the language  $l$ , with  $|E_{l}| = n$ , and let us define the function  $L(p)$  that, given a phrase  $p$ , outputs its lemma. Then, we apply a heuristic which allows
us, for each expression  $e_i \in E_l$ , to search for a sentence in WikiMatrix such that there exists at least a span of tokens  $S_{k-j}$  starting at index  $k$  and ending at index  $j$ , where  $e_i = S_{k-j} \vee e_i = L(S_{k-j}) \vee L(e_i) = S_{k-j} \vee L(e_i) = L(S_{k-j})$ . By applying this heuristic, not only do we obtain a large set of sentences containing potentially idiomatic expressions (PIEs), but - thanks to the lemmatization step - we also collect several morphological variations of the original expressions in  $E_l$ , e.g. starting from 'kick the bucket', we also obtain 'kicked the bucket' and 'kicks the bucket'. In particular, if an MWE is marked as idiomatic in Wiktionary, we mark all its occurrences as idiomatic too. Similarly, if an MWE is not marked as idiomatic in Wiktionary, we mark all its occurrences as literal. However, this has a limitation: if an MWE is labeled as idiomatic (or literal) in Wiktionary, it will not necessarily always also be idiomatic (or literal) in the WikiMatrix sentences in which it appears.
We adopt the above-described procedure to create datasets in the following 10 languages: Chinese, Dutch, English, French, German, Italian, Japanese, Polish, Portuguese and Spanish.
Automatic Validation Since the data derived from Wiktionary and WikiMatrix may contain errors, we aim at automatically improving their quality. To achieve this goal, we exploit the semantic idiosyncrasy property of idiomatic expressions, and the consequent fact that the meaning of the individual constituents of idiomatic expressions are unrelated to the surrounding context. Specifically, following this intuition, and by taking inspiration from recent advances in the main disambiguation tasks (Blevins and Zettlemoyer, 2020; Botha et al., 2020; Tedeschi et al., 2021), we design a dual-encoder architecture (Figure 1) to produce a vector representation for both the expression and its context, and then, based on their cosine similarity, label the expression as idiomatic or literal.
More formally, let us define an expression encoder  $\Psi$  and a context encoder  $\Omega$ . Then, given an expression-context pair  $\langle e, c \rangle$ , the output of the dual-encoder architecture  $\Phi$  is defined as follows:
$$
\Phi (e, c) = \left\{ \begin{array}{l l} 1, & \text {i f} \frac {\Psi (e) ^ {T} \Omega (c)}{\| \Psi (e) \| \| \Omega (c) \|} \leq \delta \\ 0, & \text {o t h e r w i s e} \end{array} \right.
$$
where  $\Phi(e, c) = 1$  means that  $e$  is idiomatic in  $c$ , while  $\Phi(e, c) = 0$  if  $e$  has a literal meaning
<sup>3</sup>https://books.google.com/
4We employ the Wiktxtract library to collect the necessary data from Wiktionary. WikiExtract (https://pypi.org/project/wiktxtract/) provides a preprocessed version of the Wiktionary dump together with useful APIs.
$^{5}$ https://github.com/facebookresearch/LASER/tree/main/tasks/WikiMatrix
<sup>6</sup>The encyclopedia-style prose of Wikipedia could have a lower idiom density compared to other textual sources, but the large size of WikiMatrix should balance this lower density.
Figure 1: Graphical representation of the dual-encoder architecture given as input an example sentence. "E" stands for Embedding. A potentially idiomatic expression  $e$  is labeled as idiomatic when the cosine similarity score between the representations  $\Omega(c)$  and  $\Psi(e)$ , where  $c$  is the surrounding context, is lower than the threshold  $\delta$ .
in  $c$ .  $\delta$  is a manually-tuned threshold. Both encoders are bert-base-multilingual-cased architectures that take as input the tokenized versions of expressions and their contexts, respectively, surrounded by the special tokens [CLS] and [SEP]. To encode an expression, we take the sum of the individual representations of all its subwords. Instead, for the representation of the context we take the representation of the [CLS] token. We evaluate the quality of our dual encoder in Section 4.3.
Additionally, to further improve the quality of the annotations produced, we follow the recent findings of Tedeschi and Navigli (2022) which demonstrated how NER can be exploited to better discriminate between idiomatic and literal usages of potentially idiomatic expressions.
3.2 Gold-Standard Data Creation
To evaluate the performance of our idiom identification system, we manually create a novel evaluation benchmark in 4 languages, i.e. English, German, Italian and Spanish. As explained in Section 3.1, we start by producing a set of sentences containing PIEs. Then, to properly label the expressions, depending on the context in which they occur, we
ask professional annotators<sup>7</sup> to perform the following binary classification task: given a context-expression pair  $\langle e, c \rangle$ , the goal is to tag this pair with a label  $y \in \{Idiomatic, Literal\}$ . In order to make our gold standard more challenging, and better evaluate the system performance, we also ask annotators to include unseen idioms, i.e. idioms that do not appear in the training set.
3.3 Idiom Identification
Task Formulation Current and past approaches to idiom identification typically take expressions-context pairs  $\langle e, c \rangle$  as input and limit themselves to determining whether  $e$  is used with a figurative meaning or not in  $c$  (Madabushi et al., 2021; Muzny and Zettlemoyer, 2013). However, this formulation has a major drawback: potentially idiomatic expressions need to be pre-identified. Importantly, we drop this requirement and reformulate the task as a sequence-labeling task, by employing the well-known BIO tagging scheme<sup>8</sup>.
<sup>7</sup>We hired a mother-tongue professional annotator for each language.
The BIO tagging scheme (short for Beginning, Intermediate, Out) is a popular tagging scheme where the B label indicates that the corresponding token is the first token of a
Language	# Sentences	# Tokens	# Idioms	# B	# I	# O	# Seen	# Unseen	# Literal
Silver Data	Chinese (ZH)	9543	244422	1301	5272	3823	235327	-	-	3918
Dutch (NL)	20935	548872	189	4530	10543	533799	-	-	16366
English (EN)	37919	1199492	4568	10102	19884	1169506	-	-	27408
French (FR)	35588	939161	188	12112	25248	901801	-	-	23238
German (DE)	26963	722109	819	8311	11500	702298	-	-	18488
Italian (IT)	29523	813445	452	8768	12353	792324	-	-	20506
Japanese (JA)	6388	211437	165	2534	1662	207241	-	-	3852
Polish (PL)	36333	862265	648	12971	14364	834930	-	-	22467
Portuguese (PT)	30942	764017	559	5824	8871	749322	-	-	24816
Spanish (ES)	28647	648776	1229	9994	13927	624855	-	-	17851
Gold Data	English (EN)	200	3287	142	159	373	2755	62	80	41
German (DE)	200	4529	111	181	377	3971	71	40	19
Italian (IT)	200	5043	139	155	271	4617	87	52	48
Spanish (ES)	200	2240	78	133	348	1759	19	59	66
More formally, given as input a raw text sequence  $X$  of  $n$  tokens  $x_{1},\ldots ,x_{n}$ , each  $x_{i}$  must be labeled by the system with a tag  $y_{i}\in \{B,I,O\}$  for each  $i\in [1,n]$ . This formulation also allows us to easily handle multiple idiomatic expressions within the same text.
In order to use our new formulation, we convert all the datasets constructed in Section 3.1 and Section 3.2 in BIO format. Table 2 shows an example of instance labeled using the BIO tagging scheme.
Our System Our model for idiom identification is inspired by the BERT-based neural architecture of Mueller et al. (2020) used for Named Entity Recognition, however, rather than encoding a word with the first contextualized subword representation as indicated by Devlin et al. (2019), we take the mean of its subwords, as suggested by recent literature (Ács et al., 2021). Then, the resulting vectors are passed through a multi-layer sentence-level BiLSTM network, whose logits are finally fed into a CRF model, trained to maximize the log-likelihood of the span-based gold label sequences (Huang et al., 2015).
In this Section, we describe our experimental setup (Section 4.1), the datasets we use to train and evaluate our idiom identification system (Section 4.2), and the results obtained (Section 4.3).
span, in this case an idiomatic expression, the I label denotes an intermediate token of a span, and O means out of a span.
Table 1: Statistics concerning the automatically-created (Silver Data) training sets and our manually-curated test sets (Gold Data). "# Seen" represents the number of expressions in the test set already encountered in the training set, whereas "# Unseen" is the number of expressions never encountered. In the count of individual idioms (# Idioms), morphological variations of a certain idiom are mapped to the same idiom.
Token	Label
After	O
some	O
reflection	O
,	O
he	O
decided	O
to	O
bite	B-IDIOM
the	I-IDIOM
bullet	I-IDIOM
.	O
Table 2: Example of instance labeled according to the BIO tagging scheme.
We implement our idiom identification system (Section 3.3) and our dual-encoder discriminator (Section 3.1) with PyTorch (Paszke et al., 2019), using the Transformers library (Wolf et al., 2019) to load the weights of BERT-base-multilingual-cased (mBERT). We fine-tune our idiom identification system for 30 epochs with a Cross-Entropy loss criterion, adopting an early stopping strategy with a patience value of 5, Adam (Kingma and Ba, 2015) optimizer and a learning rate of  $10^{-5}$ , as standard when fine-tuning the weights of a pretrained language model. For our dual-encoder discriminator, instead, we use mBERT as feature extractor since no training data for the task were available.
Hyperparameter name	Value
number of Bi-LSTM layers	2
LSTM hidden size	256
gradient accumulation steps	4
batch size	32
learning rate	0.00001
dropout	0.5
gradient clipping	1.0
adam β1	0.9
adam β2	0.999
adam ε	1e-8
Table 3: Hyperparameter values of the reference idiom identification system used for our experiments.
Language	Accuracy
English	84.12
German	81.98
Italian	82.74
Spanish	82.55
Avg.	82.85
Table 4: Accuracy of the annotations produced by our automatic system compared to those provided by the human annotators on the 4 languages covered by our gold-standard test sets.
The entire model training is carried out on a NVIDIA GeForce RTX 3090. Each training (i.e. for each language) requires  $\sim 8\mathrm{min}$ /epoch on average, for a mean of  $\sim 20$  epochs. Table 3 shows the full list of hyperparameters.
4.2 Training, Validation and Test Data
The training and validation sets that we use in our experiments are those obtained by applying the methodology described in Section 3.1, with  $\delta = 0.4^9$ . Although we automatically produce training data in 10 languages, we report results only on the 4 languages for which manually-curated test sets are available (see Section 3.2). However, since the training data has been created with the same procedure for each of the 10 languages, similar results are expected on non-tested languages. Statistics are provided in Table 1.
Figure 2: Confusion matrix of the predictions of our automatic system (X-axis) compared to the corresponding ground truth values (Y-axis). Results are averaged over the 4 languages covered by the test sets.
In what follows, we measure both the quality of our automatic annotation methodology (Section 3.1) and of our idiom identification system (Section 3.3) by means of accuracy and token-level macro  $\mathrm{F_1}$ -score metrics, respectively. In the latter case, we rely on the macro- $\mathrm{F_1}$  metric due to the high class imbalance in the datasets, i.e. the number of O tags is much higher than the sum of the number of B and I tags, see Table 1.
Silver-Data Quality Evaluation We first aim at providing an empirical evaluation of the effectiveness of the proposed automatic strategy for producing idiom-related<sup>10</sup> sentences. To do so, for each language, we apply our dual-encoder discriminator  $\Phi(e, c)$  to the expression-context pairs  $\langle e, c \rangle$  available in our manually-curated test set, and we measure the accuracy score by comparing the predictions produced by the system with the human annotations in the gold-standard test sets. The accuracy results obtained are reported in Table 4.
With this being a binary-classification task, we can observe that the performance achieved by our dual encoder is much higher than the  $50\%$  baseline of a random classifier, hence implying that the system is able to distinguish between idiomatic and literal usages of PIEs based on the surrounding contexts.
However, the accuracy is not sufficient for us to determine the strengths and the weaknesses of our system. Therefore, we group both the predic
9We use the English validation set to manually search for the best value of  $\delta$  by choosing from the following set of possible values:  $\delta = \{0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7\}$
10With the term "idiom-related sentences" we refer to sentences containing potentially idiomatic expressions.
Tag	P	R	F1	% Seen
EN	B	84.2	53.5	65.4	-
I	91.1	57.4	70.4	-
O	92.5	99.1	95.7	-
ALL	89.2	70.0	77.1	43.7%
DE	B	87.6	70.2	77.9	-
I	90.7	72.7	80.7	-
O	96.2	98.9	97.5	-
ALL	91.5	80.6	85.4	64.0%
IT	B	72.2	61.9	66.7	-
I	76.7	62.0	68.6	-
O	96.7	98.2	97.4	-
ALL	81.8	74.0	77.6	62.6%
ES	B	47.2	51.1	49.1	-
I	67.4	45.1	54.0	-
O	87.7	92.8	90.2	-
ALL	67.4	63.0	64.4	24.4%
Table 5: Results of the idiom identification system in terms of Precision (P), Recall (R) and Macro-F1 (F1) scores on the four test languages. " % Seen" represents the percentage of idioms already encountered in the training set. Morphological variations of the same idiom are considered as a unique idiom.