SlowGuess/NAACL_2022_OCR huggingface dataset:
tions and the labels coming from the 4 languages, and construct a confusion matrix in order to better analyze the system behavior. From the confusion matrix in Figure 2, we can observe that the system is able to (almost always) identify idiomatic expressions as such, mainly thanks to their semantic distance from the meaning of the surrounding words. On the other hand, when dealing with literal expressions, the system again correctly predicts the majority of these, but it makes more errors. We attribute this to the fact that the context is often not sufficiently rich to find a strong similarity (i.e. higher than the threshold  $\delta$ ) with the meaning of the individual constituents of the idiomatic expression, and hence to label the expression as literal. Indeed, the lower the value of  $\delta$ , the higher the number of literal expressions discovered, but the system inevitably classifies more idiomatic expressions as literal.
Multilingual Idiomatic Expression Identification In the previous paragraph we evaluated the performance of our dual-encoder architecture on the binary literal or idiomatic classification task, where the PIE was pre-identified. In this paragraph, instead, we use the refined silver-data produced by the aforementioned dual encoder, and measure
Language	Seen	Unseen
P	R	F1	P	R	F1
EN	91.9	71.5	79.1	87.2	68.8	75.6
DE	98.7	95.5	97.1	64.5	49.1	53.8
IT	96.5	91.3	93.8	55.9	49.7	52.2
ES	94.9	96.9	95.9	60.2	55.6	56.9
Table 6: Results on the "Seen" and "Unseen" test set subsets in terms of token-level Precision (P), Recall (R) and Macro-F1 (F1) scores.
Dual Encoder?	F1	Δ
EN	Yes	77.1	-
No	73.6	+ 3.5
DE	Yes	85.4	-
No	81.9	+ 3.5
IT	Yes	77.6	-
No	73.4	+ 4.2
ES	Yes	64.4	-
No	58.3	+ 6.1
Table 7: Comparison of the results obtained by training the system on the silver-standard data validated by our dual encoder (Yes) and non validated ones (No).
the identification capabilities of our idiom identification system on the sequence-labeling task we introduced (Section 3.3) by comparing the BIO tags produced with the corresponding gold labels. The results obtained are reported in Table 5 (further results are provided in Appendix A).
The first thing that catches the eye is that the performances on the O tags are much higher than those on the B and I tags, on all tested languages. However, this is not surprising, owing to the fact that there is a high class imbalance. An interesting result, instead, is that the system achieves an average score of about 76 F1 points, while the percentage of seen entities<sup>11</sup> is only  $48.7\%$  on average. This implies that the system is able to generalize, and consequently also to correctly predict unseen idioms. This phenomenon is particularly evident on English and Spanish, where the percentage of seen idioms is very low.
To better highlight the capability of the system to go beyond idioms already seen during training, we also analyze the system performance on the "seen" and "unseen" subsets independently, and report the results in Table 6. As we can observe, the
11Seen entities are entities in the test set which have already been encountered in the training set.
Type	Prediction
DE	Correct✔	Ich bin nur der Typ, der ihr die Stange hält.
Correct✔	Wir haben diese Gesellschaft von Grund auf aufgebaut.
Wrong ×	Durch den Wind wurden 27 Häuser in der Region zerstört.
Wrong ×	Sei nicht so'ne beleidigte Leberwurst!
EN	Correct✔	The old horse finally kicked the bucket.
Correct✔	Written tests are his Achilles' heel...
Wrong ×	Her aunt is a great cook, do you want a piece of cake?
Wrong ×	It is difficult, but possible to quit smoking cold turkey.
IT	Correct✔	Mi sono cavato gli occhi dopo aver decifrato la grafia farraginosa.
Correct✔	Invece di decidere su due piedi, diedi dispositionsi a Tom Donilon perché convocasse i delegati...
Wrong ×	A quel punto Smith lancio a terra un bicchiere.
Wrong ×	Non era affatto sontato che Romney rientrasse nei ranghi, visti i loro rapporti burrascosi con Trump.
ES	Correct✔	A la inaguración fueon cuiro gatos.
Correct✔	El=gobierno siue metiendo el dedo en la llaga.
Wrong ×	¿Has visto alguna vez a tu gato meter la pata en su bebedero?
Wrong ×	Elagente tiene vista de lince.
Table 8: Examples of idioms correctly and wrongly identified by our idiom identification system. Underline represents the target idiomatic expression (if any), while bold + italic represents the predicted idiomatic expression.
system is able to correctly predict the majority of unseen idioms on all tested languages, achieving an F1 score of 59.6 points, on average. Moreover, on seen idioms, the system behaves almost perfectly reaching an average score of 91.5 points. We underline that morphological variations of idioms encountered in the training sets are considered as seen idioms. Table 1 provides dimensions of the "seen" and "unseen" subsets, for each language.
Then, to further demonstrate the effectiveness of our dual-encoder architecture (Section 3.1), we compare the results obtained by training the system on the data produced with and without the validation performed by our dual encoder. The results reported in Table 7 highlight an average gap of 4.3 F1-score points between the refined version of the data and the original one, showing how the validation step is fundamental for improving the quality of the annotations, consequently leading the system to a better understanding of idioms.
Finally, the high results in Table 5 also prove that, thanks to our renewed task formulation (Section 3.3), common sequence-labeling architectures (e.g. those used for NER) can be successfully imported into the idiom identification task, thus enabling knowledge transfer from other research areas.
Together with the quantitative evaluation provided in Section 4.3, we now perform a qualitative analysis of our system. More specifically, in Table 8, we provide 4 examples of system predictions (2 correct
and 2 wrong) for each tested language. Although our system proves to be robust over literal PIEs (see Figure 2), its most common mistake consists in classifying a PIE used with its literal meaning as idiomatic. This is mainly due to the system bias towards the labels associated to such PIEs during training, e.g. if more than  $90\%$  of occurrences of a certain PIE are labeled as idiomatic in the training set, then the system will tend to classify as idiomatic any other of its occurrences in the test set. This result suggests that improvements over the distribution of labels of PIEs are possible. In Table 8 we provide an example of one such wrongly labeled PIE for each language. Another commonly observed error, again highlighted in Table 8, is that in which unseen idiomatic expressions are not identified by the system. However, as previously demonstrated in Table 6, the system is nevertheless able to correctly handle the majority of such cases.
On the other hand, we observe that the system is able to correctly identify both lemmatized and inflected forms of idiomatic expressions, for both seen and unseen ones.
In this work, we introduced ID10M, an innovative framework for idiom identification consisting of i) a new multilingual Transformer-based architecture, ii) a novel automatic annotation pipeline for creating high-quality silver-data in 10 languages, and iii) a challenging manually-curated benchmark in 4 languages. Moreover, while the majority of
current approaches to idiom identification need pre-identified potentially idiomatic expressions, we, instead, dropped this requirement and proposed a new formulation for the idiom identification task that lets systems be directly applicable to raw texts. Finally, our experiments showed that our system is able to generalize beyond idioms seen during training, hence achieving up to 85.4 macro F1-score on the idiom identification task.
As future work, we plan to scale our system to a greater number of languages and textual sources, but, most importantly, investigate the benefits derived from our work in key tasks such as Word Sense Disambiguation, Machine Translation and Question Answering.
The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487 under the European Union's Horizon 2020 research and innovation programme and the support of the ELEXIS project No. 731015 under the European Union's Horizon 2020
Judit Ács, Ákos Kádár, and Andras Kornai. 2021. Subword pooling makes a difference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2284-2295, Online. Association for Computational Linguistics.
Dimitra Anastasiou. 2010. Idiom treatment experiments in machine translation. Cambridge Scholars Publishing.
Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. Handbook of natural language processing, 2:267-292.
Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4330-4338. International Joint Conferences on Artificial Intelligence Organization. Survey Track.
Terra Blevins and Luke Zettlemoyer. 2020. Moving down the long tail of word sense disambiguation with gloss informed bi-encoders. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1006-1017, Online. Association for Computational Linguistics.
Jan A. Botha, Zifei Shan, and Daniel Gillick. 2020. Entity Linking in 100 Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7833-7845, Online. Association for Computational Linguistics.
Cristina Cacciari and Patrizia Tabossi. 1988. The comprehension of idioms. Journal of memory and language, 27(6):668-683.
Cristina Cacciari and Patrizia Tabossi. 2014. *Idioms: Processing, structure, and interpretation*. Psychology Press.
Chenhui Chu and Rui Wang. 2018. A survey of domain adaptation for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304-1319, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
BNC Consortium et al. 2007. British national corpus. Oxford Text Archive Core Collection.
Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2007. Pulling their weight: Exploiting syntactic forms for the automatic identification of idiomatic expressions in context. In Proceedings of the Workshop on A Broader Perspective on Multiword Expressions, pages 41-48, Prague, Czech Republic. Association for Computational Linguistics.
Paul Cook, Afsaneh Fazly, and Suzanne Stevenson. 2008. The vnc-tokens dataset. In Proceedings of the LREC Workshop Towards a Shared Task for Multiword Expressions (MWE 2008), pages 19-22.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
Mona Diab and Pravin Bhutada. 2009. Verb noun construction mwe token classification. In Proceedings of the Workshop on Multiword Expressions: Identification, Interpretation, Disambiguation and Applications (MWE 2009), pages 17-22.
Rafael Ehren. 2017. *Literal or idiomatic?* identifying the reading of single occurrences of German multi-word expressions using word embeddings. In *Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics*, pages 103–112, Valencia, Spain. Association for Computational Linguistics.
Samin Fakharian. 2021. Contextualized embeddings encode knowledge of English verb-noun combination idiomaticity. Ph.D. thesis, University of New Brunswick.
Afsaneh Fazly and Suzanne Stevenson. 2006. Automatically constructing a lexicon of verb phrase idiomatic combinations. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics.
Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. 2008. Introducing and evaluating ukwac, a very large web-derived corpus of english. In Proceedings of the 4th Web as Corpus Workshop (WAC-4) Can we beat Google, pages 47-54.
Mahak Gambhir and Vishal Gupta. 2017. Recent automatic text summarization techniques: a survey. Artificial Intelligence Review, 47(1):1-66.
Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2021. Probing for idiomaticity in vector space models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3551-3564, Online. Association for Computational Linguistics.
Waseem Gharbieh, Virendra Bhavsar, and Paul Cook. 2016. A word embedding approach to identifying verb-noun idiomatic combinations. In Proceedings of the 12th Workshop on Multiword Expressions, pages 112-118, Berlin, Germany. Association for Computational Linguistics.
Raymond W Gibbs Jr. 1992. What do idioms really mean? Journal of Memory and language, 31(4):485-506.
Hongyu Gong, Suma Bhat, and Pramod Viswanath. 2017. Geometry of compositionality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
David Graff and Cieri Christopher. 2003. English gigaword ldc2003t05. web download. In *Philadelphia: Linguistic Data Consortium*, 2003.
Reyhaneh Hashempour and Aline Villavicencio. 2020. Leveraging contextual embeddings and idiom principle for detecting idiomaticity in potentially idiomatic expressions. In Proceedings of the Workshop on the Cognitive Aspects of the Lexicon, pages 72-80.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-crf models for sequence tagging.
Nancy Ide and Catherine Macleod. 2001. The american national corpus: A standardized resource of american english. In Proceedings of corpus linguistics, volume 3, pages 1-7. CiteSeer.
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Taylor Berg-Kirkpatrick. 2021. Investigating robustness of dialog models to popular figurative language constructs.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Ioannis Korkontzelos, Torsten Zesch, Fabio Massimo Zanzotto, and Chris Biemann. 2013. SemEval-2013 task 5: Evaluating phrasal semantics. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 39–47, Atlanta, Georgia, USA. Association for Computational Linguistics.
Dilin Liu. 2017. Idioms: Description, comprehension, acquisition, and pedagogy. Routledge.
Harish Tayyar Madabushi, Edward Gow-Smith, Carolina Scarton, and Aline Villavicencio. 2021. Astitchinlanguagemodels: Dataset and methods for the exploration of idiomaticity in pre-trained language models.
Amit Mishra and Sanjay Kumar Jain. 2016. A survey on question answering systems with classification. Journal of King Saud University-Computer and Information Sciences, 28(3):345-361.
David Mueller, Nicholas Andrews, and Mark Dredze. 2020. Sources of transfer in multilingual named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8093-8104, Online. Association for Computational Linguistics.
Grace Muzny and Luke Zettlemoyer. 2013. Automatic idiom identification in Wiktionary. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1417-1421, Seattle, Washington, USA. Association for Computational Linguistics.
Vasudevan Nedumpozhimana and John Kelleher. 2021. Finding bert's idiomatic key. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 57-62.
Geoffrey Nunberg, Ivan A Sag, and Thomas Wasow. 1994. Idioms. Language, 70(3):491-538.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Jing Peng, Anna Feldman, and Hamza Jazmati. 2015. Classifying idiomatic and literal expressions using vector space representations. In Proceedings of the
International Conference Recent Advances in Natural Language Processing, pages 507-511, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.
Carlos Ramisch, Aline Villavicencio, Leonardo Moura, and Marco Idiart. 2008. Picking them up and figuring them out: Verb-particle constructions, noise and idiomaticity. In CoNLL 2008: Proceedings of the Twelfth Conference on Computational Natural Language Learning, pages 49-56, Manchester, England. Coling 2008 Organizing Committee.
Ivan A Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for nlp. In International conference on intelligent text processing and computational linguistics, pages 1-15. Springer.
Manfred Sailer and Stella Markantonatou. 2018. Multi-word Expressions: Insights from a multi-lingual perspective. Language Science Press.
Nathan Schneider, Dirk Hovy, Anders Johannsen, and Marine Carpuat. 2016. SemEval-2016 task 10: Detecting minimal semantic units and their meanings (DiMSUM). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 546-559, San Diego, California. Association for Computational Linguistics.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021. WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1351-1361, Online. Association for Computational Linguistics.
Marco Silvio Giuseppe Senaldi, Yuri Bizzoni, and Alessandro Lenci. 2019. What do neural networks actually learn, when they learn to identify idioms? Proceedings of the Society for Computation in Linguistics, 2(1):310-313.
Ozge Sevgili, Artem Shelmanov, Mikhail Arkhipov, Alexander Panchenko, and Chris Biemann. 2020. Neural entity linking: A survey of models based on deep learning. arXiv preprint arXiv:2006.00575.
Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. Metaphor identification using verb and noun clustering. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1002-1010, Beijing, China. Coling 2010 Organizing Committee.
Caroline Sporleder and Linlin Li. 2009. Unsupervised recognition of literal and non-literal use of idiomatic expressions. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 754-762, Athens, Greece. Association for Computational Linguistics.
Caroline Sporleder, Linlin Li, Philip Gorinski, and Xaver Koch. 2010. Idioms in context: The IDIX corpus. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).
Laura Street, Nathan Michalov, Rachel Silverstein, Michael Reynolds, Lurdes Ruela, Felicia Flowers, Angela Talucci, Priscilla Pereira, Gabriella Morgen, Samantha Siegel, Marci Barousse, Antequa Anderson, Tashom Carroll, and Anna Feldman. 2010. Like finding a needle in a haystack: Annotating the American national corpus for idiomatic expressions. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).
Simone Tedeschi, Simone Conia, Francesco Cecconi, and Roberto Navigli. 2021. Named Entity Recognition for Entity Linking: What works and what's next. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2584-2596, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Simone Tedeschi and Roberto Navigli. 2022. NER4ID at SemEval-2022 Task 2: Named Entity Recognition for Idiomaticity Detection. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics.
Rakesh Verma and Vasanthi Vuppuluri. 2015. A new approach for idiom identification using meanings and the web. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 681-687, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing.
Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2145-2158, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Since we reformulated the idiom identification task as a sequence-labeling task (Section 3.3), all previous approaches (that required pre-identified potentially idiomatic expressions) cannot be compared directly. Nonetheless, in order to select a
System	F1
Bi-LSTM	69.5
Bi-LSTM + CRF	70.9
mBERT	74.8
mBERT + Bi-LSTM	75.4
mBERT + Bi-LSTM + CRF	76.1
XLM-R	74.3
XLM-R + Bi-LSTM	75.4
XLM-R + Bi-LSTM + CRF	75.9
Table 9: Token-level macro F1 scores of different sequence tagging alternatives computed on our test set. Results are averaged over the four languages.
robust architecture for the idiom identification task, we compared various sequence tagging architectures. Specifically, we evaluated the performance of several alternative systems: Bidirectional LSTM (Bi-LSTM), Bi-LSTM + CRF, Multilingual BERT (mBERT), mBERT + Bi-LSTM, mBERT + BiLSTM + CRF, XLM-RoBERTa (XLM-R, Conneau et al., 2020), XLM-R + Bi-LSTM, XLM-R + BiLSTM + CRF. Results are reported in Table 9. Surprisingly, mBERT achieved performance slightly higher than XLM-R. Moreover, the addition of BiLSTM and CRF modules provided further improvements.