How Are Idioms Processed Inside Transformer Language Models?
Year: 2023

Abstract:
Idioms such as “call it a day” and “piece of 001 cake”, are ubiquitous in natural language. How 002 do Transformer language models process id003 ioms? This study examines this question by 004 analysing three models BERT, Multilingual 005 BERT, and DistilBERT. We compare the em006 beddings of idiomatic and literal expressions 007 across all layers of the networks at both the 008 sentence and word levels. Additionally, we in009 vestigate the attention directed from other sen010 tence tokens towards a word within an idiom 011 as opposed to in a literal context. Results indi012 cate that while the three models exhibit slightly 013 different internal mechanisms, they all repre014 sent idioms distinctively compared to literal 015 language, with attention playing a critical role. 016 These findings suggest that idioms are semanti017 cally and syntactically idiosyncratic, not only 018 for humans but also for language models. 019

heading: Future Work
text: In future work we will in-319 vestigate this phenomenon in models with different320 architectures, for example GPT and XLNet