From 'It's All Greek to Me' to 'Nur Bahnhof verstehen': An Investigation of mBERT's Cross-Linguistic Capabilities
Authors: Aria Rastegar, Pegah Ramezani (FAU Erlangen-Nuremberg, 2024)
Topic: Distributional Semantics

Abstract
This study investigates the impact of cross-linguistic similarities on idiom representations in mBERT, focusing on English and German idioms categorized by different degrees of similarity. We aim to determine whether different degrees of cross-linguistic similarities significantly affect mBERT's representations and to observe how these representations change across its 12 layers. Contrary to our initial hypothesis, cross-linguistic similarity did not uniformly impact idiom representations across all layers. While early and middle layers showed no significant differences among idiom categories, higher layers (from Layer 8 onwards) revealed more nuanced processing. Specifically, significant differences between the control category and idioms with similar meaning (SM), as well as between idioms with similar lexical items (SL) and those with similar semantics (SM) were observed. Our analysis revealed that early layers provided general representations, while higher layers showed increased differentiation between literal and figurative meanings. This was evidenced by a general decrease in cosine similarities from Layer 5 onwards, with Layer 8 demonstrating the lowest cosine similarities across all categories. Interestingly, a trend suggests that mBERT performs slightly better with more literal hints. The order of cosine similarity for the categorizations was: idioms with a degree of formal similarity, control idioms, idioms with both formal and semantic similarity, and finally idioms with only semantic similarity. These findings indicate that mBERT's processing of idioms evolves significantly across its layers, with cross-linguistic might affect more significantly in higher layers where more abstract semantic processing likely occurs.

Introduction
Idioms are one of the most studied linguistic concepts that broadly can be defined as multi-word expressions that are often fixed in terms of their syntactic and lexical aspects, while they usually carry meanings that cannot be directly deduced from the meaning of individual words they contain AUTHOR. Given their syntactic and structural fixedness and non-compositional aspects, they were perceived as peripheral, supplementary, or appendixes to language grammars in earlier approaches to idioms \citep[p.\~504]{fillmore1988}. However, with the increasing interest in corpus studies of language, it has been observed that much of human linguistic production is routinized and prefabricated AUTHOR. Multi-word expressions with a high degree of conventionality do not seem to be marginal or limited linguistic constructions, as they play an important role in our everyday life AUTHOR. In addition, they seem to be used in communication across various contexts, from novels to political debates and therapeutic dialogues AUTHOR. Given their characteristics and their conventionalized meanings, they pose many challenges to language speakers, especially non-native language speakers AUTHOR. However, their characteristics also make them a good case study in different experimental linguistics settings. Recent advancements in Large Language Models (LLMs) and their widespread application have prompted linguists to investigate the performance of these models across various linguistic concepts, including idioms AUTHOR. In addition, in the case of multi-lingual models, an interesting research area is how these models encode the different languages on which they are trained AUTHOR. In this study, a categorization of English and German idioms based on three cross-linguistic degrees of similarity is proposed. One category includes idioms that have similar formal and semantic aspects in these languages; the second includes idioms with formal similarities but different semantic aspects; and the third category includes idioms with similar semantic aspects but different formal aspects. The goal of our work is to consider how cross-linguistic similarities among idioms affect the representation of idioms in mBERT. More specifically, the questions underlying the following experiment were: \item Does cross-linguistic similarity have a significant impact on the representation of idioms in mBERT? \item Does the degree of cross-linguistic similarity and the representation of the model change across the 12 layers of mBERT? We hypothesized that mBERT's performance would depend on how it utilizes its multilingual training data. Namely, if mBERT draws from a collective pool of all languages, it should perform consistently across all cross-linguistic categories, similar to how it represents idioms from the language it has been given, that is English in this case. However, if it primarily retrieves data from specific languages, we expect to observe significant performance differences among the categories, potentially mirroring some of the patterns seen in cross-linguistic studies with second language speakers. That is, identical cross-linguistic idioms should be represented almost similarly to the control idioms (in this case, English idioms), and idioms with formal and lexical correspondence could both be represented similarly and, in some cases, more differently from the control idioms. Finally, idioms with only corresponding semantics and different formal aspects should be the most differently represented idioms compared to the control group. Furthermore, given the proposed categorizations based on formal and semantic similarities, we anticipated varying performance across mBERT's 12 layers. Particularly, in lower layers, we expect less differentiation among categories, as these layers typically capture more surface-level features. While in higher layers, which represent more of the semantic aspects, we anticipate more varying trends and larger differences among the categories. Mostly because we are primarily focused on the figurative meaning of idioms across different categories.

Conclusion
Our study investigated how cross-linguistic similarities among idioms affect their representation in mBERT, with a focus on English and German idioms categorized based on three degrees of cross-linguistic similarity. This study aims to answer two main research questions concerning whether cross-linguistic similarity has a significant impact on the representation of idioms in mBERT, and how the degree of cross-linguistic similarity and the representation of the model change across the 12 layers of mBERT. Our findings provide insights into these questions and our initial hypotheses. Contrary to our initial hypothesis, we found that cross-linguistic similarity does not have a uniformly significant impact on the representations of idioms across all layers of mBERT. The main effects of our translated idioms categorized into cross-linguistic categories (SI: formal and semantic similarity, SL: similar lexicon, SM: Similar Meaning), were not statistically significant when compared to the control category (English idioms) in the early and middle layers of the model. This result may suggest that mBERT might be utilizing knowledge from all languages in its training data as a collective pool, at least in the case of the studied idioms. This aligns with the idea that mBERT learns multilingual representations that go beyond simple vocabulary memorization, as suggested by \citet{pires2019multilingual}. However, the emergence of significant differences in higher layers (particularly from Layer 8 onwards) might indicate that mBERT's processing of idioms becomes more nuanced as information propagates through the network. This finding partially supports our hypothesis that mBERT might show different performances for each cross-linguistic categorization, but suggests that these differences are more significant in the model's deeper layers. Although there are no significant differences among all categories, in Figure fig:cosine-category there is a continuous trend in different layers showing more similarity first for the SL category, then Control, followed by SI, and finally the SM category. This trend indicates that BERT represents almost all categories similarly, and when there are more literal hints, BERT tends to perform better which aligns with the findings of multi-lingual transfer of \citet{pires2019multilingual}. Moreover, for idioms with semantic similarities, the model demonstrates the lowest cosine similarity between the representations of idioms and their figurative meanings, which might suggest that idioms with only semantic correspondence across the studied languages pose a greater challenge for mBERT in capturing the figurative meanings of idioms. Our second research question focused on how the representation of idioms changes across mBERT's 12 layers. In this analysis, distinct patterns were observed. In early layers (1-4) the cosine similarity for CLS embedding derived from mBERT for the idioms and their corresponding figurative meaning was high and relatively uniform across all categories, suggesting a more general representation, we believe high similarity in early layers can be related to similarity in the syntax of samples and the provided figurative entities since these layers capture more formal and syntactic information. Layer 3 demonstrated the highest cosine similarities, while from Layer 5 onwards, a general decrease in cosine similarities was observed, suggesting increased differences between literal and figurative meanings in higher layers. Layer 8 showed the lowest cosine similarities and marked the beginning of significant differences between categories, particularly for semantically similar idioms (SM category). These findings contribute to our hypothesis that we would observe different performances among the layers of mBERT given the formal and semantic similarities of idioms.