Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
Authors: Elena Sofia Ruzzetti
Federico Ranaldi
Felicia Logozzo
Michele Mastromattei
Leonardo Ranaldi
Fabio Massimo Zanzotto
Venue: EMNLP 2023 Findings (2023)
Keywords: interpretability
CKA
typological similarity
BERT
TL;DR: Compare weight matrices in Transformers for different languages and observe whether linguistically-motivated similarities emerge

Abstract: The impressive achievements of transformers force NLP researchers to delve into how these models represent the underlying structure of natural language.
In this paper, we propose a novel standpoint to investigate the above issue: using typological similarities among languages to observe how their respective monolingual models encode structural information.
We aim to layer-wise compare transformers for typologically similar languages to observe whether these similarities emerge for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices. We found that syntactic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices.

Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
Elena Sofia Ruzzetti "1 Federico Ranaldi Felicia Logozzo²
Michele Mastromattei Leonardo Ranaldī* Fabio Massimo Zanzotto' University of Rome Tor Vergata, Italy
University for foreigners of Siena, Italy
*Campus Bio-Medico University of Rome, Italy "Idiap Research Institute, Switzerland
elena. sofia. ruzzetti@uniroma..
federico. ranal di 8alumni. waamii.nniiommnn uniroma2.eu
Abstract
2022). However, there is no guarantee that when considering languages that share similar structures, the models for those languages will represent those structures the same way.
The impressive achievements of transformers force NLP researchers delve into how these models represent the underlying structure of natural language. In this paper, we propose novel standpoint to investigate the above issue: using typological similarities among languages observe how their respective monolingual models encode structural information. We aim to layer-wise compare transformers for typo- logically similar languages observe whether these similarities emerge for particular layers. For this investigation, we propose to use Cen- tered Kernel Alignment measure similarity among weight matrices. We found that syntac- tic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices.
Conversely, decades of studies in NLP have cre- ated symbol-empowered architectures where every- thing is explicitly represented. These architectures implement different levels of linguistic analysis: morphology. syntax, semantics, and pragmatics are few subdisciplines of linguistics that shaped how symbolic-based NLP has been conceived. In this case, linguistic model of language set of rules and regularities defining its behavior di- rectly influences the system processing that lan- guage. Understanding whether linguistic models emerge in opaque pre- trairrinedd transformer architec- ture is a compelling issue.
Probing transformers have mainly been used investigate whether these model classical linguistic properties of languages. Probing consists of prepar- ing precise sets of examples probe tasks - and, eventually, observing how these examples activate transformers. In this way, BERT (Devlin et al., 2019) contextual representations have been tested to assess their ability to model syntactic informa- tion and morphology (Tenney et al., 2019; Gold- berg, 2019; Hewitt and Manning, 2019; Jawahar et al., 2019; Coenen et al., 2019; Edmiston, 2020), also comparing different monolingual BERT mod- els (Nikolaev and Pado, 2022; Otmakhova et al.. 2022).
1 Introduction
Natural language processing (NLP) is dominated by powerful but opaque deep neural networks. The rationale behind this trend is that deep architec- ture training allows rules and structural information about language to emerge directly from sentences in the target language, sacrificing the interpretable and transparent definition of language regularities. Some exceptions exist where structural syntactic information explicitly encoded multilayer per- ceptrons (Zanzotto et al., 2020) with relevant re- sults on unseen sentences (Onorati et al., 2023). Yet, pre-trained transformers (Vaswani et al., 2017; Devlin et al., 2019) are offered as versatile univer- sal sentence/text encoders that contain whatever is needed to solve any downstream task. These mod- els outperform all other models nearly consistently after fine-tuning or domain-adane -ddpttttoon (Jin al.,
In this study, we take a different standpoint to in- vestigate if traces of linguistic models are encoded in monolingual BERT models: using linguistically- motivated typological similarities among languages (Dryer and Haspelmath, 2013), aim to layer- wise compare transformers for different languages to observe whether these similarities emerge be- tween weight matrices for particular layers. For this investigation, we propose to use Centered Ker- nel Alignment to measure similarity among weight
"These authors contributed equally to this work

---

Latin:
Hostes (S "Fnemies") castra "the
(Lehmann, 2013; Dunn, 2015). Indeed, languages belonging to a genealogical family can share spe- cific typological properties or not. Likewise, lan- guages that appear similar, according to a specific typological category, can be genealogically linked or not. For example, Latin, Japanese and Turk- ish are final-verb languages, whereas Italian is not, despite being directly derived from Latin (see ex- amples in Table 1).
camp") oppugnant (V "attack")
Japanese:
Maria-san (S "Mary") wa (particle) sarada
'salad') wo (particle) tabemasu (V "eats')
Turkish:
Maria (S - -Mary") elmayi (O "the apple")
yiyor (V "eats")
Italian:
Maria 'Maria' (S) mangia "cats" (V) mela
"the apple' (0)"
Table 1: Syntactic property of languages: the basic order of constituents subject (S), direct object (O) and verb (V) - in clause
In this work, we focus on typological classifica- tion because directly describes languages as a set of rules of composition of words from the syntactic point of view and morphemes in the morphological analysis. In fact, in the following analysis, our aim is to understand whether rules shared across simi- lar languages encoded into the corresponding models.
matrices (Kornblith et al., 2019). We discovered that syntactic typological similarity is consistent with the similarity among weights in the middle layers both in pretrained models and in domain- adapted models after performing domain adaption on a parallel corpus.
Typological similarities between languages have been previously investigated with probes in two ways: (1) by using syntactic probes in comparing the behavior of different monolingual BERT mod- cls (Nikolaev and Pado, 2022; Otmakhova et al., 2022); and, (2) by searching shared syntax rep- resentation in multilingual BERT (mBERT) (Chi et al., 2020; Ravishankar et al., 2019; Singh et al., 2019). Chi et al. (2020) suggest that mBERT has a behavior similar to monolingual BERT since its shared syntax reprexentation can be found in mid- dle layers. However, mBERT seems to separate the representations for each language into different subspaces (Singh et al., 2019). Hence, mBERT is exploiting typological similarities among languages to build better language models: this model does not exploit similarity among languages to reuse part of the representation builds across different languages.
2 Background and related work
Although different, natural languages show more less evident similarities at all levels of analy- sis phonetics, morphology, syntax, and seman- ties due to the fact that "human language capac- ity a species-specific biological property, essen- tially unique to humans, invariant among human groups, and dissociated from other cognitive sys- tems" (Chomsky, 2017). Hence, some properties, called "language universals", are shared by all nat- ural languages: for example, all spoken languages use vowels, all languages have nouns and verbs. Some properties, on the other hand, are common among groups of languages, and are called "univer- sal tendencies": for instance, many languages have nasal consonants. (Greenberg, 2005; Haspelmath, 2001a,b). A classification of languages enables us to organize the shared characteristics of languages. Languages are classified in two main ways: ty- pological classification and genealogical classifica- tion. Language typology is the branch of linguistics that, studying universal tendencies of languages (Comrie, 1989; Song, 2010), maps out "the vari- ation space filled by the languages of the world" and finds "regular patterns and limits of variation" " (Haspelmath, 2001a). These typological patterns can be used to classify all languages at different levels of linguistic analysis. Therefore, a typolog- ical classification enhances the shared structures among languages.
Another perspective exploit similarity among languages is to compare activation matrices of transformers produced over probing examples. this case, the underlying assumption that par- allel sentences should have similar activation ma- trices if languages are similar. To compare these activation matrices, Canonical Correlation Anal- ysis (CCA) (Hardoon et al., 2004), Singular Vec- tor CCA (SVCCA) (Raghu et al., 2017), and Pro- jection Weighted CCA (PWCCA) (Morcos et al., 2018) have been widely used. More recently, Cen- tered Kernel Alignment (CKA) (Kornblith et al., 2019) measure has been proposed to quantify rep- resentational similarity. Kornblith et al. (2019) also claims that CCA and SVCCA are less adequate
Typological classification differs from the ge- nealogical (or genetic) one, which aims to catego- rize languages originating from a common ancestor

---

for assessing similarity between representations be- cause they are invariant to linear transformations. Using these metrics, Vulié al. (2020) compared activation matrices derived from monolingual mod- els for pairs of translated words: using CKA, it measured the similarity between the representa- tion of words in different languages. They noticed that the similarity between the representations of words is higher for more similar languages. Simi- larities between mBERT activation matrices have also been used derive a phylogenetic tree of lan- guages (Singh et al., 2019; Rama et al., 2020).
analysis. Languages in WALS are represented as categorial feature-value vectors. Each feature can assume two or more values 1 For instance, the 20A Fusion of Selected Inflectional Formatives assumes 7 possible values: 1. Exclusively concatenative; 2 Exclusively isolating: 3 Exclusively tonal; 4
Tomal/isolating: 5 Tonal/concateeatiiv;;
Ablaut/concatenative and 7 The feature 25B Zero Marking of and P Arguments allows only 2 possibilities: 1. Zero-marking and 2. Non-zero marking.
Ioolatinggconcateeattie.
In our study, building on typological similari- ties among languages and on the measures to com- pare matrices, we aim to compare monolingual BERTs by directly evaluating layer-wise the simi- larities among parameters of BERTs for different languages. To the best of our knowledge, this is the first study using this idea to investigate whether transformers replicate linguistic models without probing tasks.
Since not all world languages are classified ac- cording to each feature - as the list of languages analyzed varies from feature to feature we had to integrate WALSS Specifically, we took into ac- count all the 12 features classified under "Morphol- ogy Area" and a selection of syntactic features of WALS (see Tab. 2). Two linguists added the values for features not defined by WALS.
Regarding morphological features in particular, for Dutch, Romanian, and Swedish we added the values of all features except 26A, which was the only one already compiled in WALS for these lan- guages; we also completed the missing features for Italian except 26A and 27A.
3 Method and Data
languages are typologically similar for syntac- tie or for morphological properties, their BERT models should have some similar weight matrices in some layers. This is our main intuition. Then, in this section, we introduce method to inves- tigate the above intuition. Section 3.1 describes the selected typological classification of languages and how we use it to compute the similarity be- tween languages. Section 3.2 briefly introduces the subject of our study, BERT. Finally, Section 3.3 introduces a novel way, biCKA, to compare weight matrices.
In terms of syntactical features, we selected the most important ones relating to word order (Song, 2012) - from 81A to 97A, Tab. 2 - and some repre- sentative characteristics related to linguistic nega- tion (143A, 143E, 143F, 144A). The last-mentioned features about negative morphemes and words have also been selected because these are fully speci- fied for target languages. Unlike morphological features, syntactic features are almost fully spec- ified for all target languages. We only integrated features with the following values: 84A (Italian, Russian, Rumanian: VOX, Persian: 3 XOV, Greek: 6 No dominant order); 93A (Italian and Dutch = 1 Initial interrogative phrase). The com- plete list of categorical vectors for target languages is available in Appendix A. these vectors are ex- tracted from WALS and, eventually, integrated by the two linguists.
3.1 Selected typological similarity among languages
To compute morphological and syntactic similarity among languages, we a typological classifica- tion scheme that is used to generate metric feature vectors for languages. The idea of representing languages using feature vectors and then compar- ing the similarity between these vectors is in line with previous work, such as lang2vee (Littell et al., 2017). In particular, we stem from WALS to gather typological features of languages.
Once we have defined all the values of the se- lected WALS features, we can obtain vectors that represent languages to get their similarity. We use categorical WALS vectors for languages to gen-
The World Atlas of Language Structures (WAL.S) (Dryer and Haspelmath, 2013) aims at classifying languages of the world on the basis of typological criteria at all levels of linguistic
'A complete description of cach feature and its values can he found on https: / //wwls/wwls.aaaa//... info/feature

---

WAI.5 17
Pentur
3.2 BERT Model in brief
Marphologieal fronuez
20A
Frsion of Selectal Inflectioaa Formstives
Among the different transformer-based architec- tures, BERT (Devlin et al., 2019) is the subject of our study since it is a widely known architecture, and many BERT models in different languages are available. This section briefly describes how it is organized in order to give names to weight matrices at each layer.
Exponence Selected inflectional l'omativve
Resse Aspect-Mood Iuflection
Synthesis Verb
of Marking the Cluss
Lace of Marking Posacasive Noun Phinees
25A
Lecus of Marking: Wholk-lasguage Typelogy
25R
Zamu Marking of and IP Argumens
26A
Prefising vs. Suttising in Inflectional Morphology
27A
Redoplicaion
Case Syncectism
Syncetiem
Nemun/Number Marking
Subject, Object and
82A
Order et Subject and Vert
BERT is layered transformer- based model with attention (Vaswani et al., 2017) that uses only the Encoder block. Each layer of the Encoder block is divided into two sub-layers. The first sub-layer, called Attention Layer, heavily relics on the at- tention mechanism. The second sub-layer (Feed Forward Layer) is simply feed-forward neural network. Clearly, each sub-layer has its weight ma- trices referred to in the rest of the paper as follows:
83A
Onder of Ofject and Necb
84A
Onler of Orjeci Ohlique and Varh
85A
Onler of Adposiionn and Noun Please
86A
Onder Gemine and Noun
and
Order
Fosition Question Particles
91A
Fostion Interrogative Pttrases in Content Questions
94A
Ordes of Adverbial Suberdinator and Clause
95A
Relatonship between the Orker of Objjett and Vorh and the
Ordes of Adpastion and Nomn Please
96A
Relatinship betwarn the Orker of Object and Verb and the
Under Reiative Clusse Noun
Relationshpp
Objjcct Vorh
of Acjective
143A
Onher of Negeive Morpheme and Virbh
1836
Preveroal Neguziee Morphemes
143F
Fostvetbal Negutive Morphemes
Attention
144A
and Vad
Position of Negutive Word With Respect to Subject, Ubject,
query
Qi
key
K;
Table 2: Selected morphological and syntactic features of WALS
value
V
attention output dense
OA
Feed Forward Network
erate metric vectors. Our approach is similar to lang2vec (Littell et al., 2017). In particular, each syntactic and morphological pair of feature-value (f u) is one dimension of this space, which is I for language L if has the feature f equal the value v. With this coding scheme, languages are represented as boolean vectors.
intermediate dense
DI;
output dense
DO;
For each monolingual BERT for a language L, weight matrices W; [Qr, K, V,, OA, DI, DO,} for C [O...., 11] should represent part of the linguistic model of L after pre- pre-training.
Thus, the similarity o L2) between two lan- guages L, and I.2 can be computed as the similarity between the language vectors Li and L2, using the cosine similarity (cos) as similarity measure:
3.3 Bidimensional Centered Kernel Alignment for Comparing Weight Matrices
oLL,, L2) = cos(L1, I2)
Centered kernel alignment (CKA) (Korbblith al., 2019) is metric used compare similarities be- tween representations, that is, activation matrices of the deep neural network. Linear CKA measure is defined as follows:
This measure assesses the similarity between lan- guages by counting the number of shared feature- value pairs. We can compute two versions of this measure- - xyyut (L1, L2) and Tmorph (L1, L2) -hiich respectively compute similarity among lan- guages using syntactic and morphological features. Although this formulation is similar to lang2vec, we encoded the WALS features directly into metric vectors because lang2vec does not contain mor- phological features and cannot benefit from our integration of WALS for the syntactic feature.
T CKA(X,Y) = |
(I)
where X and Y are n x p matrices and denote acti- vations of p neurons for n examples. The key idea behind this metric is to determine the similarity between pairs of clements and then compare the similarity structures. As presented by Kornblith et al. (2019), computing the numerator in Equa- tion proportional to computing the similarity, using the dot product between the lincarized ver- sion (which is referred to as vec) of the matrices
Then, we can assess the typological similarity of the languages as defined by linguists. We can now investigate whether the same similarity holds among different BERT models.

---

similarity measure, of the estimated covariance matrices XX" and YY for mean-centered acti- vation matrices X and Y:
following block diagonal matrix F(W) follows:
W 0
0
F(W)
WT
xe e)c(YY trace(Xx²xXx)) || X
(2)
Then, biCK A. our solution compute the similar- ity between weight matrices, defined as follows:
where vec(A) is the linearization of a matrix A and, thus, vee(XX"))* vec(YY")) is the equiva- lent of the Frobenius product between two matrices. Then, CKA is suitable to compare matrices rep- resentations of sentences as it is possible to derive relations between rows of the two matrices as these represent words.
biCKA(W, W2) = CK CK
Hence:
BCCKAW W2) X
WWT 0
0
W
WgW.
Among the different methods, CKA is chosen because it allows one compare matrices with p n. Other metries that are invariant to invertible linear transformations, such as CCA and SVCCA, instead assign the same similarity value to any pair of matrices having p 2 n (Kornblith et al., 2019). Morcover, the columns of the analyzed weight ma- trices are characterized by high multicollinearity. If one feature is linear combination of others, then also the covariance matrix has some row that is linear combination of others and hence is rank defi- cient. This makes difficult this setting the use of PWCCA since it requires the computation of CCA and, consequently, the inversion of the covariance matrices.
AWW, W2) - relying on the normalized similarity between the block covariance matrices of F(W,) and F(W2) takes into account the similar- ity between rows and columns of weight matrices W1 and W2.
Given a pair of weight matrices, WI and W2. biCK A is our solution to compute their similarity.
Experiments
4.1 Experimental set-up
The aim of the experiments is to use typological similarity between languages determine whether some BERT layers encode morphological syn- tactic information.
However, in the case of weight matrices, CKA is not suitable as both rows and columns of these matrices play an important role in determining the output of the network. For example, during the computation of the self-attention in a layer given X;-1 the output of the previous layer, the product between the query activation matrix X,-1Q; and the key activation matrix (transposed) X; K; is often described as a function of column vectors However,
Our working hypothesis is the following: given a set of pairs of languages L2), a particu- lar matrix W of a BERT layer encodes syntac- tic or morphological information if the similar- ity biCK A(W Wt,) between that particular weight matrix of BERTL, and BERTL, correlates with the typological similarities o ssyna L1, L2) morph (L1, L2). respectively. To evaluate the cor- relation, we compared two lists of pairs ranked according to biCKA(Wi, Wia) and o oLL, L2) by using the Spearman's 's correlation coefficients. To select languages for the sets of language pairs, we used Hugging Face that offers considerable repository of pre-trained BERTs for a variety of languages. Hence, we had the possibility to select typologically diverse languages for our investiga- tion. Languages are listed along with the respective pretrained monolingual model (retrieved from the Hugging Face repository using the transformers library (Wolf et al., 2020)).
of Q; and K; as (x- the attention mechanism is implicitly computing the product Q, K? (where rows of Q, and K, are compared). Hence, both covariance matrices of rows and columns describe something about the input transformation and need to be studied.
Indeed, given a pair of weight matrices Wi and W2. both the similarity W1 WT with respect to W" and the similarity of W1'Wi with respect to W W2 may play an important role in determin- ing the similarity between the two weight matrices.
Hence, we introduce bidimensional CKA, biCKA, A, a variant of CKA that compares matri- ces considering rows and columns.
We selected the following European languages, listed above, according to genealogical criteria:
Firstly, given a weight matrix W, we define the

---

Indo-European
Romance
(or Italian
The FULL experiments aim detect where BERT models for typological similar languages may encode syntactic and morphological features, using WALS-based similarity among languages and our novel biCK A among weight matrices.
Neo-Latin) languages:
(ITA), bert-base-italian--ass
(Schweter, 2020c); French
(FRE),
bert-base-fremee saaas--rrrnc--eurpppa--ccss
(Schweter, 2020b); Spanish
(SPA),
The FOCUSED experiments on clustering lan- guages aim to be more specific in describing whether some weight matrices encode specific lin- guistic information. Indeed, despite being all Euro- pean, selected languages show different typological features. Hence, these languages may be clustered according their WALS similarity on typological vectors. Then, we identified distinctive features among pairs of clusters by comparing the features Gini impurity (Ceriani and Verme, 2012) of each cluster and their union. Given two clusters, such as S1 and S2, the most interesting features will be those features that distinguish the two clusters. The impurities of these features in S1 and S2 are as low as possible and, at the same time, present in each cluster lower impurity than the impurity measured in the union clusters $1 U S2. Among these interesting features, we will refer to features that take entirely different values between clusters as polarizing features. Once those features are de- tected, to understand if they are encoded by given matrix, compute ranked lists of language pairs, selected pairs of languages belonging to dif- ferent clusters. The intuition is that the similarity between these pairs of languages, and, thus, Spear- man't 's correlation between ranked lists, is mainly affected by polarizing features.
(Cañecte et al., 2020); Romanian (ROM),
baw.baae-rrmmaia--ccsceee mitrescu et al., 2020);
(Du-
Indo-European Germanic languages: English (ENG), bert-base-uncased
(Devlin et al., 2019); Swedish (SWE),
bert-base-seedish cased
(Malm-
sten et al., 2020); German (GER),
bert-base-german-cased (Deepset, 2019);
Dutch (DUT), bert-bbas-dutch--case (de Vries et al., 2019);
Indo-European Slavic languages: Russian (RUS), sibert-base-csee. (Kuratov and Arkhipov, 2019);
Indo-Europpan Hellenic languages: Greek (GRK), wew-bbae--ggee--uncseeeeee Kout- sikakis et al. (2020);
Non-Indo-European ish (TUR), bert-base-turkish-aass (Schweter, 2020a); Finnish (FIN), bert-bae-riimish-caeed
European languages:
Turk-
Virtanen
et al. (2019)
In addition, we included Persian (or Farsi) (PRS), belonging to the Indo-Iranian branch with the corre- sponding model hani et al. (2020), which is one of the most repre- sentative Indo-Eut Suro-en language spoken outside the European continent.
Finally, the last DOMAIN ADAPTED experiment shows how models behave in a setting closer to the ideal one, with shared training procedure and training data. Since data are semantically identi- cal, the only differences in morphological and typological features, which hence we hypothesize should be represented similarly by models trained on similar texts. Domain adaptation is performed by fine-tuning each model on Masked Language Modeling task. The data used are from the Eu- roParl corpus (Koehn, 2005), large-scale parallel corpus of proceedings of the European Parliament. Hence, we focus on languages spoken in the Eu- ropean Union, excluding Turkish, Russian, and Persian in this analysis since no translation is avail- able for these languages. Each model is trained for four epuchs on subset of sentences that has a translation in all languages (50k sentences).
www.aaaaaannaasssseeeeess by Fara-
Given the above list languages, we performed three different sets of experiments:
1. FULL - all pairs of languages in the list are considered in the set;
2. FOCUSED first, we cluster languages in groups and then only pairs of languages from different groups retained;
3. DOMAIN ADAPTED on a selected set of lan- guage pairs, we perform domain adaption of the models on sentences from a parallel cor- pus.

---

5 Results and Discussion
the attention sub-layer. In fact, matrices OA4, Vs. O.As, O achieve Spearman's 's value higher than the fixed threshold, respectively, 0.60, 0.52, 0.66, and 0.55. The higher correlation value is 0.66 for O All Spearman's 's are tested against the null hy- pothesis of the absence of linear correlation and are statistically significant with p-value smaller than 0.01 with Student t-test. These results align with the probing experiments that assessed the predomi- nance of middle layers in the encoding of syntactic information (Tenney et al., 2019).
This section reports the results of the three sets of experiments: FULL all pairs of languages (Sec. 5.1); FOCUSED only pairs from different clusters (Sec. 5.2); DOMAIN ADAPTED analysis is performed after domain adaptation a parallel corpus (Sec. 5.3).
5.1 Correlation between Matrices in all Languages: the FULL approach
Morphological properties instead show up in a single matrix V3. The correlation between typolog- ical space and the matrix is moderate, that is, 0.55. Moreover, the correlation seems to be moderately stable until the layer 7 on these matrices V.
0.27
-0.15
0.21
0.07
0.07
-0.03
0006
0.29
0.44"
0.27 0.22 -0.44
0.18
-0.02
0.20
0001
0.15
..19
-.06
0.37
0.43
0.35-
00110
0.29
0.38"
0.39
0.52
0.43"
0.17
0.00 0.44
This analysis confirms our hypothesis that syn- tactically similar languages induce similar models and that this can be observed directly on weight matrices layers 1, 5, and 6, expected since probing tasks detect syntax in middle layers (Ten- ney et al., 2019).
0.34
0.02
0.12
6.6
0.66
0.55*
0.05
0.28
0.20
DO DI
0.335 0.17 0.16 0.15
0.14
0.20
0.25 0.07
0
19
-0 16
0.03
0
0. 10.341.0 0.30 -0.09 13.13 .0.29 -.05
00.40
0.13
9
10
11
(a) Syntactic Similarities
5.2
FOCUSED analysis: measuring Extra-Cluster Correlation
-0.0
0.18
0.22
Experiments on correlations of the ranked list of language pairs extracted from different language clusters aim to study whether specific linguistic features are related specific weight matrices in BERT.
0.17
.0.14
0.11
0.03
0.24 0.05
0.04
0.15
0.23
0.28 0.28
0.05
9.04
0.97
0.12
0.55
0.32
0.49
0.32"
0.43 0.45
0.23 0.20
0.41"
0.43
0.05
0.36*
For this set of experiments, we performed K-means clustering of languages (see Ap- pendix A.2) for both syntactic and morphological typological spaces. Four clusters emerged for the syntactic typological space languages, and three clusters emerged on the morphological one. The clusters (S) generated from syntac- tic features are: $1 = (ITA, FRE, SPA, ROM), S2
-0.24 0.27
0.08 011
0.08 0.17 0.18 0.12 0.11 0.06
0.01
0.33*
D0
.0.21
0.04 0.13 0.06 0.08 -0.0 0
08 0.16 0.08 0.02
-0.29
0.06
2
5
9
10
11
(b) Morphological Similarities
Figure 1: Spearman's correlation coefficient over all language pairs ranked with typological features and with weight matrix similarities. Rows are the matrices types, and columns are the layers. Values closer to +1 are in red, closer 0 in black, and values closer to -I are in blue. Statistically significant results with p value < 0.01 are labeled with".
(ENG, FIN, SWE, RUS, GRK], S3
{GER, DUT). The clusters (M) generated from morphologi- cal space are: M1
[TUR, PRS} and S4
{ITA, FRE, SPA, ROM).
M2
{ENG, SWE, GER, DUT) and M3 [TUR, FIN, PRS, RUS, GRK).
Experiments in the FULL configuration using all pairs of languages in single set have relevant results showing that typological similar languages led to similar models. In particular, the syntactic spaces in WALS correlate with specific layers in BERT (see Fig. 1).
The Spearman's correlation is reported for each matrix in the different layers, for both syntactic (Fig. 2a) and morphological (Fig. 2b) clustering: focus here on results obtained by comparing the larger clusters since the Spearman's 's correlations are negatively affected by smaller cluster size. A complete list of correlations can be found in Ap-
Syntactic properties seem to emerge in middle layers (Fig. la) for content matrices 0.4 and of

---

In fact, Spearman's coefficients are above the threshold across multiple layers and are statisti- cally significant with peak layer (Qo ma- trix). However, a high correlation can also ob- served in other layers, with no clear descending trend. Some of the polarizing and nearly polar- izing features are 29A Syncretism in Verbal Person/Number Marking, 21A Exponence of Selected Inflectional Formatives, 23A Locus of Marking the Clause.
0.32
0:38
.0.33
0.84
0.54
0.37
0.35
0.00
0.29
0.43
0.15
0.56
-3.35
0.29
4.16
0:24
0.65
-.40
0.37
0.11
0.31
0.47
0.52
0.09
0.16
0.29
0.00
<0.29
0.11
0.67
0.63
0.27
-0.24
0.02
0.14
4.33
D
0.03
0.13
0.31
0.12
0.01
0.18
0.40
0.10
0.72
0.46
-0.14
0.26
D0
0.08
0.31
10
11
Although these results are not conclusive, these experiments on clusters of similar languages estab- lish possible new way to investigate the linguistic properties of BERT.
(a) Syntactic Similarities Clusters S1 and S2
0.82
0.06
0.24
0.05
0.58
0.03
0.15
0.06
0.16
0.14
0.02
0
751
0.69
0.54
0.24
0.05
5.3
Correlations after Domain Adaptation on Parallel Corpus: the DOMAIN ADAPTED approach
-0.13
0.19
0.18
0.30
0.39
0.57
0.46
0.43
0.28
0.13
0.31
0.52
DO O
-0.83
-0.25 -0.02
00.02
0:01
0.21
0.25
0.04
0.19
-0.65
-0.35
0.01
0.01
0.17
0.21
0.08
0.28
0.
0.19
0.75"
0.38
0.14
0.13
.0.35
0.28
0.43*
0.12 -0.05 -0.06
0.36
-0.24
0.05
0.20
0.49
0.45*
0.24
0:10
3
4
5
8
9
10
11
-0.08
.0.16 0.26
-0.14
0.35
0.22 0.34
0.14
0.18
6.11
-0.11
0.26
(h) Morphological Similaritic on Clusters and M3
QA
0.15
0.85
0.13
0.01
0.00
Each matrix shows Spearman's correlation coefficients for extra-cluster typological similarities and weight matrix similarities. Rows are the matrices types, and columns are the layers. Values closer to +1 are in red, closer to are in black, and values closer to -I are in blue. Statistically significant results with
Figure 2:
-0.05
0.32 0.64
0.05
0.43*
DO
-0.24 0.11 -0.01
0.04
-0
19
0.28
0.02 0.22
0.03
-
21
-0.16
0.06
1
7
8
10
p - value < 0.01 are labeled with
(a) Syntactic Similarities before domain adaptation
0.08
0.17
0.09
0.17
0.37
0.04
0.24
0.13
0.18
0.32
0.08
0.02
pendix A.2.
From the syntactic point of view, the two larger clusters S1 and S2 show an interesting pattern (see Fig. 2a). The threshold of 0.5 is exceeded by the matrices in layers from 4 to 8, with a peak of 0.77 in layer on matrix V/g These values are statistically significant with p-value lower than 0.01. Those results confirm what has been observed the pre- vious section. Hence, these matrices may encode the polarizing features 87A Order of Adjective and Noun and 97A Relationship between the Order of Object and Verb and the Order of Adjective and Noun.
0.21 0.24
0.56
0.48
0.35
041 0.38
0.24
0.10
-0.11
0.16
0.29 0.32
0.43
0.40
6.26
0.34
0.37 0.33 0.16 0.01 0.11 6.13
0.40 0.58" 0.60
DO
0.20
0.41 0.50*
0.16 0.21
0.29
(b) Syntactic Similarities after domain adaptation
Figure 3: Spearman's correlation coefficient Fu- ropean language pairs ranked with typological features and with weight matrix of domain adapted BERT simi- larities 3b and of pretrained BERT similarities. Rows are the matrices types, and columns are the layers. Val- ues closer +1 are red, closer to are black. and values closer to -I are in blue. Statistically significant results with a - value < 0.01 are labeled with *
From the morphological point of view (see Fig. 2), language pairs from M1-M3 lead to interesting results. Investigating the morpholog- ical clustering, less regular patterns can be found by looking at Spearman's s correlation coefficient.

---

In this section, we observe Spearman's corre- lation on subset of models after domain adap- tion. As described in Section 4.1, we analyze here models adapted on a parallel corpus with Masked Language Modeling task. When selecting models for the FULL setup, we observed that they were all pretrained on wide variety of sources and do- mains. However, dissimilar pretraining data could negatively affect the analyzed correlations. Our hypothesis is that conditioning models' weights on semantically equivalent sentences can lead to a clearer analysis of syntactic similarities across typological similar languages. We focus here on syntactic analysis since it leads to more interesting results in all set of experiments, while morpholog- ical analysis for completeness is reported in the Appendix A.3.
vious approaches, our methodology is based on directly comparing layer-by-layer weight matrices of BERTs and evaluating whether typological simi- lar languages have similar matrices in specific lay- ers. From a different standpoint, our experimental results confirm that layers 4, 5, and 6 encode syn- tactic linguistic information. Moreover, they also suggest that the attention's 's value matrix V and the attention's 's output layer are more important than other matrices in encoding linguistic models. This latter is an important and novel result. In fact, these findings can be important because they can help interpret the inner workings of these models to go toward the so-called actionable explainability indi- cating which matrix weights could be changed to obtain desired behavior. Moreover, this method- ology could help to build more informed archi- tecture that takes advantage of the most promising layers in multilingual models.
Our hypothesis is confirmed by the syntactic analysis experiments since the positive trend is confirmed across the middle layers, and relatively higher statistically significant correlation coeffi- cients can be observed (Figure 3). We can identify positive trend from layer 2 to layer 6 on matrix DI, and on matrices V and OA with smaller yet consistent positive values on the same layers. The average increase in correlation from layer to layer 6 0.18. Morcover, the correlations do not tend to increase on the latter layers (especially layers 10 and 11), which are the layers generally associated with semantic tasks.
Hence, in future work, our findings could be helpful: (1) for defining cross- language training procedures that consider similarities between lan- guages and between models, and (2) for fostering ways act in specific weight matrices of specific layers of BERT to change the undesired behavior of final BERT downstream systems. Moreover, this methodology could be used on other transformer architectures.
Limitations
These experiments give us an important confir- mation that the typological similarity across dif- ferent models, while being present in pretrained models, is enhanced by conditioning weights to se- mantically equivalent data, which thus differ only syntactically and morphologically.
To the best of our knowledge, this is the first attempt to directly compute similarities between weight matrices of BERT and to compare it with an external resource. Hence, it has many possible limitations.
The more important limitation can be due the fact that transformers, in general, and BERT, in particular, could be mostly large memories of pre-training examples. Hence, comparing weight matrices at different layers could imply comparing pre- pre-training examples given to the different BERTs. However, this is not only limitation of our study. Indeed, could be a limitation for any linguistic analysis sis of BERTs or other transformers.
6 Conclusions
Understanding whether large language models en- code linguistic models is challenging and impor- tant research question. Despite their flexibility and their SOTA performances on multiple NLP tasks, transformer-based models do not explicitly express what they learn about language constructions and, thus, it is hard to control their failures directly.
The second limitation is the availability of mono- lingual BERTs for low-resource languages, which led to analysis that is incomplete. The growing availability of monolingual BERTs can solve this issue and may require to re-do the experiments.
Our paper proposes novel way to observe BERT capability to encode linguistic models and achieves two critical contributions: confirming syn- tactic probing results from another point of view and opening a new way to investigate specific lin- guistic features in layers. Differently from all pre-
The third limitation is the incompleteness of the World Atlas of Language Structures (WALS)

---

(Dryer and Haspelmath, 2013). Indeed, as this is growing linguistic resource, our results also de- pend on the quality of the resource at the time of download. For this reason, we selected languages that may be controlled by our linguists.
Michael Dunn. 2015. Language phylogenies. The Rout- ledge handbook of historical linguistics, pages 190- 211.
Daniel Edmiston. 2020, A Systematic Analysis Mor- phological Content in BERT Models for Multiple Languages. ArXiv:2004.03332 [es].
References
Mehrdad Farahani, Mohammad Charachorloo, Marzieh
Farahani, and Mohammad Manthouri, 2020. Pars- bert: Transformer- based model for persian language understanding. CoRR, abs/2005.12515.
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou- Hui Ho, Hojin Kang, and Jorge Pérez. 2020. Span- ish pre-trained bert model and evaluation data. In PMI4DC at ICLR 2020.
Yoav Goldberg. 2019, Assessing bert's 's syntactic abili- ties. CoRR, abs/ abs/1901.05287.
Lidia Ceriani and Paolo Verme. 2012. The origins of the gini index: extracts from variabilità mutabilità (1912) by corrado gini. The Journal of Economic Inequality, 10(3):421-44)
Joseph H. Greenberg. 2005. Language Universals. De Gruyter Mouton, Berlin, New York.
David R Hardoon, Sandor Szedmak, and John Shawe- Taylor. 2004. Canonical correlation analysis: An overview with application to learning methods. Neu- ral computation, 16(12):2639 2664.
Ethan A. Chi, John Hewitt, and Christopher D. Man- ning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3564-5577 Online. Association for Computational Linguistics.
Martin Haspelmath, editor. 2001a. Language Typol- ogy and Language Universals Sprachtypologie und sprachliche Universalien La typologie des langues et les universaux linguistiques. De Gruyter Mouton, Berlin, Boston.
Noam Chomsky. 2017. The language capacity: architec- ture and evolution. Psychonomic bulletin & review, 24(1):200-203.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda B. Viégas, and Martin Wattenberg, 2019. Visualizing and measuring the geometry of BERT. CoRR, 00/19006622155
Martin Haspelmath, editor. 2001b. Language Typology and Language Universals 2.Teilband. De Gruyter Mouton, Berlin, Boston.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Comrie. 1989. Language Universals and Linguistic Typology: Syntax and Morphology. University of Chicago Press.
Wietse de Vries, Andreas Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. Bertje: A dutch BERT model.
CoRR, abs/1912.09582.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3651-3657 Florence, Italy. Association for Computational Linguistics.
Deepset. 2019. German bert.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4185 Minneapolis, Minnesota. Association for Computational Linguistics.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022. Lifelong pretraining: Continually adapting language models to emerging corpora. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4764-4780, Seattle, United States. Association for Computational Linguistics.
Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig-
Stefan Dumitrescu, Andrei-Marius Avram, and Sampo Pyysalo. 2020. The birth of Romanian BERT. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4324 4328, Online. Association for Computational Linguistics.
Philipp Koehn. 2005. Europarl:
A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79-86. Phuket, Thailand.

---

Simon Korbliihh Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR.
Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Taraka Rama, Lisa Beinborn, and Steffen Eger. 2020. Probing multilingual BERT for genetic and typo- logical signals. In Proceedings of the 28th Inter- national Conference Computational Linguisties, pages 2214-1228, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
John Koutsikakis, llaas Chalkidis, Prodromos Malaka- siotis, and Ion Androutsopoulos. 2020. Greek-bert The greeks visiting sesame street. In 1 Ith Hellenic Conference on Artificial Intelligence, SETN 2020, page 110-117, New York, NY, USA. Association for Computing Machinery.
Vinit Ravishankar, Memduh Gökımmak, Lilja Øvrelid, and Erik Velldal. 2019. Multilingual probing of deep pre pre-trained contextual encoders. In Proceedings of the First NLPI. Workshop on Deep Learning for Nat- ural Language Processing, pages 37-47, Turku, Fin- land. Linköping University Electronic Press.
Yuri Kuratov and Mikhail Y. Arkhipov. 2019. Adapta- tion of deep bidirectional multilingual transformers for russian language. CoRR, abs/1 abs/1905.072111 1905.07213.
Winfred P Lehmann. 2013. Historical linguistics: an introduction. Routledge.
Stefan Schweter. 2020a. Berturk bert models for turk- ish.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2. Short Papers, pages 8 14, Valencia, Spain. Association for Computational Linguisties.
Stefan Schweter. 2020b. Europeana bert and electra models. Stefan Schweter. 2020c. Italian bert and electra models.
Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is not an interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47-55, Hong Kong, China, Association for Computational Linguistics.
Martin Malmsten. Love Börjeson, and Chris Haffenden. 2020. Playing with words at the national library of sweden making a swedish hert.
Ari Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural net- works with canonical correlation. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Jac Jung Song. 2010. The Oxford handbook of linguistic rypology. OUP Oxford.
Jae Jung Song. 2012. Word order.
Ian Tenney, Dipanjan Das, and Ellic Pavlick. 2019. BERT rediscovers the classical NLP pipeline. Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguisties, pages 4593- 4601, Florence, Italy. Association for Computational Linguistics.
Dmitry Nikolaev and Sehastian Pado. 2022. Word- order typology in multilingual BERT: A case study in aubordinate-claus detection. In Proceedings of the 4th Workshop on Research Computational Lin- guistic Typology and Multilingual NLP, pages 11-21, Seattle, Washington. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Dario Onorati, Leonardo Ranaldi, Aria Nourbakhsh, Elena Sofia Ruzzetti, Arianna Patrizi, Francesca Fal- lucchi, and Fabio Massimo Zanzotto. 2023. The dark side of the language: Syntax- based neural networks rivaling transformers in definitely unseen sentences. In Proceedings of the WI-IAT conference.
Yulia Otmakhova, Karin Verspoor. and Jey Han Lau. 2022. Cross- linsu-iing comparison of linguistic fea- ture encoding in BERT models for typologically dif- ferent languages. In Proceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 27-35, Seattle, Wash- ington. Association for Computational Linguistics.
Antti Virtanen, Jenna Kanerva, Rami IIo, Jouni Luoma. Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish.
Ivan Vulié, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš. and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222-722 Online, Association for Computa- tional Linguistics.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein, 2017. SVCCA: Singular Vec- tor Canonical Correlation Analysis for Deep Learn- ing Dynamics and Interpretability. In Advances in

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of tftte-of-thearr the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Fabio Massimo Zanzotto, Andrea Santilli, Leonardo Ranaldi, Dario Onorati, Pierfrancesco Tommasino, and Francesca Fallucchi. 2020. KERMIT: Comple- menting transformer architectures with encoders of explicit syntactic interpretations. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP), pages 256-267, Online, Association for Computational Linguistics.

---

A Appendix
A.1 WALS and integration of morphological and syntactic features
In this Section we present the complete list of WALS features used in this work. The values listed in the following tables are defined from the WALS website and, where needed, analyzed and inserted from the linguist Felicia Logozzo, as also described in Section 3.1.
SIA "Order Subject, Objectt and Venk"
82A of Subjecr and Verb No dominant order
83A Order of Object and Vorb
84A "Orier of Object Obiiqu, and Verb"
8SA
86A
wals_
Order of Adpesition and Noo Pleuse
Order Genätive and Nown
2SV
2V
VVX
Prepositions
2 Noun Genitive
eng
2SV
18V
2VD
1 VOX
Prepasitions
No domiannt order
tur
NOV
1SV
TOV
XOO
Posposctions
Genutive-oor Noun
Bi
2SV
18V
2VD
1 VVX
Posposttoon
1 Gesttive-Nou
SWe
2SVO
18V
2VD
VOX
Prepasitions
Geutive-Noun
18V
T
XXOV
Prepositions
2 Noun-Geniive
ger
No dominant onder
18V
No demtiann order
No doeninant order
Prepositions
2 Noun- Monn-Geniive
fre
SVO
1SV
VO
I VOX
Prepositions
2 Noun Genitive
TUS
2VO
1SV
V0
I VOX
Prepositioes
2 Noun Genitive
1pa
SVO
No dominant order
V0
VVO
Prepasitions
2 Noun Geniiive
dut
7 No dominani onder
1S
No dominant order
No dominant order
Preposinces
2 Noun Genn-ien
SV
V0
VOX
Prepositioes
2 Noun- Genuiive
gr%
7 No dominant onder
No dominant order
VV
6 No dominant order
Prepositios
2 Nomn-Genniie
S7A Odter of
ABA
12A
34
Ordir
Putssnaf (burios
Ondir d' lish
1 Fen Vew
Threewr the News
Thrrendeer puivi
Teiee rimigiie flw
Thiol devileeeeeeee will
IWizel Pag--iees
184
Le New
Tiaeerrain ane Nema
Nateraleenng pare
ITV Enppettion
Hizs.
2 tzzallizoogghh pliue
Insal soveiaae werd
and Pipo
Basearanchhea
N caoase parsde
1ixni minggain plaa
Intal peeebae wed
3
8x
liska
leznl zkmggare ptrae
Iniileeee wor
4ND Popestana
191
Pell THe Vea
doe News
TIIMI remolie flow
eevileeeeeeee ond
Ti/ei
000
Mial
qunoee pankl
Inturnaniiaa ond
grt
Hists
skmgeine plaax
4N0 Prgoikka
vbb mvd
Ouer of and Mocken
Adceeryy hhvne
Oske dfoeeco Hohood
Otfine
Pooede Haplower Rossial Moodowe
"hullin hoparr Ror mas
144
L10 Nia
Ehey
(hee
244
1362
EVIl Nit
Theyy
FA4
Men Daa pertiia
TTye
(V0
14
Vi all Neal
vit New
1s Veni rar reaeeiiee
20A
21A
ZEA Iefiectional Syutheads of Norh
23A Locus of Marking
2AA Locus Marking
25A
walk code
Fusion of Sclovded Inflectional Farmanivss
Exponencc of Seliitte Infurtional Formatiion
Lacus Marking:
the Clause
Porverrioe Noon Phiuses
Whote-Jompwape Tipolegy
Aa
Exclusovely concueeaative
No case
34-5 calopocies per wood
No marking
2 Dependem marking
Eaconsisient or other
Escluevely concsterative
No case
2 calegoeics wood
Dependent marking
2 Dependeni marking
2 Deyenslent-merking
IUr
Excluevely concstetative
Monoespasestal case
167 calogocies per word
Dopendem mariing
Double marking
Incousisient or other
lin
Eschnévely concaterorive
2 Case nimber
cilggrrsss per wori
Dupendeni marking
Dimble marking
5 Invemsiientt or afher
Excluevzly
Foo
223 caloposies per wood
No marking
Dependen marking
Exconiisignt other
DPA
Fachnevvll soncsterstive
Moneesponntialee csse
14-5 calegerios pet weed
Doubie marking
Hoad murking
Insomiitent or other
gec
Exclusvzly soncutetative
2ass mimber
2233 calegocies word
Dependem marking
2 Dependen: marking
2 Dependeat marking
Eachuévvlyy concalemorisu
calegrios per word
marking
Dependoni narking
Incomsistent
TUS
Excluevaly soncaeetative
Case + number
4-5 calegocies per word
Dependen marking
2 Dependem: marking
2 Dependent marking
yp
Fachnévely concstoretive
Monvesponetiall case
14.5 calegrirs weed
Douice marking
2 Dependeni warking
Invomiitent or other
Excluevely soncstesutive
case
2.3 calogocies per werd
No markin
Dipendem: marking
Incousisient or other
rom
Esclnéively concatenorive
3 Case referntiality
4-5 calegeries per word
Dependent marking
2 Dependeni marking
2 Dependent-
Eaclusvely
2asc mumber
calegooles word
Double marking
Double marking
Double marking
mal_sode
Prasing ns. SufKring de Ayfoctional Masphology
261
27A Redepitctiinn
28 Cur Syscertoe
21B Eqprence of Teson-Apuert Moed lnfoctice
Zee Manking 4 a P Arganeen
250
29A Syncritvee is
Lisbol PargaeNaskkrrr Marking
Siineply valining
Nuppodiziie redapliction
Core netecorc
2 Nomezzm meeling
Sissuall
muoreppsccaaa TAN
Siroogl sittiing
Productive full ind portal indilliation
syacreist
moo-eigoethall TAM
Non murking
Nor syneretic
in
Simoply samting
No Whoprddiiie magplicaionn
moneeggormial TAM
Now mering
Nor synerite
Strurly valiiinu
S Nappacctive sedupliation
2 Core casc only
monoesgenciiall 'AMM
Now-zzm merkin
1 Syrenstic
Ms
5
Weekly arthiing
Proticniv fill and ponial mntplisation
No can markiing
novveprortial TAM
Non 72m marking
5 Nor synerme
Simoply valining
NNupadizine redepliation
Core menecnee
minccgennitall TAM
Nomezz me bin
Sussual valfisiu
5
IUs
Sinogly satlking
prouctictiv redvoliztionn
Core and Bon
moneepooetaal TAM
Non eero marking
No. synertte
Simngl samiing
Saoprsdiii marplicaen
Uow and now zore
2 1iMeogreeee
Non 7rm markigg
Syncceen
du.
Smouly suiliing
3No prodcctii redupliaiion
Core zases only
amoouupciii TAM
Non 2210 mazking
Synoreii
mm
Smongy sutthing
No prodcttive rndupliation
5 Care and non zore
2 TAA--igmemmn
Non zem marting
2 Syncnetc
nk
Simoply valining
Napprdtriie redepliation
Uome mon-core
Nom-zzm merling
3 Nor synerzhe

---

A.2 Correlation Results Clustered languages
Clustering of languages As described in 5.2, a K-means clustering of languages is performed for both syntactic and morphological typological spaces. Four clusters emerged for the syn- tactic typological space of languages, and three clusters emerged on the morphological one.
tur
prs
prs
jur
Jin
130
Jre
grk
ger
ger Aut
$pa jta
rom
eng
dut
swe
swe
-30
eng
Jus
4k
fre
jta
Spa
Figure 4: I-SNE plot of clustering based on syntactic (4a) and morphological (4b) features extracted from WALS.
Results Analysis Here we report results between all syntactic and morpholocival clusters. In Figure 6 and Figure 5, one for each matrix, two different clusters are considered.
From the syntactic point of view, described in Section 5.2, middle layers similarities are positively correlated with typological similarities and this positive correlation can be observed also after clustering, in Figure 6a, when sufficient number of languages is considered. Results are very unstable when one of the considered cluster smaller (see Figure 6b - 6f). In fact, Spearman's 's values that exceed the threshold of 0.5 fluctuate too much across different layers and matrices and no pattern can be hence clearly observed. Results are rarely statistically significant, with the exception of Q3 in Figure 6e. Hence, it is hard to assert that the polarizing features of these clusters are encoded at some layers using Spearman's correlation coefficient.
From the morphological point of view (see Fig. 5), language pairs from M1-M3 and, marginally, from M2-M3 lead to interesting results. Indeed, related ranked lists for pairs for these clusters show extra-clustering Spearman's coefficients that are above the threshold across multiple layers and that are statistically significant. The rankings pairs generated from M1-M3 and M22M3 have peaks at layer 0 (Qo matrix) and layer 3 (V/ matrix), respectively. However, a high correlation can also be observed other layers, with no clear descending trend.
4000 0w--------00 384 032-20 438 425 C2) 4.33 4.12 4.15
4:40 63 011 6/ 5.16
4011
4.01
0.v 621 4.55
322 4.12
4.10
6.26
324 0.57 6M 236
211 01103114400 4.2 0.28 428 0.43 421 36 43 0.15 0.00 -107
0.0 8:7
0.5 0.00 0.7 0:79
4.25
0.14
500
0.25
333
0.42
0.6
0:5
0311
0.28 022 0/55 3822005
0:90 324 037 0.20 3:00 031 6m 838 922
004 0/3 927
031 921
0:33
0:25
9
0.0
2:8
034 0:5
3.5
027
0:33
008
914 014 924 031 00 42
(a) Clusters M1 and M2
(b) Clusters M and M/
(c) Clusters M2 and M3
Figure 5: Each matrix shows the Spearman's correlation coefficients extra-cluster morphological analysis, one matrix for each pair of clusters, M, and Mj. Values closer to +1 are in red, values closer to -I in blue. Statistically significant results with a a p-vuuue lower than 0.01 are labelled with *

---

228
0.33
6:57
0.37
0.0
0.73
0.33
6.90-0.33
0.73
0.50
9.5
0.75
6.216.20
6:00
5.00
-5.00
40.55-4900
0.5
C30
826
422
0.27-00000
6:01
4.34
6.34
0.23
0:1
6.70
560
932
6.2
0.23
6.40
0.21
224
5:00
-4:45
2.004.00
8:75
0.33
40.1
85
0:15 1.44444
0.38 422
0.23
6.2 4.23
122
0.23
6:00
0.7 0.3
8
0.39
0:30
012
0.00
10
11
12
(a) Clusters S1 and $2
(b) Clusters S1 and S3
(c) Clusters S1 and S4
0.20
031 5.0
6.15 8:10
0.89
938
0.30
0:80 9.45
0.39
338
134 0M
0.16
0:34
0.23 0.31
3:55 0.3
0.29
0.00 0.83
8
4:6
3:56
0.6
0.35
445 6.2
0.10
4:20 3:43
666
3.60
0.93
6:21
3.38
0.533
93
3.22
3:45
0.51
3:45
0.99 1999--000
0.1
055
222
0.13
2222
3.2
4.0
a26 451 332
8
4:37
8:3
0.39
4:30
3.42
4.4
0:5
000
330
0.31
051
530
0.00
0.45
4:45
3.45
0.3
0/0
3.0 0.00 2.00
(d) Clusters S2 and $3
(c) Clusters S2 and 84
(f) Clusters S3 and S4
Figure 6: Each matrix shows the Spearman's 's correlation coefficients extra-cluster syntactic analysis, one matrix for each pair of clusters, S; and Sz. Values closer to +1 are in red, and values closer -1 in blue. Statistically significant results with a p-value lower than 0.01 are laheled with
A.3
Correlation Results after Finetuning
0.13
10.21
0.23
0..12
0.07
0.26
0.07
0.10
0.22
0.24
0.11
0.37 0.25
0.23
0.04
0.15
0.37
0.31
0.14
0.07
0.08
0.19
0.12 0.38
0.31
0.21
0.12 0.10 0.14
0.11
0.05 0.04
.0.06 -0.07
40. -0. 4. 0.19 0.02 0.25 0.16 0.22
-0.08
-0.18
0A
00.02
0.07
-018
-014
0..04
0.06
0.02
0.14 0.14 0.00 0.20 -000
0.38
6
DO
-0.09
0.17 0.17 0.03
-0
13 -0
10
-0
22
-0
08
-0
09
-0.01
-0.22
-0.36
DO
-0.20
-0.01
-0.03 -0.07 -0.33 -0.32
-0.2
13
11 -0
-0.19
-021
2
5
10
11
7
8
9
10
11
(a) Spearman correlations pretrained models
(b) Spearman correlations finetuned models
Figure
As described in Section 5.3, the typological similarities among languages has a positive correlation with the similarity observed across different models, and this correlations are more clearly observed after finetuning. With morphological analysis, conversely, we cannot draw further insights into the similarities between the different models. On this smaller set of languages, positive statistically significant correlations can be observed either in pretrained models 7a after finetuning 7b.


Reviews:
[{"review_id": "vgweAyDkNV", "reviewer": "Reviewer_j8oq", "summary": "", "questions": "QuestionA: How much of an impact do you think using the monolingual models that were trained on a variety of different data sources makes an impact of the findings here? Should the models not be controlled for in any way? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Commenting on Reasons to Reject:__\n\n_Raised point: Discussion of linguistic features_\n\nIndeed! In Section 5.2., we partially addressed the issue, but we need to make it a little bit more explicit. Understanding which linguistic features shared between the languages have a larger impact on similarities between matrices is interesting. \n\nJust to clarify. In Section 5.2, we identified linguistic features that make languages more different, and we named them \"polarizing'' (meaning that they assume the same value within one cluster and the opposite value in the other). This is also why we later refer to them as important (L234), because they make clusters distinguishable.  Definitely, we need to add some words to state this point more clearly. Yet, the idea is that those matrices could be responsible for encoding information about the relative order of words since the polarizing features we detected (\"87A Order of Adjective and Noun'' and \"97A Relationship between the Order of Object and Verb and the Order of Adjective and Noun\") mainly refer to this kind of syntactical information.\n\nThis kind of information could provide insights, for example, into more effective multilingual training in languages that share the same word order. However, this analysis is meaningful only when large enough clusters are available, and hence, we commented only on these features for the larger pair of syntactic clusters.\n\n__Answering to Questions to the Authors:__\n\n_Question A: Variety of data sources_\n\nThe great variety of pretraining data can, in fact, negatively affect the analyzed correlations. We initially considered analyzing only models exposed to similar data sources. However, this would eventually lead to a smaller number of models and languages. Moreover, it could make the investigation less general.\nHence, on the one hand, we introduced a single, looser constraint and chose to include only BERT models pre-trained on a single language and with sufficiently large and documented datasets. On the other hand, we decided to have an intuition of what could happen when models are exposed to more similar sources (in Section 5.3), even if the setting can only give an intuition of what can happen when models are trained from scratch on more similar (or parallel) data sources. We will state this point more clearly!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Linguistically motivated analysis that considers similarities between languages and their monolingual model build; may help others in determining language combinations for multilingual models or for doing pretraining with multiple languages for one language \n- It is important to be able to analyze what kind of linguistic information gets encoded and used in model builds so the paper is a start for this understanding \n", "reasons_to_reject": "- It would have been interesting to see more detailed discussion of which linguistic features shared between the languages had more or less impact on the matrix activations as opposed to other dissimilar languages; that is, more of an ablation study on the different activations \n- Additional discussion on why certain features between which languages were activated may illustrate more how to use this findings in other experimental set ups - can/how this be extended to multilingual builds for example? ", "questions_to_authors": "QuestionA: How much of an impact do you think using the monolingual models that were trained on a variety of different data sources makes an impact of the findings here? Should the models not be controlled for in any way? ", "paper_topic_and_main_contributions": "This paper investigates how typological linguistic features for syntax and morphology are encoded in monolingual models, then compares different monolingual builds between typologically similar/dissimilar languages to gain insight into shared properties. The comparison is done on the weight matrices for model layers and results in showing similar activations between models with shared linguistic features. The paper contributes using bidimensional centered kernel alignment to compare activation matrices across both rows and columns. The paper overall contributes to methods of using linguistically informed decisions to make more motivated model build decisions. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L234: What defines the features as the 'most important' ones? "}, "extra_text": {}}, {"review_id": "VkJwhSnoRA", "reviewer": "Reviewer_WpkQ", "summary": "", "questions": "What are the significances of your research (opening a new way to investigate specific linguistic features in layers.)?\nHow did you extract or feed the typological features into the BERT model?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Commenting on Reasons to Reject and Answering to Questions to the Authors:__\n\n_Raised point: Significance of research and methodology_\n\nOpening a new way to investigate specific linguistic features in Transformers layers is exactly our purpose: we propose a new methodology to investigate specific linguistic features in layers based on comparing models of different languages to study whether linguistic similarities between languages are reflected in similarities among matrices in specific layers! Hence, we are not encoding syntax and morphology. We are detecting where it is encoded in a new way.\n\nAs required, we will better clarify at the end of Section 3.1 how we define language feature vectors and the similarity among them. Just a few missing lines: We produce the one-hot encoding of features described by linguists in WALS for each language and then compute the cosine similarity between them to get assessments of similarities.\n\nWith our new methodology, we discover that models' weight matrices carry important information about the languages they were trained on, as they encode syntactic and morphological features of languages in specific layers. \n\nThe core idea is that, if the same similarity that holds between languages (comparing typological features) also holds between models (comparing weight matrices), it means that the investigated BERT models encode typological features of languages. \n\nFor this purpose, we use biCKA to measure the similarity between weight matrices, and we use the cosine similarity between language feature vectors to measure typological similarity between languages. Then we correlate these two assessments of similarities with Spearman Correlation Coefficient. Hence, to define the ability of monolingual models to retain typological information, we do not need to extract typological features from weight matrices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors tried to investigate a new property of transformer-based models and the results confirm their claims. ", "reasons_to_reject": "Though the authors discovered a novel property of transformer models, I think the main limitation of this paper is they didn't discuss about how they represent the syntactic and morphological features of any languages as vectors (there is no explanation of lang2vec) or how a pretrained BERT for a language L, weight matrices Wi = {Qi,Ki, Vi,OAi,DIi,DOi} represent the typological features of a language or how they extract the syntactic and morphological features from a BERT model. According to my understanding, they just measured the similarities among weight matrices. ", "questions_to_authors": "What are the significances of your research (opening a new way to investigate specific linguistic features in layers.)?\nHow did you extract or feed the typological features into the BERT model?", "paper_topic_and_main_contributions": "This paper tried to investigate how the transformers based models represent the underlying structure of languages. The main intuition of this paper is: if different natural languages are syntactically or morphologically similar then their BERT should have similar weight matrices in some layers. According the experimental results, layers 4,5 and 6 encode syntactic information of languages. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "pife3j02Vt", "reviewer": "Reviewer_DrBj", "summary": "", "questions": "* Line 077: Can you give an example or explain more concisely what you mean by \"syntactic typological similarity\"? At this point in the paper its not at all clear what exactly you are measuring.\n* Line 144: You are saying \"[...] mBERT is not exploiting typological similarities among languages [...]\". Why would you assume that it would or should? I.e. is this enforced in the training process or the objective function?\n* Line 200: Not everybody is familiar how lang2vec works, if your work is a variation of lang2vec, it would be good to briefly explain how lang2vec works, if your approach is just similar, then it makes more sense to focus on how your approach works (my reading is that this is what is happening, however the sentence \"[...] we use a typological classification scheme that, in line with lang2vec [...]\" got me slightly confused since I am not familiar enough with lang2vec to immediately pick up what you mean).\n* Since you've completed some missing features in WALS - have you contributed this to the WALS project? If so, then this would be a very welcome (side-)contribution of the project worth mentioning.\n* Line 379: You mention that you use pre-trained models in the various languages from hugginface. These models will have been trained on a variety of different corpora. Do you expect any variation due the different training corpora used?\n* Line 604-608: Can you give more reasoning and insight why these findings are important (I don't mean to be cynical, I do find the results interesting _per se_, however I would also be interested in what way you would leverage these findings)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Answering to Questions to the Authors:__\n\n* Line 077: \nTypological classification of languages deals with similar structural features -- syntactic and morphological -- across different languages. For example, to construct a syntactic typological classification, one feature that is taken into consideration is the basic order of constituents -- subject (S), direct object (O) and verb (V) -- in a clause. In WALS, this feature is called \"81A - Order of Subject, Object and Verb\".\nFor example Japanese and Turkish generally share the same word order.\nA sentence like _Mary eats an apple_, \nin Japanese is translated as _Mary_ (S – ‘Mary’) _wa_ (particle) _ringo_ (O – ‘apple’) _wo_ (particle) _tabemasu_ (V – ‘eats’)\nand in Turkish is _Mary_ (S – 'Mary’) _elmayi_ (O – ‘the apple’) _yiyor_ (V – ‘eats’).\nThis is not the case in Italian (and in English), since the order is different _Mary_ (S – ‘Mary’) _mangia_ (V – ‘eats’) _la mela_ (O – ‘the apple’).\n\n\n* Line 144:\nWe stem from Singh et al.(2019) findings. In fact, since the representations produced by mBERT seem to be separated between different languages, this model does not exploit similarity among languages to reuse part of the representation it builds across different languages.\n\n* Line 200:\nOur approach is definitely similar to lang2vec, since it defines a list of features for each language and also contains some WALS features.\nIt also defines language vector as one-hot encoding of those features and defines their similarities via cosine similarity. \nHowever, lang2vec is not complete (morphological features are not defined). Hence, we started from WALS and we add some feature values that were missing.\n\n* We could have emphasized  more that our side contribution is an extension of WALS! We plan to contribute to the WALS project with the actual missing values that we provided.\n\n* Line 379:\nSelecting models from huggingface we observed that they were all pretrained on a large variety of sources and domains.\nHowever, dissimilar pretraining data could negatively affect the analyzed correlations.\nWe tried to understand what could happen when more similar sources are employed in Section 5.3. As future work, we would like to analyze on more similar models and expect to get higher similarities for more similar languages. In this phase, we chose to include as large number of languages as possible to make the findings more generalizable.\n\n* Line 604-608: \nThese findings can be important because they can help interpret the inner workings of these models to go toward the so-called actionable explainability - the idea of giving the possibility to act within the matrices changing some weights to obtain a desired behavior. Moreover, this methodology could help to build a more informed architecture that takes advantage of the most promising layers in multilingual models. We will add these considerations in the conclusions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This is a very well-written paper, posing a clear research question and addresses this question empirically.\n* Its great to see findings of earlier work by Tenney et al. (2019) confirmed by a different methodological approach.\n* The experimental setup is really cool - I especially liked the use of WALS to cluster languages in combination with the FOCUSED experiments.\n* A key contribution of the work is not just the findings on BERT, but the methodology as such which would be applicable to models beyond BERT (I believe this contribution could be claimed more prominently).", "reasons_to_reject": "* I don't see any reasons for rejecting this work. It poses a clear research hypothesis and contains a sound methodological approach to validate the hypothesis.", "questions_to_authors": "* Line 077: Can you give an example or explain more concisely what you mean by \"syntactic typological similarity\"? At this point in the paper its not at all clear what exactly you are measuring.\n* Line 144: You are saying \"[...] mBERT is not exploiting typological similarities among languages [...]\". Why would you assume that it would or should? I.e. is this enforced in the training process or the objective function?\n* Line 200: Not everybody is familiar how lang2vec works, if your work is a variation of lang2vec, it would be good to briefly explain how lang2vec works, if your approach is just similar, then it makes more sense to focus on how your approach works (my reading is that this is what is happening, however the sentence \"[...] we use a typological classification scheme that, in line with lang2vec [...]\" got me slightly confused since I am not familiar enough with lang2vec to immediately pick up what you mean).\n* Since you've completed some missing features in WALS - have you contributed this to the WALS project? If so, then this would be a very welcome (side-)contribution of the project worth mentioning.\n* Line 379: You mention that you use pre-trained models in the various languages from hugginface. These models will have been trained on a variety of different corpora. Do you expect any variation due the different training corpora used?\n* Line 604-608: Can you give more reasoning and insight why these findings are important (I don't mean to be cynical, I do find the results interesting _per se_, however I would also be interested in what way you would leverage these findings)?", "paper_topic_and_main_contributions": "The paper aims to validate where morphology and syntax are encoded in pre-trained monolingual BERT models. The authors leverage Centered Kernel Alignment to assess the similarity between weight matrices in BERT and conclude that indeed syntax is encoded in the middle layers of BERT and morphology is more akin to be encoded by the attention layer, though less strongly. The authors also convincingly demonstrate that syntax remains encoded in the middle layers across typologically different languages.\n\nThe main contributions of the paper are a methodology to assess where specific linguistic phenomena are encoded in pre-trained models and an experimental validation on where morphology and syntax are encoded in multiple (including on low-resource languages) pre-trained monolingual BERT models. As a consequence of their experiments, the authors arrive at a conclusion analogous to Tenney et al. (2019) on where BERT appears to encode syntactical information and thereby independently validate the earlier results with a slightly different methodological appraoch. A (side) contribution of the work is the addition of missing features in WALS which the authors leverage for their experiments. \n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* The first sentence in the abstract - \"[...] of transformers force NLP researchers [...]\" - is quite awkward, especially that we're _forced_ to delve into these models.\n* Line 030: Peters et al. (2018) did not use a pre-trained transformer (ELMo was a deep biLSTM).\n* Line 186: Missing \"representations\" in \"[...] their BERT _[representations]_ should have similar weight matrices [...]\""}}]

Author rebuttal:

Meta-review: The paper aims to validate where morphology and syntax are encoded in pre-trained monolingual BERT models. The authors leverage Centered Kernel Alignment to assess the similarity between weight matrices in BERT and conclude that indeed syntax is encoded in the middle layers of BERT and morphology is more akin to be encoded by the attention layer, though less strongly. The authors also convincingly demonstrate that syntax remains encoded in the middle layers across typologically different languages.

The main contributions of the paper are a methodology to assess where specific linguistic phenomena are encoded in pre-trained models and an experimental validation on where morphology and syntax are encoded in multiple (including on low-resource languages) pre-trained monolingual BERT models. As a consequence of their experiments, the authors arrive at a conclusion analogous to Tenney et al. (2019) on where BERT appears to encode syntactical information and thereby independently validate the earlier results with a slightly different methodological appraoch. A (side) contribution of the work is the addition of missing features in WALS which the authors leverage for their experiments.

All in all the paper is in a good shape. Some additional discussion and anaylsis may be added but this is not a major issue.
Decision: Accept-Findings