A Joint Matrix Factorization Analysis of Multilingual Representations
Authors: Zheng Zhao
Yftah Ziser
Bonnie L. Webber
Shay B Cohen
Venue: EMNLP 2023 Findings (2023)
Keywords: Representation analysis
Multilingual pre-trained models
Matrix factorization
Morphosyntactic features
TL;DR: We present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models.

Abstract: We present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models. An alternative to probing, this tool allows us to analyze multiple sets of representations in a joint manner. Using this tool, we study to what extent and how morphosyntactic features are reflected in the representations learned by multilingual pre-trained models. We conduct a large-scale empirical study of over 33 languages and 17 morphosyntactic categories. Our findings demonstrate variations in the encoding of morphosyntactic information across upper and lower layers, with category-specific differences influenced by language properties. Hierarchical clustering of the factorization outputs yields a tree structure that is related to phylogenetic trees manually crafted by linguists. Moreover, we find the factorization outputs exhibit strong associations with performance observed across different cross-lingual tasks. 
We release our code to facilitate future research.

A Joint Matrix Factorization Analysis of Multilingual Representations
Zheng Zhao
Yftah Ziser Bonnie Webber Shay B. Cohen
Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB {zheng. zhao, yftah. ziser, bonnie. webber y@ed ac uk, scohen@inf. ed. ac
Abstract
We present an analysis tool based on joint ma- trix factorization comparing latent represen- tations of multilingual and monolingual models. An alternative to probing, this tool allows us analyze multiple
of representations in a joint manner. Using this tool, we study to what extent and how morphosyntactic features reflected in the representations learned by multilingual pre-trained models. We conduct a large-scale empirical study of over 33 lan- guages and 17 morphosyntactic categories. Our findings demonstrate variations in the encoding of morphosyntactic information across upper and lower layers, with category-specifie differ- ences influenced by language properties. Hier- archical clustering of the factorization outputs yields tree structure that is related to phyloge- netic trees manually crafted by linguists. More- over, we find the factorization outputs exhibit strong associations with performance observed across different cross-lingual tasks. We release our code to facilitate future research.
A
Ug
Σg
V
Figure 1: A diagram of the matrix factorization that PARAFAC2 performs. For our analysis, A ranges over covariance matrices between multilingual model repre- sentations and éth monolingual model representations.
acquire and incorporate linguistic information in their multilingual representations.
Previous work has used singular vector canonical correlation analysis (SVCCA; Raghu et al. 2017) and other similarity statistics like centered kernel alignment (CKA; Kornblith al., 2019) to analyze multilingual representations (Singh al., 2019; Kudugunta et al., 2019; Muller al., 2021). How- ever, such methods can only compare one pair representation sets at time. In contrast to that, we analyze all multilingual representations simultane- ously using parallel factor analysis (PARAFAC2; Harshman 1972b), method that allows to fac- torize set of representations jointly by decompos- ing it into multiple components that can be ana- lyzed individually and then recombined under- stand the underlying structure and patterns the data. More precisely, we extend the sub- aub-population analysis method recently presented by Zhao et al. (2022), who compare two models as an alternative to probing: control model trained on data of inter- est and experimental model, which is identical to the control model but is trained additional data from different sources. By treating the multi- lingual experimental model shared component in multiple comparisons with different control mod- els (each one is a monolingual model trained on
1 Introduction
Pre-trained multilingual models (Conneau and Lample, 2019a; Conneau et al., 2020; Liu al., 2020; Xue et al., 2021) have gained widespread adoption recent years. They initially pre-trained in many languages and subsequently fine-tuned for specific downstream tasks. Their aim is to leverage the linguistic knowledge acquired from similar lan- guages, thereby benefiting low-resource languages and enabling zero cross -ingull transfer ability. While numerous prior works have demonstrated these models have such abilities (Gerz al., 2018; Ziser and Reichart, 2018; Aharoni al., 2019; K et al., 2020; Muller et al., 2021; Fujinuma et al., 2022: Qiu et al., 2023), there are still open ques- tions about the nature of the linguistic knowledge these models possess and the extent to which they
hhttps://gitee htt.://..commmuuuuuee////iiiioo www.llinnuua_aaalliiii...

---

subset of the multilingual model), we can better analyze the multilingual representations.
of matrices representing cross-covariance be- tween multilingual and monolingual representa- tions, PARAFAC2 allows us to compare the repre- sentations across languages and their relationship to a multilingual model. For an integer n, we use [n] to denote the set {1...., n.. For square matrix Σ, we denote by diag( () its diagonal vector.
As an alternative to probing, our representation analysis approach: a) enables standardized com- parisons across languages within a multilingual model, cireumventing the need for external perfor- mance upper bounds in making meaningful inter- pretation of performance metrics; b) directly an- alyzes model representations, avoiding the need for auxiliary probing models and potential biases from specific probing classifier architectures; and c) compares multilingual versus monolingual rep- resentations for any inputs, avoiding reliance on labelled probing datasets.
PARAFAC2 Let & index set of matrices," ? such that A₂ |XX, the matrix of cross- covariance between X, and Z. which are random vectors of dimensions d and d', respectively. This means that [Arli = Cov( Z2) for E [d] and j E [d'].
For any and two vectors, a C IRd b Rd", the following holds due to the lincarity of expectation: a Cov(a Xx, b bT Z). PARAFAC2 on the set of matrices {Ae} in this case finds set of transformations (Ut) V and set of diagonal matrices [Ee]e such that:
We use PARAFAC2 to directly compare repre- sentations learned between multilingual models and their monolingual counterparts. We apply this efficient paradigm answer the following research questions on multilingual models: Q1) How do multilingual language models encode morphosyn- tactic features in their layers? Q2) Are our findings robust to address low-resource settings? Q3) Do morphosyntactic typology and downstream task performance reflect in the factorization outputs? We experiment with two kinds of models, XLM- R (Conneau et al., 2020) and RoBERTa (Liu et al., 2019). We apply our analysis tool on the multilin- gual and monolingual representations from these models to check morphosyntacti information in 33 languages from Universal Dependencies tree- banks (UD; Nivre et al.. 2017a). Our analysis rein- forces recent findings on multilingual representa- tions, such as the presence of language-neutral sub- spaces in multilingual language models (Foroutan et al., 2022), and yields the following key insights:
A U.V.V.
(I)
We call the elements on the diagonal of Eg pseudo values, in relationship to singu- lar value decomposition that decomposes a single matrix in similar manner. The decomposition in Eq. 1 jointly decomposes the matrices such that each A₃ is decomposed into sequence of three transformations: first transforming Z into latent space (V), scaling (Eş) and then transforming it into specific /th-indexed X, space (Uy). Unlike singular value decomposition, which decomposes a matrix into a similar sequence of transformations with orthonormal matrices, PARAFAC2 does not guarantee U, and V to be orthonormal and hence they do not represent an orthonormal basis transfor- mation. However, Harshman (1972b) showed that solution can still be found and unique if add the constraint that U, Ue is constant for all In our use of PARAFAC2, we follow this variant. We provide an illustration of PARAFAC2 in Figure Representation of low-resource languages bene- fits from the presence of related languages.
Encoding of morphosyntactic information is in- fluenced by language-specificc factors such as writing system and number of unique characters.
Multilingual representations demonstrate dis- tinct encoding patterns in subsets of languages with low language proximity.
Experiment-Control Modeling for Multilingual Analysis
Our factorization method's utility reflects in hier- archical clustering within phylogenetic trees and prediction of cross-lingual task performance.
We employ factor analysis to generate a distinctive signature for a group of representations within an
"In our case, will usually index a language. We will have several collections such matrices indexed by language. for example, slicing the representations from specific mor- phosyntactic category. This becomes clear $3 and $4. We assume that for any f, E[X,) = 0 and B|Z] O.
2 Background and Motivation
this paper, we propose to use PARAFAC2 for multilingual analysis. By jointly decomposing a

---

experimental model, contrasting them with repre- sentations derived from set of control models. our case, the experimental model is a jointly- trained multilingual pre-trained language model, and the control models are monolingual models trained separately for set of languages. Formally, there is an index set of languages [L] and set of models consisting of the experimental model E and the control models C2 for & [T.].
The diagonal of Σg can now be readily used to describe a signature of the éth language rep- resentations in relation to the multilingual model (see also Dubossarsky et al. 2020). This signa- ture, which we mark by sig(f) = diag(E;)) can be used to compare the nature of representations between languages, and their commonalities in re- lationship to the multilingual model. In our case, this PARAFAC2 analysis is applied to different slices of the data. We collect tokens in different languages (both through multilingual model and monolingual models) and then slice them by spe- cific morphosyntactic category, each time applying PARAFAC2 on subset of them.
We assume a set of inputs we apply our analysis to, X' - Xz. Each set Xe = [xEII..... )(,)) represents a set of inputs for the éth language. While we assume, for simplicity, that all sets have the same size m, it does not have to be the case. In our case, each X3 is set of input words from language l, which results in a set of representa- tions as follows. For each E [L] and i E [m]] we apply the model E and the model C₁ to Xg, to get two corresponding representations Yt,; E Rd and ze,; E Rdd. Here, d is the dimension of the multilingual model and de is the dimension of the representation of the monolingual model for the 4th language. Stacking up these sets of vectors separately into two matrices (per language E), we obtain the set of paired matrices Y, R and Ze IR mxd, We further calculate the covariance matrix S2g defined as: S2 Z Ye.
For some of our analysis, we also use con- densed value derived from sig(E). We follow similar averaging approach to that used by SVCCA (Raghu et al., 2017), a popular representation analy- sis tool, where they argue that the single condensed SVCCA score represents the average correlation across aligned directions and serves as a direct mul- tidimensional analogue of Pearson correlation. In our case, each signature value within sig({) from the PARAFAC2 algorithm corresponds to a direc- tion, all of which are normalized in length, the signature values reflect their relative strength. Thus, taking the average of sig(() provides an intensity measure of the representation of a given language in the multilingual model. We provide additional discussion in $5.1.
Use of PARAFAC2 Given an integer & smaller or equal to the dimensions of the covariance matrices,
we apply PARAFAC2 on the set of joint matrices, decomposing each S2 into:
4 Experimental Setup
22 ~ V.E.V
(2) Data We use CoNLL's 2017 Wikipedia dump (Ginter et al., 2017) to train our models. Follow- ing Fujinuma et al. (2022), we downsample all Wikipedia datasets to an identical number of se- quences to use the same amount of pre-training data for cach language. In total, we experiment with 33 languages. For morphosyntactic features, we use treebanks from UD 2.1 (Nivre et al., 2017a). These treebanks contain sentences annotated with morphosyntactic information and are available for a wide range of languages. We obtain a representa- tion for every word in the treebanks using our pre- trained models. We provide further details on our pre-training data and how we process morphosyn- tactic features in Appendix B.
where U E Re and V E dd
To provide some intuition on this decomposition, consider Eq. 2 for a fixed f. If we were following SVD, such decomposition would give two projec- tions that project the multilingual representations and the monolingual representations into joint latent space (by applying Ug and V on zs and ys, respectively). When applying PARAFAC2 jointly on the set of J. matrices, we enforce the matrix V to be identical for all decompositions (rather than be separately defined if we were applying SVD on each matrix separately) and for U, to vary for each language. We are now approximating the Zi matrix, which by itself could be thought as trans- forming vectors from the multilingual space to the monolingual space (and vice versa) in three trans- formation steps: first into a latent space (V), scal- ing it (Er), and then specializing it monolingually.
Task For pre-training our models, we use masked language modeling (MLM). Following Devlin et al. (2019), we mask 15% of the tokens. To fully con- trol our experiments, we follow Zhao et al. (2022)

---

and train our models from scratch.
0.5
Models We have two kinds of models: the mul- tilingual model E, trained using all L languages available, and the monolingual model C₁ for c [4] trained only using the 9th language. We use the XLM-R (Conneau et al., 2020) architecture for the multilingual E model, and we use RoBERTa (Liu et al., 2019) for the monolingual C₁ model. We use the base variant for both kinds of models. We use XLM-R's s vocabulary and the SentencePiece (Kudo and Richardson, 2018) tokenizer for all our exper- iments provided by Conneau et al. (2020). This enables us to support all languages we analyze and ensure fair comparison for all configurations. We provide additional details about our models and training in Appendix B.2.
N00 gnature
02-
16 12
Lxyer
(a) Original data
(b) zh and ja romanized data
Figure 2: Average signature violin plots for all layers and languages on (a) original data and (b) data with Chinese (zh) and Japanese (ja) romanized.
condense each signature vector into single value by taking the average of the signature. This value encapsulates the intensity of the use directions in V. A high average indicates the corresponding language is well-represented in the multilingual model. We expect these values to exhibit general trend (either decreasing increasing) going from lowers to upper layers as lower layers more general and upper layers are known to be more task-specific (Rogers al., 2020). In addition, the trend may be contrasting for different languages and morphosyntactic features.
5
Experiments and Results
This section outlines our research questions (RQs), experimental design, and obtained results.
5.1
Morphosyntactic and Language Properties
Here, we address RQI: How do multilingual lan- guage models encode morphosyntactic features in their layers? While in broad strokes, previous work (Hewitt and Manning, 2019; Jawahar et al., 2019; Tenney et al., 2019) showed that syntactic infor- mation tends to be captured in lower to middle layers within a network,
Language Signatures Across Layers We begin by presenting the distribution of average sig(E) val- ues for all languages across all layers for all lexical tokens in Figure 2a. We observe gradual decrease in the mean of the distribution as we transition from lower upper layers. This finding is consistent with those from Singh et al. (2019), who found that the similarity between representations of dif- ferent languages steadily decreases up to the final layer in pre-trained mBERT model. We used the Mann-Kendall (MK) statistical test (Mann. 1945; Kendall, 1948) for individual languages across all layers. The MK test is rank-based non- -on-maraettiic method used assess whether set of data values increasing or decreasing over time, with the null hypothesis being there no clear trend. Since we perform multiple tests (33 tests in total for all lan- guages), we also control the false discovery rate (FDR; at level q = 0.05) with corrections to the p- values (Benjamini and Hochberg, 1995). We found that all 33 languages except for Arabic, Indonesian, Japanese, Korean, and Swedish exhibit significant monotonically decreasing trends from lower layers to upper layers, with the FDR-adjusee p-values (p < 0.05). Figure 2a shows that the spread of the
ask more refined question here, and inspect whether different layers are specialized for specific morphosyntactic fea- tures, rather than providing an overall picture of all morphosyntax in a single layer. As mentioned in $3, we have a set of signatures, sig(() for é E [L]. each describing the ?th language representation for the corresponding morphosyntactic category we probe and the extent which it utilizes infor- mation from each direction within the rows of V. PARAFAC2 identifies a single transformation V that maps a multilingual representation into a latent space. Following that, the signature vector scales in specific directions based on their importance for the final monolingual representation it is trans- formed to. Therefore, the signature can be used to analyze whether similar directions in V are im- portant for the transformation to the monolingual space. By using signatures of different layers in a joint factorization, we can identify comparable similarities for all languages. Analogous to the SVCCA similarity score (Raghu et al., 2017), we

---

distribution for each layer (measured in variance) is constantly decreasing up until layer 6. From these layers forward, the spread increases again. small spread indicates that the average intensity of scaling from a multilingual representation to the monolingual representation is similar among all languages. This provides evidence of the multi- lingual model aligning languages into a language- neutral subspace in the middle layers, with the up- per layers becoming more lusk-focueed (Merchant et al., 2020). This result is also supported by find- ings of Muller et al. (2021)- - different languages representations' similarity in mBERT constantly increases up to mid-layer then decreases.
-071
Compriees-tt
Deferiooos
90
Mesd-Ete
Numbo-Ett
Doves-LE
-0.00
Poscoies-
Figure 3: Pearson correlation results between the aver- age sig(E) for all languages and their data size for cach morphosyntactic category among all layers.
Logogram vs. Phonogram In Figure 2a, we ob- serve long bottom tail in the average sig(E) plots for all languages, with Chinese and Japanese show- ing lower values compared to other languages that are clustered together, suggesting that our models have learned distinct representations for those two languages. We investigated if this relates to the logographic writing systems of these languages, which rely on symbols represent words or mor- phemes rather than phonetic elements. We con- ducted an ablation study where we romanized our Chinese and Japanese data into Pinyin and Ro- maji,* 4 respectively, and retrained our models. One might ask why we did not normalize the other lan- guages in our experiment to use the Latin alphabet. There are two reasons for this: 1) the multilingual model appears to learn them well, as evidenced by their similar signature values to other languages; 2) our primary focus on investigating the im- pact of logographic writing systems, with Chinese and Japanese being the only languages employing logograms, while the others use phonograms. Fig- ure 2b shows that, apart from the embedding layer, the average sig(f) are more closely clustered to- gether after the ablation. Our findings suggest that logographic writing systems may present unique challenges for multilingual models, warranting fur- ther research to understand their computational pro- cesses. Although not further explored here, writing systems should be considered when developing and analyzing multilingual models.
while most attributes exhibit a similar decreasing trend from lower to upper layers, some attributes, such as Comparison and Polarity, show consistent distributions across all layers. Since these attributes
occur rarely in our data (< 1% of tokens), it is pos- sible that the model is only able to learn general representation and not distinguish them among the layers. To investigate the effect of attribute fre- quency on our analysis, we performed Pearson correlation analysis (for each attribute) between the average sig(E) for all languages and their data size- the number of data points available in the UD anno- tations for a particular language and morphosyntac- tic feature. The results are shown in Figure 3. Our analysis of the overall dataset (all words) shows no evidence of correlation between attribute frequency and average sig(E). However, upon examining in- dividual categories, we observe a decrease in corre- lation as we move up the layers, indicating that the degree a morphosyntactic attribute is represented in the multilingual model is no longer associated with simple features like frequency but rather with some language- spercfffc properties. This observa- tion holds true for all categories, with the excep- tion of Animacy, which is predominantly found in Slavic languages within our dataset. This aligns with the findings of Stanezak et al. (2022), who noted that the correlation analysis results can be af- fected by whether a category is typical for a specific genus. Next, we further explore the relationship between signature values and language properties.
Morphosyntactic Attributes Looking at indi- vidual morphosyntactic attributes, we observe that
'We use libraries available at: https:://pppp /ttp:.//pppp./////////p./ project/pypinyin/ and https://pypi..rr www.orrg//ooooooo.oo properjject/ pykakasi/. We use the lazy_pinyin feature to generate Pinyins without tone marks.
Language Properties In addition to data size, we explore the potential relationship between

---

Unique Choractues
-0.20
0..00
4.7k
-0.19
0.13
+0,47
4,00
-0.22
4057
-0.40
-0.38
0:45
0.12
-0.40
40,27
-0.4
-0.65
Typeso- Railo
40500
-0.35
4.29
0.3
-0.04
0.37
-0.26
0.08
-025
-0.26
0.10
0.85
-0.01
-0.20
0.00
0.25
-0:55
-015
-0.00
-0.1
0.11
-0.30
-0.14
40.18
0.39
Animacy
Appeco
Compaiiien
Deflatimeat
Flatlones
Genda
Moed
Nambler
Prnvs
Pakasty
Pultemeet
Powetssiin
Tame
Valmayy
Vokce
All Wonds
0.5
Figure Pearson correlation results between the average sig(f) for all languages and the number of unique characters, type-token ratio (TTR), and data size for each morphosyntactic category, averaged across all layers.
Lingugge-ppeciti properties and the average sig(E). We consider two language properties: the num- ber of unique characters and the type-token ratio (TTR), a commonly used linguistic metric to assess a text's vocabulary diversity. TTR is calculated by dividing the number of unique words (measured in lemmas) by the total number of words (measured in tokens) obtained from the UD annotation meta- data. Typically, a higher TTR indicates greater degree of lexical variation. We present the Pearson correlation, averaged across layers, in Figure 4, To provide a comprehensive comparison, we in- clude the results for data size as well. The detailed results for each layer can be found in Appendix C. Examining the overall dataset, observe strong negative correlation between the number of unique characters and signature values. Similarly, the TTR exhibits similar negative correlation, indicating that higher language variation corresponds to lower signature values. When analyzing individual cate- gories, we consistently find a negative correlation for both the number of unique characters and the TTR. This further supports our earlier finding that Chinese and Japanese have lower signature values compared to other languages, they possess higher number of unique characters and TTR.
using a public XLM-R checkpoint." 6 The setup remained the same, except that we used represen- tations obtained from this public XLM-R instead of our own trained XLM-R. We observe that the trends for signature values were generally similar, except for the embedding and final layers, where the values were very low. This expected, the cross cova-covv was calculated with our mono- lingual models. The similar trend among the mid- dle layers further supports the idea that these lay- ers learn language- and data-agnostic representa- tions. Furthermore, the Pearson correlations be- tween the number of unique characters, TTR, data size, and the average sig(E) for the overall dataset were follows: -0.65, -0.28, and -0.02, respec- tively. These values are nearly identical to those shown in Figure 4, confirming the robustness of our method and its data- agnostic nature.
5.2 Language Proximity and Low-resource Conditions
Here, we address RQ2: Are our findings robust to address language subsets and low-resource set- tings? In RQ1, our analysis was based on the full set of pre-training languages available for each morphosyntactic category we examine. In this question, we aim to explore subsets of representa- tions derived from either related diverse set of pre-training languages, and whether such choices yield any alterations the findings established in RQI. Furthermore, we extend our analysis fow-resource settings and explore potential changes in results on low-resuurce languages, particularly when these languages could receive support from other languages within the same language family. We also explore the potential benefits of employing language sampling techniques for enhancing the representation of low-resource languages.
Generalization to Fully Pre-trained Models To ensure equal data representation for all languages in our experiment--controlle modeling, we down- sampled the Wikipedia dataset and used an equal amount for pre-training our multilingual models. To check whether our findings are also valid for multilingual pre-trained models trained on full- scale data, we conducted additional experiments
"To ensure accurate analysis, we filter out noise by count- ing the number of characters that account for 99.9% of oc- currences in the training data. This climinates characters that only appear very few times.
"nttps://huggingfaace co/xlm-roberta.

---

(a) Layer
Layer
(c) Layer
Figure 5: Cosine distance matrices between all language pairs and their signature vectors based on overall representations obtained from layer 0, and 12. Darker color indicates the cosine distance being close to 1.
Language Proximity We obtain the related set of languages by adding all languages that are from the same linguistic family and genus (full infor- mation available in Appendix A). In total, ob- tained three related sets of languages: Germanic languages, Romance languages, and Slavic lan- guages. There are other related sets, but we do not include them in our experiment since the size of those sets is very small. For the diverse set of languages, we follow Fujinuma et al. (2022) and choose ten languages from different language genera that have diverse set of scripts These languages are Arabic, Chinese, English, Finnish, Greek, Hindi, Indonesian, Russian, Spanish, and Turkish. We use the >²-quur variance test to check whether the variance of the diverse set's 's av- erage signatures from a particular layer is statisti- cally significant from the variance of that of the related set, given morphosyntactic category. We test layers 0 (the embedding layer), 6, and 12, cov- ering the lower, middle, and upper layers within the model. We first find that for the overall dataset, the variance of the diverse set average signatures is significantly different (at a = 0.05) from all three related set variances for all three layers. This suggests that, in general, multilingual representa- tions are encoded differently for different subsets of languages with low language proximity, For the attributes of number, person, and tense, the vari- ance within the diverse set significantly differs from the variances within the three related sets across all three layers, with statistical significance level a 0.05. This finding sensible as all these three attributes have distinctions in the diverse set of languages. For example. Arabic has dual nouns to denote the special case of two persons, animals,
or things, and Russian has a special plural form of nouns if they occur after numerals. On the other hand, for attributes like gender, we do not witness significant difference between the diverse set and related set since there only four possible values (mascul ine, feminine, neuter, and common) in the UD annotation for gender. We speculate that this low number of values leads to low variation among languages, thus the non-significant differ- ence. This finding concurs with Stanczak et al. (2022), who observed a negative correlation be- tween the number of values per morphosyntactic category and the proportion of language pairs with significant neuron overlap. Hence, the lack of sig- nificant differences variance between the diverse and related sets can be attributed to the substantial overlap of neurons across language pairs.
Low-resource Scenario In order to simulate a low-resource scenario, we curtailed the training data for selected languages, reducing it to only 10% of its original size. The choice of low-resource lan- guages included English, French, Korean, Turkish, and Vietnamese. English and French were selected due to the availability of other languages within the same language family, while the remaining lan- guages were chosen for their absence of such famil- ial relationships. Notably, Korean was specifically selected as it utilizes a distinct script known as Hangul. To examine the impact of low-resource conditions on each of the selected languages, we re-trained our multilingual model, with each indi- vidual language designated low-resource. To address potential confounding factors, we also re- trained monolingual models on the reduced dataset. Additionally, we explored a sampling technique (Devlin, 2019) to enhance low-resource languages.

---

Dataset
PLangg
Metric
MBERT
XLM
XLM-R
MMTE
XNIJ (Conncau et al., 2018)
12
Acc.
.36
.30
.36
.21
Classification
PAWS-X X (Yang et al., 2019)
7
Acc.
.67
.65
.75
.69
POS (Nivre el al., 2017b)
22
Fl
.36
.36
.66
.40
Struct. pred.
NER (Pan et al., 2017)
22
F1
.46
.46
.55
.46
XQuAD (Attetxe et al., 2020)
10
FI ET/EM
.607.35
81 .56
.73 .45
.727 .61
QA
MLQA (Lewis al., 2020)
F1 / EM
.23 /31
.46 .48
.64 6.8
.28 /-
TyDiQA-GoldP (Clark al., 2020)
F1 /EM
.41/ .00
.43 .43
.46 /
.66 .45
BUCC (Zweigenbaum al., 2017)
FI
.72
.83
Retrieval
Tatoeba (Artetxe and Schwenk, 2019)
Acc.
.15
.28
Table 1: Pearson correlations between final layer's (i and XTREME benchmark performances various tasks.
Further details can be found Appendix D.
metic mean (UPGMA; Sokal and Michener, 1958) to construct a phylogenetic tree. We show the dis- tance matrices between all language pairs and their signature vectors based on overall representations obtained from layers O, 6 and 12 in Figure 5, We can observe that signatures for Arabic, Chinese, Hindi, Japanese, and Korean are always far with respect those for other languages across layers. From the distance matrix, we construct a phyloge- netic tree using the UPGMA cluster algorithm. We present our generated trees and a discussion in Ap- pendix E. In short, the constructed phylogenetic tree resembles finguistically-crffii trees.
Our analysis reveals the impact of low-resource conditions on signature values. English and French, benefiting from languages within the same lan- guage family, exhibit minimal changes in signa- ture values, indicating& a mitigation of the effects of low-resource conditions on language represen- tation. Remaining languages without such sup- port experience a significant decline in signature values (dropping from 0.3 to nearly O), particu- larly after the embedding layer. This implies that low-resource languages struggle to maintain robust representations without assistance from related lan- guages. Additionally, our findings suggest that lan- guage sampling techniques offer limited improve- ment signature values of low-resource languages.
Performance Prediction To establish a robust connection between our factorization outputs and downstream task performances, we conducted an analysis using the XTREME benchmark, which includes several models: mBERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019b), XLM- R. and MMTE (Arivazhagan al., 2019). This benchmark encompasses nine tasks that span four diffcrent categories: classification, structured pre- diction, question answering, and retrieval. These tasks demand reasoning on multiple levels of mean- ing. To evaluate the relationship between the met- rics of each task and our average sig(E) across all available languages for that task, we calculated the Pearson correlation. For each task's performance metrics, we use the results reported by Hu et al. (2020). The obtained correlation values using sig- nature values from the last layer are presented Table 1, along with pertinent details about each task, such as the number of available languages, and the metrics employed. For a comprehensive analysis, also provide results using zig()) from every layer in Appendix E.2. Observing the results, becomes evident that the XLM-R model exhibits the highest correlation, which is expected since the sig(E) values obtained from our factorization pro- cess are also computed using the same architecture.
5.3 Utility of Our Method
Here we address RQ3: Do morphosyntactic typol- ogy and downstream lask performance reflect in the factorization outputs? Having conducted quantita- tive analyses ses of our proposed analysis tool thus far, our focus now shifts to exploring the tool's ability to unveil morphosyntactic information within multi- lingual representations and establish a relationship between the factorization outputs and downstream task performance. To investigate these aspects, we conduct two additional experiments utilizing the signature vectors obtained from our analysis tool. Firstly, we construct a phylogenetic tree using co- sine distance matrices of all signature vectors. Sub- sequently, we examine the correlations between the results of the XTREME benchmark (Hu et al., 2020) and the sig(() values.
Phylogenetic Tree We first compute cosine dis- tance matrices using all signature vectors for all 33 languages and 12 layers for each morphosyntactic attribute. Then, from the distance matrix, we use an agglomerative (bottom-up) hicrarchical clustering method: unweighted pair group method with arith-

---

Furthermore, for most tasks, the highest correla- tion is observed with the final layers, which is rea- sonable considering their proximity to the output. Notably, we consistently observe high correlation across all layers for straightforward tasks like POS and PAWS-X operating on the representation level. However, for complex reasoning tasks like XNLI, only the final layer achieves reasonable correlation. These results suggest that the factorization outputs can serve as valuable indicator of performance for downstream tasks, even without the need for fine-tuning or the availability of sseccipci data.
covered that typological properties are persistently encoded across layers in mBERT and XLM-R. Liang et al. (2021) demonstrated with probing that language-specific information is scattered across many dimensions, which can be projected into a linear subspace. Intrinsic probing, on the other hand, explores the internal structure of linguistic information within representations (Torroba Henni- gen al., 2020). Stanczak et al. (2022) conducted a large-scale empirical study over two multilingual pre-trained models, mBERT, and XLMM-,, and in- vestigated whether morphosyntactic information is encoded in the same subset of neurons in different languages. Their findings reveal that there is con- siderable cross-lingual overlap between neurons, but the magnitude varies among categories and is dependent language proximity and pre-training data size. Other methods, such as matrix factoriza- tion techniques, are available for analyzing repre- sentations (Raghu et al., 2017; Morcos et al., 2018; Korbblith et al., 2019) and even modifying them through model editing (Olfat and Aswani, 2019; Shao et al., 2023a; Kleindessner et al., 2023; Shao et al., 2023b). When applied to multilingual analy- sis, these methods are limited to pairwise language comparisons, whereas our proposed method en- ables joint factorization of multiple representations, making well-suited for multilingual analysis.
6 Related Work
Understanding the information within NLP mod- els' internal representations has drawn increasing attention in the community. Singh et al. (2019) ap- plied canonical correlation analysis (CCA) on the internal representations of a pre-trained mBERT and revealed that the model partitions representa- tions for each language rather than using a shared interlingual space. Kudugunta et al. (2019) used SVCCA to investigate massively multilingual Neu- ral Machine Translation (NMT) representations and found that different language encoder represen- tations group together based on linguistic similarity. Libovický et al. (2019) showed that mBERT rep- resentations could be split into a language-specific component and a language-neutral component by centering mBERT representations and using the centered representation on several probing tasks to evaluate the language neutrality of the repre- sentations. Similarly, Foroutan et al. (2022) em- ployed the lottery ticket hypothesis discover sub- networks within mBERT and found that mBERT is comprised of language- neutral and language- specific components, with the former having a greater impact on cross-lingual transfer perfor- mance. Muller et al. (2021) presented a novel layer ablation approach and demonstrated that mBER could be viewed the stacking of two sub-networks: multilingual encoder followed by a spec-fi agnostic predictor.
7 Conclusions
We introduce a representation analysis tool based on joint matrix factorization. We conduct a large- scale empirical study over 33 languages and 17 morphosyntactic categories and apply our tool to compare the latent representations learned by mul- tilingual and monolingual models from the study. Our findings show variations in the encoding of morphosyntactic information across different lay- ers of multilingual models. Language-specific dif- ferences contribute these variations, influenced by factors such as writing systems and linguistic relatedness. Furthermore, the factorization out- puts exhibit strong correlations with cross-lingual task performance and produce phylogenetic tree structure resembling those constructed by linguists. These findings contribute to our understanding of language representation in multilingual models and have practical implications for improving perfor- mance in cross-lingual tasks. In future work, we would like to extend our analysis tool to examine representations learned by multimodal models.
Probing (see Belinkov 2022 for review) is a widely-used method for analyzing multilingual rep- resentations and quantifying the information en- coded by training a parameterized model, but its effectiveness can be influenced by model parame- ters and evaluation metrics (Pimentel al., 2020). Choenni and Shutova (2020) probed representa- tions from multilingual sentence encoders and dis-

---

Limitations
Linguistics, pages 4623-4631 Online, Association for Computational Linguistics.
Our research has several limitations. First, we only used RoBERTa and its muitilingual variant, XLM- R, for our experiments. While these models are widely used in NLP research, there are other op- tions available such as BERT, mBERT, T5, and mT5, which we have yet to explore due to a limited budget of computational resources. Second, en- sure equal data representation for all languages we experimented with, we downsampled Wikipedia resulting in a corpus of around 200MB per lan- guage. While we validated our findings against a publicly available XLM-R checkpoint trained on much larger resource, further verification is still necessary. Third, our analyses are limited mor- phosyntactic features, and in the future, we aim to expand our scope to include other linguistic aspects, such as semantics and pragmatics.
Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguisties. 7:597-610.
Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguis- rics, 48(1:2277-
Yoav Benjamini and Yosef Hochberg. 1995, Control- ling the false discovery rate: practical and pow- erful approach multiple testing. Journal of the Royal statistical society: series (Methodological), 57(1):289-300.
Rochelle Choenni and Ekaterina Shutova. 2020. What does it mean to be inguage-agnostic probing multi- lingual sentence encoders for typological properties. CoRR, Abs/2009.12865
Acknowledgments
Jonathan H. Clark, Eunsol Choi, Michael Collins. Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information seeking question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 3:454-470.
This work was supported by the UKRI Centre for Doctoral Training (CDT) in Natural Language Pro- cessing through UKRI grant EP/S022481/1 and CDT funding from Huawei Technologies. We would like to thank Balint Gyevnar, Matthias Lin- demann, Edoardo Ponti, Ivan Titov and the anony- mous reviewers for their helpful feedback. We ap- preciate the use of computing resources through the CSD3 cluster at the University of Cambridge and the Baskerville cluster at the University of Birm- ingham.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke 7ettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- poss-linggul representation learning at scale. In Pro- ceedings of the 58th Annuul Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Alexis Conneau and Guillaume Lample. 2019a. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems 32: An- nual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057-7067
References
Roce Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 3874-3884, Minncapolis, Minnesota. Association for Computa- tional Linguistics.
Alexis Conneau and Guillaume Lample. 2019b. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems 32: An- nual Conference Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver. BC, Canada, pages 7057-7067.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges. ArXiv preprint, abs/1907.05019.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- wral Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational
Jacob Devlin. 2019. Multilingual BERT README.

---

Jacob Devlin, Ming-Wor Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generaliza- tion. ArXiv preprini, abs/2003. 1080.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah, 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 3651-3657 Florence, Italy. Association for Computational Linguistics.
Haim Dubossarsky, Ivan Vulié, Roi Reichart, and Anna Korhonen. 2020. The secret is in the spectra: Pre- dicting cross-lingual task performance with spectral similarity measures. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2377-2390, Online. As- sociation for Computational Linguistics.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-linuual ability of multilin- gual BERT: an empirical study. In 8th International Conference on Learning Represenations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Negar Foroutam, Mohammadreza Banaci. Rémi Lebret, Antoine Bosselut, and Karl Aberer. 2022. Discov- ering language-neutral sub-networks in multilingual language models. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 7560-7575, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Maurice George Kendall. 1948, Rank correlation meth- ods. Griffin.
Christo Kirov, Ryan Cotterell, John Sylak-Glassman Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya McCarthy. Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. UniMorph 2.0: Universal Mor- phology. In Proceedings of the Eleventh Interna- rional Conference on Language Resources and Eval- uation (LREC 2018), Miyazaki, Japan, European Lan- guage Resources Association (ELRA).
Yoshinari Fujinuma, Jordan Boyd-Graber, and Katha- rina Kann. 2022. Match the script, adapt if multilin- gual: Analyzing the effect of multilingual pretraining on cross- cross-lingual transferability. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1500-1512, Dublin, Ireland. Association for Compu- tational Linguistics.
Matthäus Kleindessner. Michele Donini, Chris Russell, and Muhammad Bilal Zafar. 2023. Efficient fair pea for fair representation learning. In Proceedings of The 26th International Conference telligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 5250-5270. PMLR.
Artificial In-
Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the rela- tion between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing, pages 316-327, Brussels, Belgium. Association for Computational Linguistics.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey F. Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learn- ing. ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA. volume 97 of Proceedings of Machine Learning Research, pages 3519-35222 -3299. PMLR.
Filip Ginter, Jan Hajië, Juhani Luotolahti, Milan Straka, and Daniel Zeman. 2017. CoNLL 2017 shared task automatically annotated raw texts and word embed- digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles Univer- sity.
dings. LINDAT/CLARIAH-
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. 2019. Tensorly: Tensor learning in python. Journal of Machine Learning Research, 20(26):1-6.
R. A. Harshman. 1972b. PARAFAC2: Mathematical and technical notes. UCLA Working Papers in Pho- netics, 22:30-44.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019.
structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 1129-4138 Minneapolis, Minnesota. Association for Computational Linguistics.
Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual NMT representations at scale. In Proceedings of the

---

2019 Conference on Empirieal Methods Natu- val Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1565-1575, Hong Kong, China. Association for Computational Linguistics.
Conference on Neural Information Processing Sys- tems 2018, NeurlPS 2018, December 3-8, 2018, Mon- tréal, Canada, pages 5732-5741.
Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Un- derstanding the cross-lingual ability of multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguisties: Main Volume, pages 22 2214-2231, Online, Association for Computational Linguistics.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evalu- ating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Asso- ciution for Computational Linguistics, pages 7315- 7330, Online. Association for Computational Lin- guistics.
Joakim Nivre, Daniel Zeman, Filip Ginter, and Francis Tyers. 20 Universal Dependencies. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Tutorial Abstracts, Valencia, Spain. Association for Computational Linguisties.
Sheng Liang, Philipp Dufter, and Hinrich Schüttze 2021, Locating anguage-specifie information in contextu- alized embeddings. CoRR, 0tt/2209908040
Jindrich Libovicky, Rudolf Rosa, and Alexander Fraser. 2019. How language-neutral is multilingual bert? ArXiv preprint, abs/ 1911199113333 1.03310.
Joakim Nivre Daniel Zeman, Filip Ginter, and Francis Tyers. 201 Universal Dependencies. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Tutorial Abstracts, Valencia, Spain. Association for Computational Linguistics.
Chunxi Liu, Qiaochu Zhang, Xiaohui Zhang, Kritika Singh. Yatharth Saraf, and Geoffrey Zweig. 2020. Multilingual graphemic hybrid ASR with massive data augmentation. In Proceedings of the 1st Joint Workshop Spoken Language Technologies for Under-resourced languages (SLTU) and Collabora- tion and Computing for Under-Resourced Languages (CCURL), pages 46-52, Marseille, France. European Language Resources association.
Matt Olfat and Anil Aswani. 2019. Convex formula- tions for fair principal component analysis. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 33, pages 663-670.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. ArXiv preprint, htts/1997.1166
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth- man, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics ( (Volume I: Long Papers), pages 1946-1958, Vancouver, Canada. As- sociation for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison. Andreas Köpf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning li- brary. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 8024 8035.
Henry B Mann. 1945. Nonparametric tests against trend. Econometrica: Journal of the econometric society, pages 245-259.
Arya D. McCarthy, Miikka Silfverberg, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2018 Marrying Universal Dependencies and Universal Morphology. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 91-101, Brussels, Belgium. Association for Computational Linguistics.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to BERT embed- dings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and In- terpreting Neural Networks for NLP, pages 33-44, Online. Association for Computational Linguistics.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay. Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4609-4622, Online. Association for Computa- tional Linguistics.
Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights representational similarity in neural net- works with canonical correlation. In Advances in Neural Information Processing Systems 31: Annual
Yifu Qiu, Yftah Ziser, Anna Korhonen, Edoardo M. Ponti, and Shay B. Cohen. 2023. Detecting and miti- gating hallucinations in multilingual summarisation.

---

Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Dick-Dickstein.. 2017. SVCCA: singular vec- tor canonical correlation analysis for deep learning dynamies and interpretabillty. In Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6076-6085.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush, 2020. Trans- formers: State-of -hee-----hhheeeeee art natural language processing. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Association for Computational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866 -666.
I. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguisties.
Shun Shao, Yftah Ziser, and Shay B. Cohen. 2023a. Grasure of Unaligned Atrrbutes from Neural Rep- resentations. Transactions of the Association for Computational Linguistics, 11:488-510.
Shun Shao, Yitah Ziser, and Shay B. Cohen. 2023b. Gold doesn't always glitter: Spectral removal of lin- ear and nonlinear guarded attribute information. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 1611-1622, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
Yinfei Yang. Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: cross-lingual adversar- ial dataset for paraphrase identification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-UJCNLP), pages 3687-3692, Hong Kong, China. Association for Computational Linguis- tics.
Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is not interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (Deeplo 2019), pages 47-55, Hong Kong. China. Association for Computational Linguistics.
Zheng Zhao, Yftah Ziser, and Shay Cohen. 2022. Un- derstanding domain learning in language models through subpopulation analysis. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 192- 209, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguisties.
Robert R. Sokal and Charles Duncan Michener. 1958. A statistical method for evaluating systematic rela- tionships. University of Kansas science bulletin. 38: 143881138.
Yftah Ziser and Roi Reichart. 2018. Deep baset modeling for cross-language cross-domain transfer with minimal guidance. In Proceedings of the 2018 Conference Empirical Methods in Natural Lan- guage Processing, pages 238-249, Brussels, Bel- gium. Association for Computational Linguistics.
Karolina Stanczak, Edoardo Ponti, Lucas Torroba Hen- nigen, Ryan Cotterell, and Isabelle Augenstein. 2022. Same neurons, different languages: Probing mor- phosyntax in multilingual pre-trained models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1589-1598, 1598, Seattle, United States, Association for Computational Linguistics.
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2017. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60 67, Vancou- ver, Canada. Association for Computational Linguis- Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy, Association for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NIP pipeline. In
tics.
Lucas Torroba Hennigen, Adina Williams, and Ryan Cotterell. 2020. Intrinsic probing through dimension selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 197-216, Online. Association for Computational Linguistics.

---

A Information on Attributes and Languages
Finnish (fi): Case, Comparison, Finiteness. Mood, Number, Part of Speech, Person, Polar- ity, Possession, Tense, Valency, Voice
We first provide information about all languages use our experiment in Table The information includes ISO 639-1 codes for all languages, the language family and the genus they belong to. In Table 3, we present all morphosyntactic attributes we experiment. For each attribute, we list all lan- guages that have the attribute. We also provide reverse list where we list by languages:
French (frr): Aspect, Definiteness, Finiteness, Gender, Mood, Number, Part of Speech, Per- son, Polarity, Tense, Valency
Galician (gl): Part of Speech, Polarity
German (de): Case, Comparison, Definite- ness, Finiteness, Mood, Number, Part of Speech, Person, Polarity, Politeness, Posses- sion, Tense, Valency
Arabic (ar): Aspect, Case, Definiteness, Finiteness, Gender, Mood, Number, Part of Speech, Person, Polarity, Politeness, Voice
Greek (el): Aspect, Case, Comparison, Defi- niteness, Finiteness, Gender, Mood, Number, Part of Speech, Person, Tense, Voice
Bulgarian (bg): Aspect, Case, Comparison, Definiteness, Gender, Mood, Number, Part of Speech, Person, Polarity, Tense, Valency, Voice
Hebrew (he): Case, Definiteness, Finiteness, Number, Part of Speech, Person, Polarity, Pos- session, Tense, Valency, Voice
Catalan (ca): Aspect, Case, Definiteness, Finiteness, Gender, Mood, Number, Part of Speech, Person, Polarity, Possession, Tense
Hindi (hi): Aspect, Case, Finiteness, Gen- der, Mood, Number, Part Speech, Person, Polarity, Politeness, Tense, Voice
Chinese (zh): Aspect, Case, Number, Part of Speech, Person, Polarity, Valency, Voice
Hungarian (hu): Case, Comparison, Defi- niteness, Finiteness, Mood, Number, Part of Speech, Person, Possession, Tense, Valency
Croatian (hr): Animacy, Case, Comparison, Definiteness, Finiteness, Gender, Mood, Num- ber, Part of Speech, Person, Polarity, Posses- sion, Tense, Valency, Voice
Indonesian (id): Part of Speech, Polarity
Czech (es): Animacy, Aspect, Case, Com- parison, Finiteness, Gender, Mood, Number, Part Speech, Person, Polarity, Possession, Tense, Valency, Voice
Italian (it): Aspect, Comparison, Definite- ness, Finiteness, Gender, Mood, Number, Part of Speech, Person, Tense
Japanese (ja): Part of Speech
Danish (da): Case, Comparison, Definiteness, Finiteness, Gender, Mood, Number, Part of Speech, Person, Possession, Tense, Valency, Voice
Korean (ko): Part of Speech
Polish (pl): Animacy, Aspect, Case, Com- parison, Finiteness, Gender, Mood, Number, Part of Speech, Person, Polarity, Possession, Tense, Valency, Voice
Dutch (nl): Case, Comparison, Definiteness, Finiteness, Gender, Number, Part of Speech, Person, Tense, Valency
Portuguese (pt): Aspect, Case, Definiteness. Finiteness, Gender, Mood, Number, Part Speech, Person, Polarity, Tense
English (en): Case, Comparison, Definite- ness, Finiteness, Gender, Mood, Number, Part of Speech, Person, Tense, Valency
Romanian (ro): Aspect, Case, Definiteness. Finiteness, Gender, Mood, Number, Part of Speech, Person, Polarity, Possession, Tense, Valency
Estonian (et): Aspect, Case, Comparison, Finiteness, Mood, Number, Part of Speech, Person, Polarity, Tense, Valency, Voice

---

Russian (ru): Animacy, Aspect, Case, Com- parison, Finiteness, Gender, Mood, Number, Part of Speech, Person, Polarity, Tense, Va- lency, Voice
sequences (about 512 tokens per sequence) to mea- sure model loss (cross-entropy) during pre-training, For morphosyntactic features, we use treebanks from UD 2.1 (Nivre et al., 2017a). These treebanks contain sentences annotated with morphosyntactic information and are available for wide range of languages. We obtain contextual representation for every word in the treebanks by feeding them to mutillingua/mmoooiiguua models. We then use the UniMorph schema (Kirov al., 2018) map each word with its parts of speech and mor- phosyntactic properties. We provide a list of mor- phosyntactic categories we use in Appendix A. We follow Stanczak et al. (2022) and use the converter (McCarthy et al., 2018) 18) to switch morphosyntactic annotations from UD v2.1 to UniMorph schema.
Slovak (sk): Animacy, Aspect, Case, Com- parison, Finiteness, Gender, Mood, Number, Part of Speech, Person, Polarity, Possession, Tense, Valency, Voice
Slovenian (sl): Animacy, Aspect, Case, Com- parison, Definiteness, Finiteness, Gender, Mood, Number, Part of Speech, Person, Polar- ity, Possession, Tense, Valency
Spanish (es): Aspect, Case, Comparison, Def- initeness, Finiteness, Gender, Mood, Number, Part of Speech, Person, Polarity, Tense, Va- lency
B.2 Details for Models
We use the XLM-R (Conneau al., 2020) archi- tecture for the multilingual model, and we use RoBERTa (Liu et al., 2019) 9) for the monolingual C, model. We use the base variant for both kinds models, which consists of 12 layers, 768 hidden di- mensions, 8 attention heads for RoBERTa, and 12 attention heads for XLM-R. We use XLM-R's vo- cabulary and the SentencePiece (Kudo and Richard- son, 2018) 8) tokenizer for all our experiments pro- vided by Conneau al. (2020) order to support all languages we analyze and enable fair compari- son for all configurations. We do not use the origi- nal RoBERTa vocabulary and tokenizer since they only support English. We pre-train all models for a maximum of 150K steps, and all models use the val- idation set cross- cross-entropy loss perform early stop- ping. We use the AdamW optimizer (Loshchilov and Hutter, 2019) with learning rate of 10-4. Our monolingual models were trained on four NVIDIA Gelorce GTX 1080 Ti GPUs with a batch size two per GPU, and our multilingual models were trained on four NVIDIA A100 GPUs with a batch size of 16 per GPU. Both models take about two days train. We use PyTorch (Paszke et al., 2019). the HuggingFace library (Wolf al., 2020) and the TensorLy library (Kossaifi et al., 2019) for all model implementation and PARAFAC2 computa- tion.
Swedish (sv): Case, Comparison, Definite- ness, Finiteness, Gender, Mood, Number, Part of Speech, Polarity, Tense, Voice
Turkish (tr): Aspect, Case, Mood, Number, Part of Speech, Person, Polarity, Politeness, Possession, Tense, Valency, Voice
Ukrainian (uk): Animacy, Aspect, Case, Comparison, Finiteness, Gender, Mood, Num- ber, Part of Speech, Person, Polarity, Tense, Valency, Voice
Vietnamese (vi): Part of Speech, Polarity
Notice that in this list and in work, omit language that has less than 100 instances labeled for particular morphosyntactic category.
B Additional Details for Experiments
B.1 Details for Data
We use CoNLL's 2017 Wikipedia dump (Ginter al., 2017) train our models. Following Fu- jinuma et al. (2022), we downsample Wikipedia datasets to an identical number of sequences in order to use the same amount of data for all lan- guage pre-training. The downsampled dataset is standardized to the Hindi corpus, which has the smallest size among all languages examine. For each language's pre-training data, there are about 30M tokens (approximately 200MB). In total, experiment with 33 languages. We provide the full list of languages used for our experiments in Ap- pendix A. We also create a validation set with 1K
C Additional Results for RQ1
We present the Pearson correlations between the average sig(E) for all languages and their number of unique characters for each morphosyntactic cat-

---

egory among all layers in Figure 6. The results for type-token ratio (TTR) presented in Figure 7. Please note that in the figures, the last row is la- beled as "All Words", representing the results ob- tained from using representations taken from the dataset with Part-of-Speech (PoS) attributes. Given that each lexical token in the dataset is associated with a PoS tag. this analysis encompasses the entire dataset, enabling us observe and comprehend the overall trends captured in the representations.
D Additional Results for RQ2
Adhering to the current standard practice of lan- guage sampling during pre-training of multilingual models, we also experimented a setting inspired by the approach described by Devlin (2019). Fol- lowing their approach, we applied sampling tech- nique to boost the representation of lower-resource languages. This involved sampling examples based on the probability P(L) x |L| where P(L) rep- resents the probability of selecting text from a given language during pre-training, and [L] denotes the number of examples available in that language. For our study, we set the value of a to 0.3.
Asiinay-- 4.25 4:37
4.15 4:15 4:1 4:15 4012 4.20 4.0 4.7
Aspes
-0.5
Clos 6/78
695
695
698 (3) 1 1M
Campaiiss 6:04 8:08
8.11
1:11 1:11 1:22
416
4.11
Delikkase-
0.0
E Additional Results for RQ3
Fisieosss
6:6 4.0 651 3/2
3.0
Mogbosytaehi Dingee
Gesder
4.12
0:1
0.13
486
4:18
4-12
4.15
4.37
-110
4114
4.20
0115
We include in this section a phylogenetic tree anal- ysis based on our approach, and a performance prediction experiment.
Nonter
6:18 696
015
1114
Pases
4029 403
8:5
Polusty
Pollewes
609
699
(07)
0079
6.11
812
812
022
E.1 Phylogenetic Tree
0A22 026 028 0.18
8.8
8.84
-0.0
4.36
4.35
We show the tree generated from layer 6's matrix is provided in Figure 8a.
Vileegg-a111 4222 6:14 421 6.27 402
111 1111 8.
Vikee-ire 4v12 492 4:1 4x77 4:77
41
4th
4475
There is ongoing discussion over the specifics of the linguistic evolutionary phylogenetic tree of lan- guages, and a tree model has limitations because not all evolutionary connections are fully hierar- chical, and it is difficult to account for horizontal transmissions (Singh et al., 2019). Despite this, still see that the constructed phylogenetic tree closely matches the language tree that linguists created to describe the links and development of human languages. We can see that generally, Ger- manic, Romance, and Slavic languages are clus- tered in different sub-trees. In particular, West Slavic languages, South Slavic languages, and East Slavic languages are generally clustered together before being combined into the common Slavic language family. Also, Fastern Romance language Romanian are merged together with Western Ro- mance languages to form the Romance language family cluster. Similar to the findings of Singh et al. (2019), we also observe that trees generated across different layers are generally similar. They may have different structures as the branching of the tree may differ, but languages within the same family genus are also close in the tree.
Wonde-at5
Layar
Figure 6: Pearson correlation results between the aver- age égg for all languages and their number of unique characters for each morphosyntactic category among all layers.
Atimay
Asgeet
0.05 4:35
4:23
431
A02 4/3 0355
4.40 4.44 4.4) 4:02
-08
Cumpation -948
042
0:61
0.5
DeSistesess.. -9.7 088 4.84
4:91 A0D 0/3 AZ1
4.85 4.55 411
Kochhopptoth Congry
Goodo- 0.84 6:14
6:27 6:4 6.1 4.3 AM 431 8:11 8:91
04
Mood 9.17 045
4.80 686 402 8.0 002 402 0.37 0.33 0:35 0.17
0.1
Nandor
-03
PcDAm 322 4:37
4:34 422 4.38 A31 4.2 4.7 4.53 4.35 4.5 4.11 0.2
Folarty
Pilicnoo
3.09 6.71 655 087 0.80 3 897 0.11 0.85
6.6
Pasooalonn 3.61 6.6 666 6.66 6/6 0/8 0/ 0.6 8:81 8: 678 8.06 400
-02
Valmay
11 166 8.9 661 6:0 0/0 035 015 685 641 8: 4m
Voloz 9.18 0:33 023 022 0.25 0300 0.51 025 022 022
0.2
10
31
13
Lascr
So far, we have constructed trees based on the full slice of the data using representations from the PoS attribute. We also tried to generate trees us- ing all other morphosyntactic attributes. However, since for most morphosyntactic attributes, some
Figure 7: Pearson correlation results between the aver- age sig(E) for all languages and their type-token ratio (TTR) for each morphosyntactic category among all lay- ers.

---

Arabic
Arabic
Aaaanee
Bulgarian
Careon
Creatian
Bulgarian
Estonian
Czach
Finnish
Polish
Slovek
Ukrainian
Croatan
Slovek
Slavenlan
Dannish
Russian
Swedlsh
Ukrainian
Hebrow
Turkish Finnish
Indonesian
Hungarian
Vietnemese
Germain
Swedish
ettaloo 6m
Geman
Partuguesa
Englih
Spanish
Hungarfan
Romanian
Catalan
Dutch English
French
Hinndl
Galician
Chinese
Portuguese
Italian Romanian
Indonesian
Vietnamesse
Japanese
Hindi
Konkan
(a)
(b)
Figure 8: Phylogenetic trees of languages based the distance between signatures vectors for all languages: (a) using overall representations; (b) averaged over morphosyntactic categories.
languages are always missing, i.e., the attribute is not available in that language, we construct an "av- erage" tree by using the average distance matrices from all morphosyntactic categories except for PoS. If a language is missing in a category, we assign a distance of to all other languages. We provide this average tree in Figure 8b. In this average tree, we observe better fit, for example, Arabic and Hebrew now in the same branch, and Chinese and Japanese are their own branch as they belong to language families that are distinct from all other languages.
E.2 Performance Prediction
We present correlations for all layers for mBERT, XLM, XLM-R, and MMTE in Table 4, Table 5, Table 6, and Table 7.

---

Language
ISO 639-1
Family
Genus
Arabic
Afro-Asiatic
Semitic
Bulgarian
bg
Mudo-uur
Slavic
Catalan
ca Indo-European
Romance
Chinese
zh Sino-Tibetan
Chinese
Croatian hr Indo-European
Slavic
Czech
cs Indo-European
Slavic
Danish da
Euro-eur
Germanic
Dutch nl Indo-European
Germanic
English
Indo-European
Germanic
Estonian et Uralic
Finnic
Finnish fi Uralic
Finnic
French fr Indo-European
Romance
Galician gl Indo-Euroon European
Romance
German de Indo-European
Germanic
Greek el Indo-European
Greek
Hebrew he Afro-Asiatic
Semitic
Hindi hi Indo-Europen
Indie
Hungarian hu Uralic
Ugric
Indonesian
id Austronesian
Malayo-Sum
Sumbawan
Italian it Indo-European
Romance
Japanese ja Japanese
Japanese
Korean
ko Korean
Korean
Polish
pl Indo-European
Slavic
Romanian pt Europenn
Portuguese
Romance
Indo-European
Romance
Russian
Indo-European
Slavic
Slovak
sk Indo-European
Slavic
Slovenian
sl
Indo- Europen
Slavic
Spanish
es Indo-European
Romance
Swedish
sv Indo-Eure -uropen
Germanic
Turkish
tr Altaic
Turkic
Ukrainian
uk
Slavic
Vietnamese vi Austro-Asiatic
Vietic
Table 2: All languages used in this work along with language family and genus they belong Family and genus information is obtained from WALS.

---

Property
Language
Animacy
bg. hr, es, pl, ru, sk, sl, uk.
Aspect
ar, bg, ca, zh, hr, es, ct, fr, cl, hi, hu, it, pl, pt. ro, ru, sk, sl, es, tr, uk
Case
ar, bg. ca, zh, hr, cs, da, nl, en, et, fi, fr, de, el, he, hi, hu, pl. pt, ro, ru, sk, sl, es, sv, uk
Comparison
bg. hr, cs, da, ul, en, et, fr, de, cl. hu, it. pl. ro, ru, sk, sl, es, sv, uk
Definiteness
ar, bg. ca, hi, da, ul, en, fr. de, cl, he, hu, it. pt, ro, sl, es, sv
Finiteness
ar, ca, hr, cs, da, nl, en, et, fi, fr, de, el, he, hi, hu, it, pl. ar, bg, ca, hr, es, da, nl, en, fr,, cl, hi, it, pl, pt. ro, ru, sk, sl, es, sv, uk
ro, ru, sk, sl, es, sv, uk
Gender
Mood
ar, bg. ca, hr, cs, da, en, ct, fi, fr. de, el, he, hi. hu, it. pl, pt, ro, ru, sk, sl, es, tr, uk
Number
ar, hg, ca, zh, hr, es, da, nl, en, et, fi, fi, de, cl, he, hi, hu, it, pl, pt, ro, ru, sk, sl, es, sv, tr, uk
Part of Speech
all 33 languages
Person
ar, bg, ca, zh, hr, cs, da, nl, en, et, fi, fi,, de, el, he, hi, hu, it, pl, pt, ro, ru, sk, sl, es, tr, uk ar, bg. ca, zh, hr, et, fi, fr, gl, de, he, hi, id, it, pl, pt, ru, sk, sl, es, sv, tr uk,
Polarity
Politeness
ca, da, de, hi, es, tr
Possession
an, ca, hr, da, fi, de, he, hu, pl. ro, sk, sl, bg. ca, hr, cs, da, nl, et, fi. fr. de, el, he, hi, hu, it. pl. ro, sk. sl, es, tr, uk bg. zh. hr, da, nl, et, fi, fr. de, he, hu, pl, ro, sk, sl, es, uk
Tense
Valency
Voice
bg. zh, hr, da, et, fi, el. he, hi, pl, sk, sv, uk
Table 3: All language properties (morphosyntactic categories) we analyzed in this work along with languages that have the corresponding property.
Pair PAWS-X
Structured prediction
Question answering
Sentence retricval
Layer
XNLI
POS
NER
XQuAD
MILAA
TyDiQA-GoldP
BUCC
Tatocha
Acc.
Acc.
FI
FI
FI/EM
FI/EM
FI/EM
FI
Acc.
0.01
0.71
0.26
0.47
0.3370.15
0.037 0.15
0.6170.20
0,64
-0.11
-0.03
0.72
0.23
0.43
-0.00
-0.18 -0.11
0.20/ -0.28
0.70
0.05
0.01
0.76
0.32
0.49
0.31 /0.09
-0.15 -0.04
0.68
0.05
0.18
0.71
0.33
0.49
0.41 /0.13
-0.01 / 0.04
0.16/ -0.26
0,66
0.13
4
0.10
0.71
0.38
0.55
0.4
-0.01 /0.10
0.41/
0.65
0.12
0.16
0.69
0.33
0.51
0.29
0.06 0.15
-0.03
0.59
0.11
0.24
0.74
0.41
0.59
0.54
0.12/
0.54/0.03
0.66
0.18
0.16
0.70
0.29
0.41
0.43/0.21
0.02
0.35/-0.07
0.69
0.10
0.09
0.56
0.14
0.20
0.32/0.12
-0.11/-0.04
0.13/-0.18
0.64
0.03
0.06
0.54
0.11
0.16
0.30/0.11
-0.16/-0.07
0.12/-0.16
0.72
-0.01
0.15
0.55
0.15
0.23
0.37
-0.02/0.04
0.14/-0.13
0.71
0.01
0.18
0.58
0.24
0.42/0.19
0.02/ 0.09
0.17/-0.13
0.67
0.03
0.36
0.67
0.36
0.46
0.60 0.35
0.23/
0.41 0.05
0.72
0.15
Table 4: Pearson correlations between sig(!) and XTREME benchmark performances of mBERT various tasks.
Pair sentence
Structured prediction
Question answering
Sentence retrieval
Layer
XNLI
PAWS-X
POS
NER
XQuAD
MLQA
TyDiQA-GoldP
BUCC
Tatoeba
Acc.
Acc.
FI
FI
FI/EM
FI/EM
FI/EM
Acc.
-0.01
0.68
0.28
0.53
0.61/0.38
0.24/0.29
0.90
-0.02
-0.1
0.71
0.25
0.52
0.51 0.222
0.03/0.06
0.59 0.553
0.96
0.13
-0.07
0.74
0.34
0.57
0.54/0.27
0.05/
0.65 /..6
0.97
0.12
0.09
0.70
0.37
0.55
0.65 0.336
0.23/0.22
0.42/0.36
0.98
0.21
0.05
0.70
0.39
0.59
0.68 0.440
/
0.58 00.44
0.94
0.20
0.09
0.67
0.34
0.54
0.42
0.27
0.50/
0.93
0.18
6
0.17
0.72
0.41
0.63
0..4
0.32
0.67 /0.65
0.94
0.27
0.08
0.68
0.29
0.46
0.69/0.45
0,26/0228
0.53/0.51
0.95
0.19
-0.02
0.54
0.16
0.24
0.53
0.12
0.29/0.26
0.97
0.09
-0.44
0.51
0.12
0.20
0.53
0.09/
0.25/
0.92
0.06
10
0.07
0.52
0.17
0.26
0.61 0..38
0.23
0.298
0.92
0.08
11 0.10
0.56
0.25
0.34
0.40
0.25
0.33 / 0.31
0.97
0.11
12
0.30
0.65
0.36
0.46
0.81 0.56
0.46 0.48
0.43 / 00.43
0.96
0.24
Table 5: Pearson correlations between sig(E) and XTREME benchmark performances of XLMM on various tasks.

---

Pair sentence
Structured prediction
Question answenng
Sentence retrieval
Layer
XNLI
PAWS-X
POS
NER
XQUAD
MLQA
TyDiQA GoldP
BUCC
Tatoeba
Acc.
Acc.
FI
FI
FI/EM
EM
FI/EM
Acc.
0.11
0.80
0.67
0.62
0.6370.34
0.59 0.59/0.66 70.62
0.58
0.91
0.14
-0.33
0.80
0.61
0.57
0.110.14
0.39/0.44 /.440
0.61 /0.47
0.84
0.21
0.02
0.83
0.67
0.64
0.58/0.24
0.56
0.63 /0.47
0.85
0.23
0.13
0.78
0.66
0.62
0.55
0.43/0.45
0.37/0.16
0.85
0.19
0.15
0.80
0.73
0.68
0.71/0.39
0.56/0.61
0.59/0.46
0.90
0.31
0.17
0.78
0.69
0.63
0,70/0.40
/..63
0.51/0.43
0.93
0.28
0.24
0.82
0.73
0.70
0.73/0.42
0.63/0.64
0.68 /0.62
0.89
0.35
0.15
0.78
0.62
0.53
/.22
0.52
0.53/0.46
0.87
0.25
0.04
0.66
0.47
0.33
0.36/0.05
0.32/
0.30/0.23
0.88
0.13
0.13
0.63
0.44
0.30
/0.03
/.39
0.26/0.19
0.85
0.10
10
0.13
0.64
0.49
0.36
/0.10
0
0.29/
0.86
0.11
11
0.19
0.67
0.56
0.43
0.22
/
0,36
0.85
0.15
12
0,36
0.75
0.66
0.55
0.73/ 0.45
0.64 0.68
0.46 0.46
0.83
0.28
Table Pearson correlations between sig(() and XTREME benchmark performances XLM-R on various tasks.
Pair sentence
Structured prediction
Question answering
Sentence retrieval
Layer
XNLI
PAWS-X
POS
NER
XQuAD
MLQA
TyDiQA-Golds
BUCC
Tatocha
Acc.
Acc.
F1
F1
F1/EM
FI/EM
FT/
F1
Acc.
0.02
0.73
0.33
0.51
0.51/
0.067-
0.92/0.35
0.59
-0.10
0.75
0.26
0.46
0.43/0.37
-0.15/-
0.82/0.00
0.61
-0.06
0.78
0.38
0.53
0.48
-0.14/-
0.93/-0.04
0.58
0.02
0.73
0.36
0.51
0.44
0.07
0.79/-0.32
0.55
0.01
0.74
0.44
0.58
0.59
0.00/-
0.91 /0.06
0.58
0.08
0.71
0.39
0.54
0.54
0.07
0.85 0..2
0.53
0.15
0.76
0.44
0.61
0.69
0.13/-
0.91
0.58
0.07
0.73
0.35
0.59
0.09/-
0.78
-0.22
0.59
0.21
0.46/0.44
0.00/-
/
0.55
-0.44
0.56
0.19
0.44/0.44
-0.04/
0.54
0.66
0.05
0.58
0.20
0.54
0.10/-
0.55
0.64
0.08
0.31
0.59
0.11/-
0.54
0.57
0.21
0.69
0.40
0.72/0.61
0-28/-
0.66/0.45
0.63
Table Pearson correlations between sig(E) and XTREME benchmark performances of MMTE various tasks.


Reviews:
[{"review_id": "OwyWC8Pr3k", "reviewer": "Reviewer_85Fw", "summary": "", "questions": " Could you provide more details on the sampling technique explored in this paper to enhance low-resource languages? How does it work and what were the results of your experiments with it?\n\n How do you plan to extend the analysis tool based on joint matrix factorization developed in this paper to consider other linguistic features such as semantics or pragmatics?\n\n How do you envision the findings of this paper being applied in practical NLP applications, particularly for low-resource languages? \n\n Could you discuss any potential limitations or biases in the data used in this study, and how you addressed them in your analysis? \n\n How do you plan to make the software and pre-trained models developed in this paper publicly available, and what steps have you taken to ensure their usability and accessibility for researchers with varying levels of expertise?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback and recognition of our contributions. \n\n### Reasons To Reject\n*__\"The analysis is limited to morphosyntactic features and does not consider other linguistic features such as semantics or pragmatics.\"__*\n\nOver many years, linguists have worked with native speakers (both locally and via field work around the world) to come up with accurate characterizations of the world's languages. This work always starts by characterizing the morpho-syntactic properties of a language, as this is what is most evident in what they elicit from their native informants. Morphosyntactic attributes are important in NLP as they provide a deeper understanding of a language and what is encoded in words, including syntactic properties such as grammatical function and semantic properties such as agency, aspect, specificity, etc. It thus makes sense for an exploratory study of PARAFAC to use morphosyntactic features for three reasons: Such features are (i) semi-categorical, (ii) highly useful in many multilingual applications, such as grammatical error identification [1], PoS tagging [2], and dependency parsing [3], and (iii) available for more of the world's languages than any other properties. \n\n*__\"The paper's focus on multilingual models may limit its generalizability to monolingual models or models trained on specific tasks\"__*\n\nOur proposed analysis tool is to be used for analyzing multilingual models where an efficient comparison is being made between multilingual models and their monolingual counterparts. Our analysis tool is not limited to pre-trained models and can be easily applied to models trained on specific tasks. \n\n*__\"The paper's reliance on pre-trained models may limit its applicability to researchers who are developing their own models from scratch\"__*\n\nIn our work, we actually train our models from scratch (Section 4), and we also show that our findings are robust with public pre-trained checkpoints (Section 5.1). \n\n### Questions For The Authors\n*__Q: Could you provide more details on the sampling technique explored in this paper to enhance low-resource languages? How does it work and what were the results of your experiments with it?__*\n\nWe will provide a more detailed account of sampling techniques in Appendix D. To briefly elaborate, when constructing the multilingual dataset, we sample each language’s data using a probability that is exponentially smoothed and proportional to their availability in the dataset. Thus low-resource languages will be over-sampled, and high-resource languages will be under-sampled. The results under low-resource settings are provided in Section 5.2. The signature values for English and French are very similar to the original setting, whereas for Korean, Turkish, and Vietnamese, the signature values dropped to nearly 0 after the embedding layer.   \n\n*__Q: How do you plan to extend the analysis tool based on joint matrix factorization developed in this paper to consider other linguistic features such as semantics or pragmatics?__*\n\nOur analysis tool seamlessly accommodates the extension to other linguistic features. No modification of the analysis tool is necessary; as long as we possess the requisite data, we can readily input them into our analysis tool to derive corresponding factorization outcomes, such as signature values. As discussed above, one bottleneck of extending this work to other linguistic features, such as semantics and pragmatics, is the availability of data (especially availability for non-English languages). To start with, we can first extend to a smaller number of languages and analyze common semantic information like colour terms and kinship terms (which are commonly collected). \n\n*__Q: How do you envision the findings of this paper being applied in practical NLP applications, particularly for low-resource languages?__*\n\nThe findings from our study in low-resource contexts propose a practical approach for enhancing low-resource languages within NLP applications. By incorporating linguistically related languages during pretraining, we can bridge data gaps and enrich the model's understanding of underrepresented languages. This strategy holds promise in various NLP tasks, like machine translation and sentiment analysis, by leveraging shared linguistic traits. Ultimately, this approach offers a pragmatic solution to empower low-resource languages and improve multilingual NLP outcomes. Furthermore, our study's insights can be harnessed to optimize the allocation of resources in NLP projects targeting low-resource languages. By identifying languages with linguistic affinities that offer the most substantial improvements in representation, researchers and practitioners can make informed decisions about where to focus efforts and resources, leading to more efficient and effective NLP solutions for languages with limited data.\n\n*__Q: Could you discuss any potential limitations or biases in the data used in this study, and how you addressed them in your analysis?__*\n\nAs mentioned in the limitation, to ensure equal data representation for all languages we use in the experiment, we downsampled our data which resulted in about 200 MB per language. In reality, multilingual models will not be trained on an equal amount of data per language, so we have actually validated our findings using a public checkpoint of XLM-R which has been trained on a much larger resource with an unequal amount of data per language. We find that our findings are still robust with this model (Section 5.1). Furthermore, recognizing the inherent variance in linguistic properties, we scrutinized the potential influence of morphosyntactic feature frequency on our analysis (Section 5.2). Pearson correlation analysis revealed no significant correlation between feature frequency and average signature values across layers, addressing concerns of bias from varying feature frequencies across languages.\n\n*__Q: How do you plan to make the software and pre-trained models developed in this paper publicly available, and what steps have you taken to ensure their usability and accessibility for researchers with varying levels of expertise?__*\n\nWe intend to host the software and pre-trained models on GitHub. This will allow researchers to easily access, download, and contribute to the project. We will provide extensive documentation accompanying the software and models. This documentation will encompass installation guides, usage instructions, code examples, and explanations of key functionalities. By offering clear and concise documentation, we aim to cater to both newcomers and experienced practitioners. \n\n### Missing References\nThank you for identifying these and we will incorporate the missing references in the revised version of the paper. \n\n__References:__\n\n[1] Pratapa el al., 2021 https://aclanthology.org/2021.emnlp-main.570/\n\n[2] Plank and Klerke, 2019 https://aclanthology.org/W19-6103/ \n\n[3] Agic et al., 2014 https://aclanthology.org/W14-4203/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include the development of a novel analysis tool based on joint matrix factorization, a large-scale empirical study of over 33 languages and 17 morphosyntactic categories, and the demonstration of variations in the encoding of morphosyntactic information across different layers and influenced by language properties. The paper also provides insights into the relationship between morphosyntactic typology and downstream task performance, and the robustness of the findings in low-resource settings. \n\nIf this paper were to be presented at the conference or accepted into Findings, the main benefits to the NLP community would be the availability of a new analysis tool for comparing latent representations of multilingual and monolingual models, a better understanding of how morphosyntactic features are reflected in the representations learned by multilingual pretrained models, and insights into the relationship between morphosyntactic typology and downstream task performance. The paper's contributions to computationally-aided linguistic analysis, NLP engineering experiments, and publicly available software and pre-trained models would also be valuable to the NLP community. Overall, this paper has the potential to advance the state-of-the-art in multilingual NLP and contribute to the development of more effective and efficient multilingual models.", "reasons_to_reject": "One potential weakness of this paper is that the analysis is limited to morphosyntactic features and does not consider other linguistic features such as semantics or pragmatics. Additionally, the paper's focus on multilingual models may limit its generalizability to monolingual models or models trained on specific tasks. \n\nThe main risks of having this paper presented at the conference or accepted into Findings would be the potential for misinterpretation or overgeneralization of the findings. The paper's focus on multilingual models may lead some researchers to assume that the findings apply equally to monolingual models or models trained on specific tasks, which may not be the case. Additionally, the paper's emphasis on morphosyntactic features may lead some researchers to overlook other important linguistic features such as semantics or pragmatics. Finally, the paper's reliance on pre-trained models may limit its applicability to researchers who are developing their own models from scratch.", "questions_to_authors": " Could you provide more details on the sampling technique explored in this paper to enhance low-resource languages? How does it work and what were the results of your experiments with it?\n\n How do you plan to extend the analysis tool based on joint matrix factorization developed in this paper to consider other linguistic features such as semantics or pragmatics?\n\n How do you envision the findings of this paper being applied in practical NLP applications, particularly for low-resource languages? \n\n Could you discuss any potential limitations or biases in the data used in this study, and how you addressed them in your analysis? \n\n How do you plan to make the software and pre-trained models developed in this paper publicly available, and what steps have you taken to ensure their usability and accessibility for researchers with varying levels of expertise?\n", "paper_topic_and_main_contributions": "This paper is about a Joint Matrix Factorization Analysis of Multilingual Representations. The authors present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models. The paper addresses the problem of understanding how morphosyntactic features are reflected in the representations learned by multilingual pretrained models. The main contributions of this paper are the development of a novel analysis tool based on joint matrix factorization, a large-scale empirical study of over 33 languages and 17 morphosyntactic categories, and the demonstration of variations in the encoding of morphosyntactic information across different layers and influenced by language properties. The paper also provides insights into the relationship between morphosyntactic typology and downstream task performance, and the robustness of the findings in low-resource settings. The paper's contributions include computationally-aided linguistic analysis, NLP engineering experiments, and publicly available software and pre-trained models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": " the following references should be included in the bibliography:\n\n- Devlin, J. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.\n- Ginter, F., Hajiˇc, J., Luotolahti, J., Straka, M., & Zeman, D. (2017). CoNLL 2017 shared task - automatically annotated raw texts and word embeddings. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.\n- Sokal, R. R., & Michener, C. D. (1958). A statistical method for evaluating systematic relationships. University of Kansas science bulletin, 38(22), 1409-1438.\n- Stanczak, K., Ponti, E., Henningen, L. T., Cotterell, R., & Augenstein, I. (2022). Same neurons, different languages: Probing morphosyntax in multilingual pre-trained models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1589-1598).\n- Tenney, I., Das, D., & Pavlick, E. (2019). BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 4593-4601)."}}, {"review_id": "5emPGNpLqU", "reviewer": "Reviewer_3nrq", "summary": "", "questions": "1. How sensitive are the results to the choice of k (the \"hyper parameter\" in matrix factorization)?\n\n2. How are the representations derived for each word? Are they feed into models one word at a time? Or do you take the contextual representations? It is not clear from the writing.\n\n3. Is there any particular reason to use different model architecture for the monolingual models and the multilingual model (e.g., difference in number of attention heads).", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your feedback and pointing out the key advantages of our proposed method compared with existing matrix factorization techniques. \n\n### Reasons To Reject:\n*__\"lack of clarity on the condensed values that are used across a number of analyses in the paper.\"__*\n\nWe will make sure in the revised paper, we provide enough justification and explanation on taking averages on the signature values. We follow the same take average approach used by SVCCA [1], a very popular representation analysis tool, where they argue that the single condensed SVCCA score is the average correlation across aligned directions and is a direct multidimensional analogue of Pearson correlation. In our case, each pseudo singular value from our PARAFAC2 algorithm is associated with a direction, which are all normalized in length, and therefore, the pseudo singular values determine their strength. Thus, the average of the pseudo singular values gives the intensity of the representation of a specific language in the multilingual model. \n\n### Questions For The Authors\n\n*__1: How sensitive are the results to the choice of k (the \"hyper parameter\" in matrix factorization)?__*\n\nThe choice of k will certainly have some effect on the factorization process itself. In fact, setting k is not trivial for the PARAFAC2 algorithm [2]. That being said, based on our empirical observation, we noticed that a small portion of pseudo singular values of the components derived from the factorization tend to be exceedingly small, almost approaching 0. As a result, the impact of those singular values on our results (e.g. average signature values) is considered negligible. \n\n*__2: How are the representations derived for each word? Are they feed into models one word at a time? Or do you take the contextual representations?__*\n\nFollowing [3], we compute contextual representations for every individual word in the treebank by feeding them to our multilingual/monolingual models. We will make sure this is clear in the revised version of the paper. \n\n*__3: Is there any particular reason to use different model architecture for the monolingual models and the multilingual model (e.g., difference in number of attention heads).__*\n\nThe standard base RoBERTa employs 8 attention heads, while XLM-R uses 12 heads. When training our XLM-R from scratch, we had the option to align the attention heads to 8, matching RoBERTa. However, since one of our experimental setups involves a public XLM-R checkpoint (with 12 heads), we chose 12 heads for comparability. Importantly, our method doesn't mandate that multilingual and monolingual models must share the same architecture, such as the number of hidden dimensions. The core principle of our approach is that each monolingual model is trained on a subset of data utilized for the multilingual model's training. Our method is versatile and can accommodate monolingual models with distinct architectures and hidden dimensions.\n\n### Typos Grammar Style And Presentation Improvements\n*__\"It is not clear whether type-token ratio is computed based on word pieces or (UD-style) words.\"__*\n\nThe ratio is computed based on the token count provided in UD metadata (which is UD-style words). We will make sure this is clear in the revised paper.\n\n__References:__\n\n[1] Raghu et al., 2017 https://arxiv.org/abs/1706.05806 \n\n[2] Bro, 1997 https://www.sciencedirect.com/science/article/abs/pii/S0169743997000324 \n\n[3] Stanczak et al., 2022 https://aclanthology.org/2022.naacl-main.114/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The application of joint matrix factorization, PARAFAC2, to the analysis of representations derived from a set of monolingual and a multilingual model is interesting. Previous matrix factorization techniques used in the literature for analyzing multilingual representations are typically restricted to analyzing one pair at a time. This works demonstrates a way to perform analysis across all language pairs at the same time.\n- Some of the results from the analysis is interesting, for example, the phylogenetic tree derived from distance matrix, which is in turn constructed from the language-specific signature vectors, appears to be linguistically viable.", "reasons_to_reject": "My major concern is the lack of clarity on the condensed values that are used across a number of analyses in the paper. The signature vectors (taken from the language-specific \\sigma matrices) are condensed down to a single value by taking the average of the vector. And then the paper base a lot of analysis on top of these condensed values, e.g., look at the magnitude and spread of these values across languages and layers. It is not straightforward to understand what these average values mean (especially PARAFAC2 does not impose the same set of constraints as SVD). Maybe because it has been difficult to establish a clear idea of these condensed values, a lot of the following analyses and conclusions became hard to follow.  (Authors have responded below that this is following some prior practices to look at average correlations across aligned directions.)", "questions_to_authors": "1. How sensitive are the results to the choice of k (the \"hyper parameter\" in matrix factorization)?\n\n2. How are the representations derived for each word? Are they feed into models one word at a time? Or do you take the contextual representations? It is not clear from the writing.\n\n3. Is there any particular reason to use different model architecture for the monolingual models and the multilingual model (e.g., difference in number of attention heads).", "paper_topic_and_main_contributions": "The paper applies a joint matrix factorization method, PARAFAC2, to analyzing multilingual representations in pertained models. The PARAFAC2 is similar to SVD, but the factorization is done jointly over all pairs of models across the multiple monolingual models and the multilingual model, with the constraint that the V matrix is shared across all factorizations. The paper then focuses on the language-specific \\sigma matrices.\n\nIn particular, the paper shows that by analyzing the average signatures of the \\sigma matrices, we can infer about the encoding of morphosyntactic information in the different layers of the pertained models. Additionally, the paper shows that the language-specific \\sigma matrices can be used to interpret the similarities across languages, and the resulting phylogenetic tree appears to be linguistically viable. The factorization outputs can also be used to predict task performance across different languages.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It is not clear whether type-token ratio is computed based on word pieces or (UD-style) words."}, "extra_text": {}}, {"review_id": "QSBUEeolaw", "reviewer": "Reviewer_ETeL", "summary": "", "questions": "1) How to \"enforce the matrix V to be identical for all decompositions\"?\n\n2) It's mentioned that you \"downsample all Wikipedia datasets to an identical number of sequences in order to use the same amount of data\". How do you make sure this is fair to all languages (if there is a reference setting)? Due to the difference in linguistic properties, some languages e.g. Chinese might have much fewer tokens. Would it be better to add another condition on size in MB?\n\n3) How could you explain \"We found that all 33 languages except for Arabic, Indonesian, Japanese, Korean, and Swedish exhibit significant monotonically decreasing trends from lower layers to upper layers\"?\n\n4) In your low-resource settings where the training data is \"reducing it to only 10% of its original size\", is there a reference to support this setting? In a reality setting e.g. XLMR is trained on some low-resource language with 0.1%-1% of the English data size, do you consider this kind of setting would have any impact on your findings or not? \n\n5) How do you get performances on the XTREME benchmark? Directly copy from the XTREME paper or finetune the classifier by yourself? \n\n6) It's stated that \"These results suggest that the factorization outputs can serve as a valuable indicator of performance for downstream tasks, even without the need for finetuning or the availability of task-specific data.\" Could you provide a more general comparison - what are the computational requirements of these two approaches - to make this statement more convincing?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed review and for acknowledging the contributions presented in our work. We hope that the additional experiments we conducted have helped address your primary concerns regarding the lack of quantitative comparison with previous related methods.\n\n### Reasons To Reject\n\n*__\"The paper is too dense and lacks clarity in explaining some technical details of the joint matrix factorization analysis\"__*\n\nIn the revised version of the paper, we will enhance the algorithmic explanation by providing additional details in Sections 2 and 3. Moreover, we will add specifics regarding the PARAFAC2 implementation in the Appendix. As we aim to make our analysis tool accessible to the public, we will furnish comprehensive documentation, complete with illustrative examples, within the published repository. We note that PARAFAC2 is a well-known tensor factorization method, and we will include additional expository references.\n\n*__\"The paper does not provide a comprehensive quantitative discussion of the difference compared to other existing similar methods mentioned such as SVCCA and CKA\"__*\n\nWe first would like to thank you for mentioning Liang et al., 2021 and Choenni and Shutova, 2020. Both of these works focus on analyzing multilingual representations with language-specific and their typological properties, which is relevant to our work. We will make sure we discuss them in our revised paper. \n\nHowever, it’s worth noting that both of these approaches rely on probing, a widely employed method for analyzing multilingual representations and quantifying the information encoded by training a parameterized model. As highlighted in our paper, the efficacy of probing can be influenced by model parameters and evaluation metrics [2, 3, 4, 5]. In contrast, our proposed methodology centers around matrix factorization, offering a distinct approach to representation analysis. While techniques like SVCCA and CKA share similarities, they are confined to pairwise comparisons, which may not be optimally suited for comprehensive analysis of multilingual representations.\n\nTaking your suggestion into consideration, we performed quantitative comparisons with our method by calculating correlations between SVCCA scores and XTREME benchmark scores (from XLM-R model). This analysis was conducted for XSQuAD, MLQA, and BUCC due to time limitations. \n\n| Method   | XSQuAD (F1) | MLQA (F1) | BUCC (F1) |\n|----------|-------------|-----------|-----------|\n| SVCCA    | 0.36        | 0.28      | 0.56      |\n| Ours     | **0.73**        | **0.64**      | **0.83**      |\n\nEncouragingly, our proposed method consistently demonstrates higher correlation scores in all cases when compared to SVCCA. \nWe will add the results of the rest of the benchmarks upon acceptance. \n\n*__\"The models used are limited (XLMR and mBERT), the latest encoder-decoder and decoder-only models are not involved. \"__*\n\nWhile we acknowledge the importance of exploring various model architectures, as we have already highlighted in our limitations section, it's important to note that the primary emphasis of this work is not on exhaustive experimentation with all conceivable models but the novel multilingual analysis tool. Moreover, it's been observed that potent black-box LLMs like ChatGPT demonstrate shortcomings in comparison to more concise, task-specific models, especially when evaluating non-English scenarios [6]. Given that our study revolves around the analysis of multilingual capabilities and linguistic proficiency inherent in multilingual models, our focus remains on well-established models (e.g. XLM-R). An additional noteworthy point is the advantage of our analysis tool, which remains applicable as long as we can extract the desired linguistic features' representations from the model.\n\n*__\"It's an open question whether this method can be useful in a wider range of scenarios nowadays, e.g., in-context learning, prompt-tuning, and prefix-tuning.\"__*\n\nIn this work, we conducted tests of our multilingual analysis tool on models we trained from the ground up, as well as on publicly available checkpoints. These tests effectively underscore the resilience of our proposed analysis tool. As previously stated, the applicability of our tool extends as long as we can extract the intended representations from the model. In future work, we would like to investigate other inference methods such as prompting, in-context learning, and prefix-tuning.\n\n### Questions For The Authors\n*__1: How to \"enforce the matrix V to be identical for all decompositions\"?__*\n\nThe enforcement of V to be identical for all decompositions comes directly with the algorithm of PARAFAC2 [1]. We will make sure this is clearly explained in the revised paper. \n\n*__2: It's mentioned that you \"downsample all Wikipedia datasets to an identical number of sequences in order to use the same amount of data\". How do you make sure this is fair to all languages (if there is a reference setting)? Due to the difference in linguistic properties, some languages e.g. Chinese might have much fewer tokens. Would it be better to add another condition on size in MB?__*\n\nIn our preprocessed Wikipedia dataset, each language’s data is processed by chunks with the same sequence length. We ensured that the downsampled dataset is similar in size and each language in the end has approximately 200 MB file size (mentioned in Appendix B.1). We will make sure we are clear about this in the revised version. Addressing your valid concern about variations in linguistic properties and the possibility of certain languages having fewer tokens or morphosyntactic features that could impact our factorization outcomes, we conducted an investigation into the potential influence of morphosyntactic feature frequency on our analysis (as detailed in Section 5.2). We did a Pearson correlation analysis between feature frequency and average signature values across all layers. Our findings yielded no conclusive evidence of a correlation between these factors, thereby assuaging concerns about potential bias arising from diverse feature frequencies across languages.\n\n*__3: How could you explain \"We found that all 33 languages except for Arabic, Indonesian, Japanese, Korean, and Swedish exhibit significant monotonically decreasing trends from lower layers to upper layers\"?__*\n\nWhile the observed trends do not attain statistical significance, it's noteworthy that some p-values approach the 0.05 threshold. We also postulate that these languages might lack significant representation of closely related siblings in the pretraining languages. This aligns with our prior experiment's observation that languages indeed benefit from related counterparts. Recognizing the intriguing nature of this observation, we intend to delve deeper into this aspect in our future research.\t\n\n*__4: In your low-resource settings where the training data is \"reducing it to only 10% of its original size\", is there a reference to support this setting? In a reality setting e.g. XLMR is trained on some low-resource language with 0.1%-1% of the English data size, do you consider this kind of setting would have any impact on your findings or not?__*\n\nWe choose 10% since, in our low-resource experiment, we also pretrain the monolingual model from scratch using the 10% data for that language. Our initial trials using 1% or 0.1% data yielded an underfitting of the monolingual models. If our training capacity were expanded to 10 or 100 times the current scale, we would certainly incorporate these supplementary scenarios into our experimentation. However, the constraints of our computational resources currently prevent us from undertaking such extensive investigations. Additionally, we agree that in a realistic setup, languages are not represented equally in the data, and thus we experimented using a public checkpoint of XLM-R which we find that our findings are still robust (Section 5.1). \n\n*__5: How do you get performances on the XTREME benchmark?__*\n\nWe get the results on the XTREME benchmark from their paper. We will be clear about this in our revised version. \n\n*__6: It's stated that \"These results suggest that the factorization outputs can serve as a valuable indicator of performance for downstream tasks, even without the need for finetuning or the availability of task-specific data.\" Could you provide a more general comparison - what are the computational requirements of these two approaches - to make this statement more convincing?__*\n\nOur factorization outcomes carry significant potential for predicting performance, as they have showcased robust correlations with downstream task effectiveness (Section 5.3). To illustrate, consider a scenario involving a pre-trained multilingual model. Given knowledge of downstream performance for a set of languages, we aim to forecast the model's performance on language X—a language for which performance data is lacking. Traditionally, this process would demand task-specific data collection for language X, followed by GPU-based model fine-tuning and subsequent performance evaluation. However, with the aid of our analysis tool, this sequence simplifies remarkably. By extracting signature values through factorization (requiring only CPU resources), we can extrapolate language X's performance based on its signature values and the performances of other languages, streamlining the prediction process significantly.\n\n### Missing References\nFinally, thank you for identifying the missing references, and we will make sure to include them in the revised paper. \t\t\n\n**References:**\n\n[1] Harshman 1972 https://psychology.uwo.ca/faculty/harshman/wpppfac2.pdf \n\n[2] Saphra and Lopez, 2019 https://aclanthology.org/N19-1329/ \n\n[3] Zhang and Bowman, 2018 https://aclanthology.org/W18-5448/ \n\n[4] Pimentel et al., 2020 https://aclanthology.org/2020.acl-main.420/\n\n[5] Zhao et al., 2022 https://aclanthology.org/2022.blackboxnlp-1.16/ \n\n[6] Lai et al., 2023 https://arxiv.org/abs/2304.05613", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper proposes a different analytical tool based on joint matrix factorization, which enables a comprehensive comparison of multilingual representations and provides a unique perspective on the underlying structures and patterns.\n\n2) The answers to the proposed research questions in this paper are logically clear. Analyzing how multilingual models encode morphosyntactic features and how they vary across languages (with different linguistic properties) and across different layers provides valuable insights. Especially, The unique analysis of logographic writing systems caused differences in Chinese and Japanese morphosyntactic features with those in other languages is inspiring and interesting. \n\n3) The solid large-scale empirical research across multiple languages and morpho-syntactic categories enhances the generalizability of the findings.\n\n4) The paper demonstrates the utility of analytical tools by showing the correlation between decomposition output and performance in cross-lingual tasks, which has practical implications for downstream task selection.", "reasons_to_reject": "1) The paper is too dense and lacks clarity in explaining some technical details of the joint matrix factorization analysis e.g. How to \"enforce the matrix V to be identical for all decompositions\", and How to get token embedding (is it contextualized or the input is \"<bos> token <eos>\"). Please find more of these in the Questions. \nMore information (even in the appendix) on the specific implementation and algorithms used would be helpful for the readers to understand the analysis tool.\n\n2) The paper does not provide a comprehensive quantitative discussion of the difference compared to other existing similar methods mentioned such as SVCCA (Raghu et al., 2017) and CKA (Kornblith et al., 2019), and not mentioned such as [Liang et al., 2021](https://arxiv.org/abs/2109.08040) and [Choenni and Shutova, 2020](https://arxiv.org/abs/2009.12862). Therefore, the advantages and disadvantages of the methods are not quantitatively compared. \n\n3) The research scope of this paper is narrow and lacks novelty. The author did not quantitatively distinguish their proposed method from the previous related methods. The models used are limited (XLMR and mBERT), the latest encoder-decoder and decoder-only models are not involved. The adaptation to downstream tasks is limited to fine-tuning the classifier, did do not involve other novel finetuning and inference methods, such as cloze-style mask prediction, in-context learning, prompt-tuning, and prefix-tuning. It's an open question whether this method can be useful in a wider range of scenarios nowadays.", "questions_to_authors": "1) How to \"enforce the matrix V to be identical for all decompositions\"?\n\n2) It's mentioned that you \"downsample all Wikipedia datasets to an identical number of sequences in order to use the same amount of data\". How do you make sure this is fair to all languages (if there is a reference setting)? Due to the difference in linguistic properties, some languages e.g. Chinese might have much fewer tokens. Would it be better to add another condition on size in MB?\n\n3) How could you explain \"We found that all 33 languages except for Arabic, Indonesian, Japanese, Korean, and Swedish exhibit significant monotonically decreasing trends from lower layers to upper layers\"?\n\n4) In your low-resource settings where the training data is \"reducing it to only 10% of its original size\", is there a reference to support this setting? In a reality setting e.g. XLMR is trained on some low-resource language with 0.1%-1% of the English data size, do you consider this kind of setting would have any impact on your findings or not? \n\n5) How do you get performances on the XTREME benchmark? Directly copy from the XTREME paper or finetune the classifier by yourself? \n\n6) It's stated that \"These results suggest that the factorization outputs can serve as a valuable indicator of performance for downstream tasks, even without the need for finetuning or the availability of task-specific data.\" Could you provide a more general comparison - what are the computational requirements of these two approaches - to make this statement more convincing?", "paper_topic_and_main_contributions": "This paper presents an analysis tool based on PARAFAC2 - a classic joint matrix factorization method for comparing latent representations of multilingual and monolingual language models -  to analyze the encoding of morphosyntactic features in representations. With this analysis tool, the authors try to answer the following research questions: \nHow do multilingual language models encode morphosyntactic features in layers  (Both in common pretraining settings and low-resource settings).\nAnd what is the correlation between downstream task performance and the factorization outputs.\n \nThe authors pre-train multilingual and monolingual language models in XLM-R (RoBERTa) architecture with 33 languages from Wikipedia data. And conduct an empirical study of 17 morphosyntactic categories from UD treebanks by applying PARAFAC2 to analyze multilingual representations and monolingual representations. \nThey find variations in the encoding of morphosyntactic information across different layers, with category-specific differences influenced by language properties e.g. writing systems. The authors also show that the factorization outputs exhibit a strong correlation with performance observed across different cross-lingual tasks in the XTREME benchmark. \n\nThe main contributions of this paper include 1) proposing an analysis tool for multilingual representations; 2) the solid analysis and demonstration of morphosyntactic information in multilingual representations with language-specific differences; 3) the correlation analysis between factorization outputs and performance in cross-lingual tasks, which give an insight on practicality.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[Choenni and Shutova, 2020](https://arxiv.org/abs/2009.12862)\n [Liang et al., 2021](https://arxiv.org/abs/2109.08040)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree on the main reasons to accept:

* Novel analysis approach that compares across all language pairs at the same time.
* Large-scale empirical study.
* Insights into the relationship between morphosyntactic typology and downstream task performance.

However, they are mixed on excitement, due perhaps in part to a perceived lack of clarity in the paper's technical details.
Decision: Accept-Findings