Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization
Authors: Ningyu Xu
Qi Zhang
Jingting Ye
Menghan Zhang
Xuanjing Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: cross-lingual generalization
syntax
multilinguality
language models
TL;DR: This paper supports a prototype view of structural concepts within LLMs and points out the potential to leverage the conceptual correspondence between languages for cross-lingual generalization, especially to low-resource languages.

Abstract: Large language models (LLMs) have exhibited considerable cross-lingual generalization abilities, whereby they implicitly transfer knowledge across languages. However, the transfer is not equally successful for all languages, especially for low-resource ones, which poses an ongoing challenge. It is unclear whether we have reached the limits of implicit cross-lingual generalization and if explicit knowledge transfer is viable. In this paper, we investigate the potential for explicitly aligning conceptual correspondence between languages to enhance cross-lingual generalization. Using the syntactic aspect of language as a testbed, our analyses of 43 languages reveal a high degree of alignability among the spaces of structural concepts within each language for both encoder-only and decoder-only LLMs. We then propose a meta-learning-based method to learn to align conceptual spaces of different languages, which facilitates zero-shot and few-shot generalization in concept classification and also offers insights into the cross-lingual in-context learning phenomenon. Experiments on syntactic analysis tasks show that our approach achieves competitive results with state-of-the-art methods and narrows the performance gap between languages, particularly benefiting those with limited resources.

Are Structural Concepts Universal in Transformer Language Models?
Towards Interpretable Cross-Lingual Generalization
Ningyu Xu 122 Qi Zhang', Jingting Ye', Menghan Zhang Xuanjing Huang
Xuanjing
School of Computer Science, Fudan University
2 Institute of Modern Languages and Linguistics, Fudan University
3 Depaarmeett of Chinese Language and Literature, Fudan University
"eeeearch Institute of Intelligent Complex Systems, Fudan University nyxu22@m. fudan. Ludan.edu.com cn yej,gejingting wwwmmmgtzaapaaaa )
Abstract
relations (e.g.. "subject" and "object") are defined in cross-iingu consistent way. While large language models (LLMs) have demonstrated their capacity to induce these concepts within individual languages (Tenney et al., 2019; Liu et al., 2019; Chi et al., 2020; Linzen and Baroni, 2021), they encounter difficulties in generalizing the knowledge across languages (Joshi et al., 2020; Blasi et al., 2022; Majewska al., 2022). This raises questions about whether LLMs are able capture the underlying conceptual correspondence and how to harness the knowledge for improved generalization.
Large language models (LLMs) have exhibited considerable cross-lingual generalization abi]- ities, whereby they implicitly transfer knowl- edge across languages. However, the transfer is not equally successful for all languages, especially for low-resource ones, which poses an ongoing challenge. It is unclear whether we have reached the limits of implicit cross- lingual generalization and explicit knowl- edge transfer viable. In this paper, investigate the potential for explicitly aligning conceptual correspondence between languages enhance cross-lingual generalization. Using the syntactic aspect of language testbed, our analyses of 43 languages reveal high degree of alignability among the spaces of structural concepts within each language for both encoder-only and decoder-only LLMs. We then propose meta-learning-oat hased method to learn to align conceptual spaces different languages, which facilitates zero-shot and few- shot generalization in concept classification and also offers insights into the cross-lingual in- context learning phenomenon. Experiments syntactic analysis tasks show that our approach achieves competitive results with state-of-the- art methods and narrows the performance gap between languages, particularly benefiting those with limited resources.
Previous work has shown that cross-linguistic similarities are automatically captured in the representation space of LLMs, enabling zero-shot cross- transfer (Pires et al., 2019; Wu and Dredze, 2019; Chi et al., 2020; Papadimitriou et al., 2021; Muller et al., 2021; Xu et al., 2022). Efforts have been made to further enhance their generalization by exploiting high-resource languages for parameter and information sharing (Üstün et al., 2020; Nooralahzadch et al., 2020; Choenni et al., 2023) or enforcing alignment between languages (Cao et al., 2019; Schuster et al.. 2019; Sherborne and Lapata, 2022), whereas these approaches typically rely structural and lexical similarities between languages and fall short when dealing with low-resource languages distant from high-resource ones (Ponti al., 2021; de Lhoneux et al., 2022). With the ever-increasing size LLMs, recent work has explored methods to elicit the multilingual ability of LLMs via in-context learning (Winata et al., 2021; Tanwar et al., 2023), alleviating the cost of parameters updates, but the generalization performance lags behind and is highly sensitive prompt design (Lai et al., 2023; Ahuja et al., 2023). Overall, the generalization is predominantly realized implicitly and it remains unclear whether the commonalities shared across languages have been fully exploited.
1 Introduction
Cross-lingual generalization entails repurposing the knowledge acquired in one language to another with little supervision, thereby mitigating the digital language divide. Despite the vast variations across languages, it is possible to identify corre- sponding concepts among them, which provides a basis for cross-linguistic generalization (Croft, 1991; Haspelmath, 2010, 2021). This has been instantiated by frameworks such as Universal Dependencies (UD) (de Marneffe et al., 2021), where the structural concepts including word classes (e.g., "noun" and "verb") and grammatical

---

In this paper, we investigate the potential to explicitly leverage the conceptual correspondence between languages for cross-lingual generalization. We focus on the structural concepts outlined in UD, which are at the core of syntactic analyses across languages (Croft, 1991) and have shown to be learned by LLMs, offering valuable testbed for our analyses. For each language, we learn linear transformation of an LLM'' representation space, which defines conceptual space where samples can be classified by their distances to prototypes for each concept. Concepts represented by the LLM are then regarded as clusters discernible from others based on their prototypes. Analyses across 43 typologically distinct languages reveal a high de- gree of alignability among their conceptual spaces, indicating that the conceptual correspondence is implicitly established in LLMs (Section 2). We then present a ceata-learning--bee based method that learns to explicitly align different languages with limited data available, facilitating zero-shot and few-shot generalization in concept classification. We demonstrate the effectiveness of our approuch for both encoder-only (Section 3) and decoder-only LLMs (Section 4), achieving encouraging results especially for low-resource languages.
correspondence between different languages from plain text, which could lay the foundation for better cross-lingual generalization. Concretely, we first derive structural concepts within individual lan- guages, and then evaluate whether these concepts are readily alignable across languages.
2.1 Method
Deriving concepts Let = {x, J=1 denote dataset consisting of N feature-label pairs in a language I,, where the features X; E R are n- dimensional representations yielded by LLMs and the labels yi É {1,... .K} are the corresponding structural concept, and Dx the set samples with label k. We compute an m-dimensionall prototype C₁ C mh for each concept l by learning linear transformation A E Raxm such that
C₁ = DAA
Σ (X..B)))D&
Axi,
(1)
and the label of a feature x can be identified with respect to its distances to the prototypes in the transformed representation space. Specifically, the probability that x is an instance of concept is given by
In summary, our contributions are as follows: 1) We demonstrate that the conceptual correspon- dence between languages, in terms of the structural concepts defined in UD, is implicitly established in both encoder-only and decoder-only LLMs. 2) We propose weta-lcarning--asee approach explicitly aligning conceptual correspondence between different languages, enabling cross-lingual gemeralization in zero-shot and few-shot scenarios without requiring parameter updates to the I.LMs. Our method achieves competitive results with state- of-the-art methods and reduces the performance gap between languages, particularly benefiting low-resource ones. 3) Our approach provides insights into the cross tross-lingual in in-context learning phenomenon. Integrated with the prompt-based learning paradigm, achieves promising gains in generalizing to novel languages.
exp(-d ck)) PA -== EN exp (- (Ax, cw))
(2)
where d (-,-) is the squared Euclidean distance. The parameters of the transformation, the matrix A, is learned by minimizing the negative log probability L
log PA (y k x) of the gold concept k through gradient descent. Our probing method is inspired by Prototypical Networks (Snell et al., 2017), but we constrain the transformation to be linear as our goal is to investigate the geometry of LLMs, akin to Hewitt and Manning (2019).
Measuring Alignability We employ two com- plementary methods measure the alignabil- ity between structural concepts different lan- guages: representational similarity analysis (RSA) (Kriegeskorte et al., 2008) and Procrustes analysis. RSA is non-marametric and has been widely used for measuring the degree of topological aligmment between representation spaces based on their (dis)similarity matrices (Kriegeskorte and Diedrichsen, 2019). Procrustes analysis evaluates the extent to which two spaces can be aligned lincarly by finding the optimal orthogonal transformation.
2 Correspondence between Structural Concepts within Different Languages
This section investigates whether Transformer- based LLMs are able to induce the conceptual
'Our cnde is available at https://github.com com/ http:////stuuuuurracoooocppsotoo

---

FSA (MBERT)
RSA (LLAMA)
Procrurtes (1BERT)
Procruntes (LLaMA)
8
54
04
14
11
1
11
01
47
Figure I: Alignability between structural concepts (word classes) in different languages measured by RSA and Procrustes analysis, which is significantly higher than baselines. (The results for all languages, along with the alignability between grammatical relations, presented in Appendix A.2.)
Given two languages L1 and L2 with K shared structural concepts, we derive prototypes for each concept, which serve as parallel points that allows for comparison among different languages. Infrequently used concepts with fewer than 20 samples are excluded for analysis. For RSA, we compute dissimilarity matrix M R for each language, where the entry Mij d (c, e₁) is the distance between the ,th and jth prototypes (1 & i < K). The alignability is computed as the Spearman's correlation between the lower diagonal portion of the two matrices, ranging from -1 to 1. For Procrustes analysis, we evaluate the fitness of the linear transformation through the average proportion of explained variance.
NODETT
LLa
077
05-
06-
11.0
206
125
128 251
Figure Accuracy identifying word classes (POS) and grammatical relations (REL) averaged across different languages, with the linear transformation constrained to varying maximum dimensionality. The colored bands denote 95% confidence intervals. Strue- tural concepts can be identified based on prototypes within relatively low-dimensional space.
cases where the structure of their spaces is deformed. Given the prototypes for K structural concepts derived from two languages, construct the following baselines: (i) RP, which randomly swaps each prototypes for another one of the two languages; (ii) RC, where we randomly select a sample of each concept instead of their prototypes in one language; (iii RS, where we randomly select K samples in one language.
2.2 Setup
Model We use Multilingual BERT (mBERT) (Devlin et al., 2019) and LLaMA 7B model (Touvron et al., 2023) for our experiments. Both are pretrained on multiple languages without explicit cross-lingual information, enabling us to probe the cross- linsusiin knowledge induced exclusively from raw text. A linear transformation A E m with varying is trained project the features x E IRn yielded by the LLM into m-dimensional space, whereby we what rank of transformation is needed to extract structural concepts.
2.3 Results
Structural concepts can be identified based on prototypes The structural concepts includ- ing word classes and grammatical relations can be successfully distinguished according to their distances to the prototypes (Figure 2)². The structural information can be effectively encoded within relatively low-dimensional space, which varies across different models. We note that more expressive probing models are needed extract structural concepts, especially grammatical relations, from LLaMA; we leave exploration of
Data The data used in all our experiments is from UD v2.10 (Zeman et al., 2022). For mBERT, we select 43 typologically distinct languages that represent diverse range of language familics. For on 20 languages, including one that not included in its pretraining corpus for comparison.
LLaMA,
The alignability between the structural concepts in two languages should contrast with
Baselines
*ee take the 7th layer of mBERT here as it most effective in encoding structural information (Appendix A.
"See Appendix E for the languages and trechanks we use.

---

this to future work.
3 Aligning Conceptual Correspondence for Cross-Lingual Generalization
Structural concepts are readily alignable across languages Figure depicts the alignablity be- tween the structural concepts in different languages. Both word classes and grammatical relations are highly correlated across languages and can be approximately aligned through orthogonal trans- formation (rotation, reflection, etc.). Moreover, the alignability significantly higher than baselines, reinforcing that the conceptual correspondence between languages is reflected in the representation space.
The previous section shows that the universal structural concepts are readily alignable in the LLMs. Next, we investigate how to leverage the knowledge for cross-lingual generalization. We rely on meta-learning to learn to align conceptual correspondence between different languages both zero-shot and few-shot scenarios, analyzing how different factors including the number of available examples and the languages used for meta-training may impact the generalization.
2.4 Discussion
3.1 Method
It has been suggested that word embeddings in different languages are approximately isometric and can be aligned through linear transformation (Mikolov et al., 2013; Lample al., 2018; Schuster al., 2019). However, the meaning of each indi- vidual words, rather than the underlying concepts (Youn et al., 2016; Xu al., 2020), might not be indeed alignable across languages (Thompson et al., 2020), and enforeing such alignment can hurt downstream performance (Glavaš et al., 2019; Wu and Dredze, 2020). We here propose to establish the alignment based conceptual correspondence that can serve yardsticks for cross-linguistic comparison (Haspelmath, 2021), and the structural concepts defined in UD are generally designed to meet the need.
Learning to align with few examples We first derive prototypes cf from a source language Ls following the method in Section 2.1, and then learn to align samples in different languages with cf, via meta-learning. We employ a composite function F = 9a o fo to establish the alignment. The function fø R" + R" with parameters ø is language-agnostt and projects features yielded by LMs into an m-dimensional space, where samples belonging to each concept in target language LT are expected cluster around their prototypes c/ As shown in Section 2.3, an orthogonal transformation suffices to align the prototypes in two languages while preserving the geometry of the original spaces. We thus use language-spede specific linear mapping Ja m -> Rh to convert cf into prototypes c which allows us to identify the structural concepts according to
The alignability of structural concepts across dif- ferent languages is relatively consistent, providing evidence that their correspondence implicitly en- coded in LLMs, though not well aligned. However, variations remain between different language pairs. Besides subtle cross-linguistic differences with regard to the structural concepts (e.g., Ponti al., 2018), this might result from the lack sufficient data in the UD treebank for approximating the prototypes, and ii) the degenerate representation spaces of certain languages, which has been at- tributed to factors including inadequate pretraining data and deficiencies in tokenization methods for specific languages (Vulié al., 2020; Rust al., 2021; Blaschke et al., 2023; Purkayastha et al., 2023). The disparities among languages are also reflected in our experiment results regarding the classification of structural concepts, especially for languages not well represented in the pretraining corpora*.
(xp(-d((xx).c))) Expep( (x
PF (y =
(3)
The parameters are together optimized by minimiz- ing the negative log-probability the gold concept k. We use labeled data in multiple languages to learn the function F during meta-training. The lgnnsscii function fØ is optimized over the entire training procedure, and the language- specific function Ia is learned separately for different languages. During meta-testing, So is kept fixed while ga is learned from scratch using the provided examples.
Aligning with unified prototypes for zero-shot generalization In the zero-shot setting, instead of being given few examples to learn the alignment, we rely on meta-learning to establish unified prototypes for each concept. We learn a linear
"See Appendix A.I for details.

---

sr
gal
be
br
cy
10'
gow
kk
mr
pem
sa
ta
le
wbp"
yo
AVG
STD
UDAPTER
96.9
72.2
69.7
79.6
65.9
83.4
66.5
54.7
42.2
70.3
84.2
78.4
34.1
63.7
58.4
0.214
M28-0
92.2
71.4
90.3
75.4
69.9
84.6
64.7
77.2
79.3
37.7
45.2
73.1
78.2
73.8
51.6
61.0
56.6
0.212
M28-10
94.3
73.9
89.3
79.4
73.4
85.1
68.8
79.0
80.1
43.8
49.4
77.5
80.2
80.5
54.6
63.1
59.0
0.208
M28-50
95.2
75.9
90.7
81.1
76.9
86.3
70.6
79.8
79.0
64.3
52.0
79.5
80.3
83.7
58.8
67.0
62.5
0.183
M43-0
96.2
86.8
90.3
73.9
88.5
83.8
63.4
77.7
87.5
47.3
46.0
81.3
90.4
70.0
49.3
58.0
57.9
0.228
M43-10
95.8
86.6
89.5
80.3
87.3
84.9
69.0
80.2
86.7
54,5
48.7
81.9
90.0
78.6
53,9
62.2
60.8
0.211
M43-50
96.4
87.0
91.6
82.3
88.9
85.7
72.2
80.3
88.6
70.6
49.8
81.5
90.0
82.8
58.8
66.7
64.4
0.187
Table I: The zero-shot and few-shot generalization performance on POS tagging for subset of languages unseen during meta-traina meta-trrining and low-resource languages. Languages marked with "*" are not included in the pretraining corpus, "f" indicates that the language is involved in meta-training for M43, AVG and STD denotes the average accuracy and standard deviation respectively for 30 low-resource languages. (The results for all languages, together with the performance on the classification of grammatical relations, are given Appendix B.)
mapping h R. - Rm to convert cx to unified prototypes og = ht (o)) and use a language- agnostic function fa to match samples with them. The classification
0.9
0.8
0.7
then performed based on d (fo (x), e)) We optimize the parameters during meta-training and directly apply the models to other languages for meta-testing.
0.8
0.5
0.4
Nathod
0.3
10
N Nahoos
3.2 Setup
Figure Accuracy of M28 in identifying word classes in languages used during meta-training (depicted by bluish lines) and novel languages encountered during meta-testing (depicted by reddish lines) given N sentences. (The results for the classification of grammatical relations are shown in Appendix B.)
Model We derive representations from the 7th layer of mBERT. The language-agnostic function fo is parameterized by 2-layer perceptron with h hidden units that projects the features derived from mBERT m-dimensional space. We set h 256 and m 32 for word class identification, i.e.. pfrt-of-speech (POS) tagging. For grammatical relation, we set h 384 and m = 64.
it as the task-specific layer. (ii) UDapter, which is state-of-ther ofrtte-rrt multilingual parser (Üstün et al., 2022) that extends mBERT with adapters whose parameters are generated based on language embeddings from URIEL (Littell al., 2017).
Data To investigate the impact of languages involved in meta-training on performance, we examine two distinct settings: i) M28 with 28 high-resource languages included in meta-training and ii) M43 with 15 additional languages, which are primarily low-resource ones. Unless otherwise stated, we use English as the source language. The languages and datasets used here is shown in Appendix E.
3.3 Results
Explicit alignment benefits low-resource lan- guages Our method achieves competitive results with UDAPTER without any parameter updates to the LLM (Table 1), and even surpasses it for some languages distant from high-resource ones like Marathi (mr) and Warlpiri (wbp). Moreover, our method helps mitigate the performance gap between different languages, shown in the reduced standard deviation among languages.
We randomly select N sentences from the training set of target language as the support set, and evaluate the model on its test set in terms of accuracy. We vary N from 0 to 200 to test the number of sentences needed for generalization.
Evaluation
Meta-learning supports efficient alignment Comparing M28 and M43, we observe that, by incorporating low-resource languages like Marathi (mr) into meta-training, even with limited data substantial performance gains can be achieved Forr languages without training sets, we select sentences from their test sets and evaluate on the remaining data.
Baselines We compare our method with the following baselines: (i) FT The mBERT model is fine-tuned with the N available sentences in the target language, with linear classifier top of
"The training set of Marathi consists of only 373 sentences.

---

without compromising performance in other languages. This suggests that the inclusion of diverse languages in meta-training is critical for learning to derive conceptual spaces from LLMs. Based on the knowledge meta-learned from various languages, the model efficiently learns to align unseen languages with little supervision and optimizes at about 50 sentences, in contrast to FT (Figure 3), supporting the effectiveness of our method.
Iepit P'ezee
Oclputs Dleazow EREO Lsowtnnn 2008 Ieto
Impret the hoseend are new
Ouiguts "hheeer louoes-900 aCC SArewAD2 LACACCT
Impen: would finn
sieguti "he Havaa wiy would diaasx becouux "ymp
Head
KOUN
ADJ
1648
Mete-laaree Head
Figure 4: Sequence tagging via structured prompting. The classification of structural concepts is performed in a sequential manner.
3.4 Discussion
Our results taps into the potential for aligning systems possessing common structure, which has shown to support generalization even the absence of explicit supervision (Roads and Love, 2020; Aho et al., 2022). Previous work has suggested that word embeddings in different languages are automatically aligned through joint pretraining (Cao et al., 2019; Conneau al., 2020). In terms of structural concepts, their correspondence is reflected in the geometry of pretrained LLMs and have been aligned to certain extent (Chi et al., 2020). However, the alignment is not optimal and can be improved with a few examples, as suggested by Lauscher et al. (2020). Our approach may be expanded to other aspects of language and, by utilizing the knowledge acquired via self- supervision, it is promising to better accommodate the rich linguistic diversity despite the scarcity of labeled data.
the model is given small number of demonstration examples and then required to label additional sen- tences either the same language or a different one. During in-context learning, an LLM is provided with N pairs of sentences and tagged sequences as task demonstrations and a query sentence to be labeled. It is then required to iteratively tag the words. Specifically, given an input sequence é with N demonstration examples and the query sentence S = 81, 81...., Sn, at each time step I, the LLM M encodes [C; st] and generate the label of & with G = arg max PM (c ft) The input sequence is then updated with the predicted label è and the following word 81411 appended to the end of L.
Probing underlying mechanisms Firstly, we take the demonstration examples query tences, and evaluate the accuracy of the LLM in classifying these examples. We then investigate whether the representation space contextualized by the demonstrations effectively serves as a conceptual space where samples can be classified based on their distances to prototypes for each concept. We use the N demonstration examples query sentences and obtain their representations h, at time step for generating the label é. whereby we construct dataset D {x;, 34) ) based on h, and the gold label e.. We follow the approach in Section 2. to probe the extent to which structural concepts can be derived from the contextualized representation space, with the exception that the linear transformation is an identity matrix. Finally, modify the labels and languages provided in demonstrations to assess whether results generalize across different settings.
4 Aligning Conceptual Correspondence during In-Context Learning
In this section, further explore whether the correspondence between structural concepts can be harnessed for cross-lingual generalization in the in-context learning setting. Specifically, we first show that the cross-lingual syntactic abilities of LLMs can be elicited through in-context learning. We then rely on our method to probe the underlying mechanisms and further enhance the cross-lingual generalization by aligning the structural concepts within different languages.
4.1 Method
Learning cross-lingual structural concepts in context We focus on the POS tagging task and use the structured prompting method proposed by Blevins et al. (2022) to evaluate the few-shot in- context learning ability of LLMs (Figure 4), where
Meta-learning for better generalization Our analysis demonstrates that the LLM learns to ac- curately label the demonstration examples through

---

in-context learning, but the generalization per- formance falls short in both monolingual and cross-lingual settings. We thus rely on our meta- learning-base method to improve the LLM's generalization ability. As previously mentioned, we obtain prototypes by utilizing the demonstration examples query sentences, and then learn to align representations of other query sentences, contextualized by these demonstrations, with the prototypes. This resembles the zero-shot setting Section 3.1, but we discard the linear mapping applied to the prototypes, as the protolypes themselves are projected into a contextualized representation space and serve as good anchor points. During meta-learning, we introduce varying demonstration examples to construct different training episodes, and the model is directly applied across different contexts for meta-testing.
UPOS
SHFL
PXY
WORD
ICL
99.4
99.4
99.9
99.2
Proto
92.7
93.5
99.3
96.1
Table 2: Accuracy in POS tagging in English (en) for demonstrations taken as query sentences. ICL denotes the performance achieved through in-context learning, and Proto for classification based on prototypes computed based our method.
0.8
Acpurecy
0.6
0.4
Nets
0.0
da
de
ent
85
h1
ja
pl
tr
2h
Language
Figure 5: The few-shot generalization performance on POS tagging cross-lingual (EN) in-context learning settings, with English as the source language. META denotes our meta-learning-basee method. Error bars represent the standard deviation calculated from 10 runs. Languages marked with "x" are not included in the pretraining corpus. "i" indicates that the language is involved in meta- meta-training training for META. (The results for all languages are presented Appendix C.2.)
monolingual (MONO) and
4.2 Setup
Model We use LLaMA-7B as the underlying LLM. The network used for meta-learning resem- bles the one described in Section 3.2, which is a 2-layer perceptron with a hidden layer of size 5 512.
Data We employ 24 languages for our exper- iments, among which languages are used for meta-training, We represent each POS tag with the token corresponding the surface form of the label defined in UD by default (UPOS), c.g., "NOUN". We investigate three additional settings where the label forms are modified: i) SHFL, which shuffles the surface forms of the labels, ii) PXY, which uses proxy labels where each class is represented by an arbitrary token we employ capital alphabet letters here, and iii) WORD, which uses words as labels, e.g.. "adverb"?
the demonstrations provided are in English and perform 10 runs for each language with 50 query sentences randomly sampled from the training of UD. The evaluation for each language consists of runs under similar settings, with the exception that the query sentences are sampled from the test set of UD.
4.3 Results
LLM successfully learns to label the demonstra- tion examples, but with limited generalization abilities Table shows that the LLM able to accurately label demonstration examples, regard- less of changes in the label forms. Moreover, the contextualized representations of the demonstra- tions can be effectively classified based on the prototypes, indicating a good conceptual space for them. However, the performance significantly decreases when generalizing to unseen sentences, and the cross- lrogs-lingual generalization performance can be even worse (Figure 5).
Evaluation We randomly select 9 sentences from the training set in source language as demonstrations, ensuring they cover the label space if possible. For in-context learning, the LLM is evaluated on 50 randomly selected sentences from the test set for each language. We report the average accuracy across 10 runs, where a sample considered correctly labeled only the first word the LLM generates after seeing the delimiter matches the form of the gold label. In terms of the probing and meta-learning experiments,
A Aligning with demonstrations supports general- ization Through learning align with prototypes derived from a few demonstration examples, our method achieves remarkable gains in generaliza- tion for both monolingual and lingull scenar-
focus on the setting where
'The languages and datasets are listed in Appendix F, and the detailed label forms can be found in Appendix C.

---

ios (Figure 5). Extending the inclusion of diverse languages in our method holds promise for further improving cross-lingual generalization, particularly for languages that are not well represented in the meta-training languages, like Japanese (ja). These findings support the effectiveness of our method even in the face of variations introduced by changes in demonstrations, and suggest that better generalization performance be achieved through explicit alignment.
Bjerva and Augenstein, 2021; Xu et al., 2022), en- abling efficient zero-shot cross-lingual transfer. We here further explore whether the correspondence between structural concepts are reflected in LLMs' representation space, which could potentially be harnessed for better generalization.
Cross-lingual generalization While multilin- gual LLMs are capable of zero-shot cross-lingual transfer across various tasks (Pires al., 2019; Wu and Dredze, 2019; Winata et al., 2021), the performance sensitive to linguistic diversity. Efforts have been made to facilitate generalization to low-resource languages by learning proper information sharing (Ammar et al., 2016; Üstün et al., 2020; Nooralahzadeh et al., 2020) and optimizing data selection (Ponti et al., 2018; Lin et al., 2019; Glavaš and Vulić, 2021), but problems remain when it comes to outlier languages for which there is no high-resource related ones (Blasi et al., 2022). Another line of work strives to overcome the language barriers by imposing alignment of word embeddings (Lample et al., 2018; Ruder et al., 2019; Schuster et al., 2019; Cao et al., 2019). While the aligmment typically ensures that semantically and syntactically similar words are clustered together, it is left implicit whether the underlying conceptual correspondence between languages are properly aligned.
4.4 Discussion
While in-context learning has shown an efficient way to leverage LLMs for various downstream tasks and enables few-shot generalization (Brown et al., 2020; Winata et al., 2022), the underlying mechanisms remain unclear. Previous research has suggested that prompting can be regarded as probing knowledge from LLMs (Li al., 2022a; Blevins et al., 2022; Alivanistos al., 2022), but the performance is sensitive to prompt engineering, including factors such as the label space and the input distribution (Zhao et al., 2021; ; Lu et al,, 2022; Min et al., 2022; Mishra et al.. 2022). We here investigate the representation space of LLMs and find the demonstrations them despite changes in the label forms. These demonstrations establishes conceptual space with which we may align different samples. Our findings are in line with Olsson et al. (2022), suggesting that LLMs learn to match the patterns in the context, and offer insights into improving the generalization abilities of LLMs beyond prompt design.
effectively learned by
Meta-learning for generalization Meta- learning has showcased great success in enabling effective generalization. Through the process learning to learn, namely, improving learning over multiple learning episodes (Wang et al., 2020; Huisman et al.. 2021; Hospedales et al., 2022), it facilitates rapid adaptation novel contexts with limited data available. Prior work has exploited methods including Model-Agnostic Meta-Learning (Finn et al., 2017), Reptile (Nichol et al., 2018) and Prototypical Networks (Snell et al., 2017) for improved cross-lingual generalization (Ponti et al., 2021; Langedijk et al., 2022; Sherborne and Lapata, 2023; Cattan et al., 2021). Our method is similar to Prototypical Networks, but instead of estimating prototypes with the few available samples for each class, we derive prototypes from source language and learn to align different
5 Related Work
Probing linguistic knowledge Pretrained LLMs have shown able to induce sophisticated linguistic knowledge via self-supervision (Manning et al., 2020; Linzen and Baroni, 2021). As evidenced by probing analyses, structural information in- cluding word classes, grammatical relations and syntactic parse trees can be decoded from their representations to a remarkable extent (Conneau et al., 2018; 8; Blevins et al., 2018; Liu et al., 2019; Tenney et al., 2019; Clark et al., 2019; Hewitt and Manning, 2019; Eisape al., 2022). Expanded the multilingual setting, these models automatically capture nuanced similarities and differences between languages (Chi et al., 2020;
languages with them, verifying the conceptual correspondence captured by LLMs.
Papadimitriou et al., 2021; Singh et al., 2019;

---

6 Conclusion
Acknowledgements
The authors would like to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62076069, (100699961199 61976056).
We have demonstrated that multilingual LMs are able to induce the correspondence between structural concepts within different languages without any explicit supervision. This knowledge is encoded in their geometry and can be exploited for generalization, whereby we rely on meta- learning to learn to align different languages with minimal examples available. Our approach can be used to evaluate the correspondence between different systems that has been acquired by LLMs, and explicitly leverage them for sample-efficient generalization, suggesting new path toward mea- suring and manipulating the knowledge encoded in LLMs. Future research may generalize to other contexts (e.g., other modalities) to probe the commonalities and differences shared between systems and develop more sophisticated way for alignment and generalization.
References
Kaarina Aho, Brett Roads, and Bradley C. Love. 2022. System alignment supports cross-domain learning and zero-shot generalisation. Cognition, 227:105200.
Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12522
Dimitrios Alivanistos, Selene Báez Santamaría, Michael Cochez, Jan-Christoph Kalo, Emile van Krieken, and Thiviyan Thanapalasingam, 2022. Prompting as probing: Using language models for knowledge base construction. arXiv preprint arXiv:2208, 1055..
Waleed Ammar, George Mulcaire, Miguel Ballesteros. Chris Dyer, and Noah A. Smith. 2016. Many languages, one parser. Transactions of the Association for Computational Linguistics, 4:431- 444.
Limitations
Our goal in this work is measure the underlying conceptual correspondence between languages en- coded in LLMs, and leverage it for generalization. While we have demonstrated the effectiveness of our approach, it is only a first step toward the more general goal. The foremost limitation of our approach is that it relies on the comparable concepts defined by linguists and manually created datasets derive proper features for analyses. Continued research could expand upon it other tasks where prior knowledge of association is available and explore how different kinds of alignment impact the performance of LMs.
Johannes Bjerva and Isabelle Augenstein. 2021. Does typological blinding impede cross-lingual sharing? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 480 480,486, Online. Association for Computational Linguistics.
Verena Blaschke, Hinrich Schütze, and Barbara Plank. 2023. Does manipulating tokenization aid cross-lingual transfer? study on pos tagging for con-standard languages. arXiv preprint arXiv:2304. 10158
Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance actoss the world's lan- guages. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 5486-5505 Dublin, Ireland. Association for Computational Linguistics.
Despite the correspondence, our findings suggest that the alignability as well as the generalization performance still varies across languages, though not as pronounced as in the zero-shot cross- lingual transfer scenario. These variations may be attributed to factors like nuanced cross-linguistic differences, degraded representation space for some low-resource languages and insufficient task- specific data. Further investigation is needed to fully understand the cause of these disparities and how they can be reduced improve the generalization abilities of pretrained LMs, e.g., how to improve the representation spaces of individual languages.
Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. Prompting language models for linguistic structure. arXiv preprint
Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep RNNs encode soft hierarchical syntax. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume
2: Short Papers), pages 14-19, Melbourne, Australia. Association for Computational Linguistics.

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhimi Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
William Croft. 1991.
Syntactic categories and grammatical relations: The cognitive organization of information. University of Chicago Press.
Miryam de Lhoneux, Sheng Zhang, and Anders Søgaard. 2022. Zero-shot dependency parsing with worst- case aware automated curriculum learning. In Proceedings of the 60th Annual Meeting of the for Association for Computational Linguistics (Volume
2: Short Papers), pages 578- 578-587, Dublin, Ireland. Association for Computational Linguisties.
Catre-Cattherne Marneffe, Christopher D. Manning, Joakim Nivre, and Damiel Zeman. 2021. Uni- versal Dependencies. Computational Linguistics, 47(2):255-308.
Steven Cao, Nikita Kitaev, and Dan Klein. 2019. Multilingual alignment contextual word represen- tations. In International Conference on Learning Representations.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Oralie Cattan, Sophie Rosset, and Christophe Servan. 2021. On the cross- pooss-lingual transferability of multilingual prototypical models across NLU tasks. In Proceedings the st Workshop on Meta Learning and Its Applications to Natural Language Processing. pages 36-43, Online. Association for Computational Linguistics.
Tiwalayo Eisape. Vineet Gangireddy, Roger Levy, and Yoon Kim. 2022. Probing for incremental parse states in autoregressive language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2801-2813, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ethan A. Chi, John Hewitt, and Christopher Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564-5577 Online. Association for Computational Linguistics.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMIR.
Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. 2023. Cross-Lingual Transfer with Language- Specific Subnetworks for Low-Resource Dependency Parsing. Computational Linguistics, pages 1-29.
Proceedings of the 34th
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher Manning. 2019. What does BERT
Goran GlavaK, Robert I. Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (properly) evaluate cross- lingual word embeddings: On strong baselines. comparative analyses, and some misconceptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 710-721, Florence, Italy. Association for Computational Linguistics.
look at? analysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLPP: Analyzing and Interpreting Neural Networks for NLP. pages 276-286, Florence, Italy. Association for Computational I Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loie Barrault, and Marco Baroni. 2018. What you can cram into a single ! vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 2126-2136, Melbourne. Australia. Association for Computational Linguistics.
Goran Glavaš and Ivan Vulić. 2021. Climbing the tower of treebanks: Improving low-resource dependency parsing via hierarchical source selection. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 0878-4888. Online. Association for Computational Linguistics.
Alexis Conncau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov, 2020. Emerging cross-line cross-lingual structure pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022-6034, Online. Association for Computational Linguistics.
Harald Hammarström, Robert Forkel, Martin Haspel- math, and Sebastian Bank. 2022, gltttolg/ggootolog: Glottolog database 4.7.
Martin Haspelmath. 2010. Comparative concepts and descriptive categories crosslinguistic studies. Language, 8663):663-687.

---

Martin Haspelmath. 2021. General linguistics must be based on universals (or non-conte con-conventional aspects of language). Theoretical Linguistics, 1-11-
2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 4483-4499, Online. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019.
Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. 2022a. Probing via prompting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1144-1157, Seattle, United States. Association for Computational Linguistics.
structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 4129-412 4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. 2022b. Cross-domain few-shot learning with task-specific adapters. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 7161-7170.
T. Hospedales, A. Antoniou, Micaelli, and A. Storkey. 2022. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis & Machine Intelligence, 44(09):514-5516)
Mike Huisman, Jan N. van Rijn, and Aske Plaat. 2021. survey of deep meta-learning Artificial Intelligence Review, 54(6):448344511
Yu- Lin, Chian- Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang. Mengzhou Xia, Shruti Rijhwani, Junxian He. Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125-3135, Florence, Italy. Association for Computational Linguistics.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury, 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguistics.
Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics. 7(1) 195-222.....
Nikolaus Kriegeskorte and Jörn Diedrichsen. 2019. Peeling the onion of brain representations. Annual Review of Neuroscience, 42(1):407 PMID: 31283895.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turnrr, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. pages 8-14, Valencia, Spain. Association for Computational Linguistics.
Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. 2008 Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2.
Viet Dac Lai. Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Demoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. CoRR, abs/2304.05613.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov. Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contex- tual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Guillaume Lample. Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018 Word translation without parallel data. In Interna- tional Conference on Learning Representations.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel. and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8086-8098, Dublin, Ireland, Association for Computational Linguistics.
Anna Langedijk, Verna Dankers, Phillip Lippe, Sander Bos, Bryan Cardenas Guevara, Helen Yannakoudakis, and Ekaterina Shutova. 2022. Meta-leane leeanne for fast cross-lingual adaptation in dependency parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Pupers), pages 8503-8520, Dublin, Ireland. Association for Computational Linguistics.
Proceedings of
Olga Majewska, Ivan Vulić, and Anna Korhonen. 2022. Linguistically guided multilingual nip: Current approaches, challenges, and future perspectives. Algebraic Structures in Natural Language, pages 163- 188.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the

---

Christopher D. Manning, Kevin Clark, John He- witt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 1 117(48)300066 30054.
In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy, Association for Computational Linguistics.
Edoardo Maria Ponti, Rahul Aralikatte, Disha Shri- vastava, Siva Reddy, and Anders Søgaard. 2021. Minimax and neyman Penannn meta-learning for outlier languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1245-1260, Online. Association for Computa- tional Linguistics.
Tomas Mikolov, Quoe V Le. and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309,44
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Arteixe, Mike Lewis, Hannameh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-11066 Abu Dhabi, United Arab Emirates.
Edoardo Maria Ponti, Roi Reichart, Anna Korhonen, and Ivan Vulié. 2018. Isomorphic transfer of syntactic structures in cross-lingual NLP. In Proceedings of the 56th Annual Meeting of the for Association for Computational Linguisties (Volume I: Long Pupers), pages 1531-1542, Melbourne, Australia. Association for Computational Linguisties.
Association for Computational Linguistics.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022. Reframing instructional prompts GPTk's language. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 589-612, Dublin, Ireland. Association for Computational L Linguistics.
Sukannya Purkayastha, Sebastian Ruder, Jonas Pfeiffer, Iryna Gurevych, and Ivan Vulić. 2023. Romanization- based large-scale adaptation of mul- tilingual language models. arXiv preprint arXiv:2304.088 08805.8.6...
Brett D. Roads and Bradley C. Love. 2020. Learning as the unsupervised alignment of conceptual systems. Nature Machine Intelligence 2(1))76-82.
Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Understanding the cross-lingual ability of multilin- gual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 2214-2231, Online. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Artif. Int. Res., 65(1):569-633) 630.
Phillip Rust, Jonas Pfeiffer, Ivan Vulić. Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 3118-3132 Online. Association for Computational Linguistics.
Alex Nichol, Joshua Achiam, and John Schulman. 2018 On first-order meta-learning algorithms. CoRR,
Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning. In Proceed- ings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 4547-4562, Online. Association for Computational Linguistics.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson, 2019. Cross-lingual alignment of con- textual word embeddings, with applications to zero- shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 1599-1613, Minncapolis, Minnesota. Association for Computational Linguistics.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.1 11895.
Isabel Papadimitriou. Ethan A. Chi, Richard Futrell, and Kyle Mahowald. 2021. Deep subjecthood: Higher-order grammatical features in multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2522-2532, Online. Association for Computational Linguisties.
Tom Sherbome and Mirella Lapata. 2022. Zero-shot cross-lingual semantic parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 4134-4153, Dublin, Ireland. Association for Computational Linguistics.
Tom Sherhorne and Mirella Lapata. 2023. Meta- Learning Cross-lingual Manifold for Semantic Parsing- Transactions of the Association for Computational Linguistics, 1:49-677.
Telmo Pires, Eva Schlinger, and Dan Garrette, 2019. How multilingual is multilingual BERT?

---

Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is not an interlingua and the bias of tokenization, In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (Deeplo 2019), pages 47-55, Hong Kong, China. Association for Computational Linguistics.
Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietes 2022. Cross- lingual few-shot learning unseen languages. In Proceedings of the 2nd Conference of the Paa-Paccficc Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 777-791, Online only. Association for Computational Linguistics.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. In Proceedings of the 1st Workshop Multilingual Representation Learning, pages 1-15, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. 2023. Multilingual Iims are better cross loos-linguual in-context learners with alignment. CoRR, abs/2305.059400.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising gross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNL pages 833-844, Hong Kong, China, Association for Computational Linguistics.
Jan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Bllie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference Learning Representations.
Bill Thompson, Seán G. Roberts, and Gary Lupyan. 2020. Cultural influences on word meanings revealed through scage-scale semantic alignment. Nature Human Behaviour, 4 1029-1003-
Shijie Wu and Mark Dredze. 2020. Do explicit alignments robustly improve multilingual encoders? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4471-4482, Online. Association for Computa- tional I Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
Ningyu Xu, Tao Gui, Ruotian Ma, Qi Zhang, Jingting Ye, Menghan Zhang, and Xuanjing Huang. 2022. Cross-linguistic syntactic difference in multilingual BERT: How good it and how does affect transfer? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 8073-8092, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ahmet Üstüin, Arianna Bisazza, Gosse Bouma, and Gertjan Noord. 2020. UDapter: Language adaptation for truly Universal Dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2302-2315, Online. Association for Computa- tional Linguistics.
Yang Xu, Khang Duong, Barbara C. Malt, Serena Jiang. and Mahesh Srinivasan. 2020. Conceptual relations predict colexification across languages. Cognition, 201: 104280.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan Noord. 2022. UDapter: Typology- based language adapters for multilingual dependency parsing and sequence laheling. Computational Linguistics, 48(3):555-500 592.
Hyejin Youn, Logan Sutton, Eric Smith, Cristopher Moore, Jon F. Wilkins, Ian Maddieson, William Croft, and Tanmoy Bhattacharya. 2016. On the universal structure of human lexical semantics. Proceedings of the National Academy of Sciences. 113(7):1666-7711
Ivan Vulić, Sebastian Ruder, and Anders Søgaard. 2020. Are good word vector spaces isomorphic? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3178-3192, Online. Association for Computa- tional Linguistics.
Daniel Zeman, Joakim Nivre, Mitchell Abrams, Flia Ackermann, Noëmi Aepli, Hamid Aghaei, Željko Agié, Amir Ahmadi, Lars Ahrenberg. Chika Kennedy Ajede, Gabrielé Aleksandravičiūte, Ika Alfina, Avner Algom, Erik Andersen, Lene Antonsen, Katya Aplonova, Angelina Aquino, Carolina Aragon. Glyd Aranes, Maria Jesus Aranzabe, Bilge Nas Arica,, Pórunn Amardóttir, Gashaw Arutie, Jes- sica Naraiswari Arwidarasti, Masayuki Asahara, Deniz Baran Aslan, Cengiz Asmazoglu, Luma
Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020. Generalizing from few examples: A survey on few-shot learning. ACM Comput. Swre, 53(3).

---

Ateyah, Furkan Atmaca, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Keerthana Balasubramani, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, Starkaður Barkarson, Rodolfo Basile, Victoria Basmov, Colin Batchelor, John Bauer, Seyyit Talha Bedir, Kepa Bengoetxea, Yifat Ben Moshe, Gözde Berk, Yevgeni Berzak, Irshad Ahmad Bhat, Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agnè Bielinskiené, Kristín Bjarnadóttir, Rogier Blokland, Victoria Bobicev, Loic Boizou, Emanuel Borges Völker, Carl Börstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Anouck Braggaar, Kristina Brokaité, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Lauren Cassidy, Tatiana Cavalcanti, Gülzen Cebiroglu Eryigit, Flavio Massimiliano Cecchini, Giuseppe G. A. Celano, Slavomír Cépló, Neslihan Cesur, Savas Cetin, Özlem Çetinoglu, Fabricio Chalub, Shweta Chauhan, Ethan Chi. Taishi Chika, Yongseok Cho, Jinho Choi, Jayeol Chun, Juyeon Chung. Alessandra T. Cignarella, Silvie Cinková, Aurélie Collomb, Çağn Çöltekin. Miriam Connor, Daniela Corhetta, Marine Courtin, Mihaela Cristescu, Philemon Daniel, Elizabeth Davidson, Mathieu Dehouck, Martina de Laurentiis. Marie- -atherCa de Marneffe, Valeria de Paiva, Mehmet Oguz Derin, Elvis de Souza, Arantza Diaz de Ilaraaa, Carly Dickerson, Arawinda Dinakaramani, Elisa Di Nuovo, Bamba Dione, Peter Dirix, Kaja Dobrovolje, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Hanne Eckhoff, Sandra Eiche. Marhaba Eli, Ali Elkabky. Binyam Ephrem, Olga Erina, Tomaž Brjavec, Aline Etienne, Wograine Evelyn, Sidney Facundes, Richárd Farkas. Federica Favero, Jannatul Fer- daousi, Marília Fernanda, Hector Fernandez. Alcalde, Jennifer Foster, Cláudia Freitas, Kazunori Fujita, Katarína Gajdošová, Daniel Galbraith. Federica Gamba, Marcos Garcia, Moa Gärdenfors, Sebastian Garza, Fabrício Ferraz Gerardi, Kim Gerdes, Filip Ginter, Gustavo Godoy. lakes Goenaga. Koldo Gojenola, Memduh Gökımak, Yoav Goldberg, Xavier Gómez Guinovart, Berta González Saavedra, Bernadeta Griciuté, Matias Grioni. Loic Grobol. Normunds Grüzitis, Bruno Guillaume, Céline Guillot-Barbanee Tunga Güngör, Nizar Habash, Hinrik Hafsteinsson, Jan Hajič, Jan Hajië jr.. Mika Hämäläinen, Linh Ha Mý, Na-Rae Han, Muhammad Yudistira Hanifmuti, Takahiro Harada, Sam Hardwick, Kim Harris, Dag Haug. Johannes Heinecke, Oliver Hellwig, Felix Hennig. Barbora Hladká, Jaroslava Hlaváčová, Jaroslava, Florinel Hociung, Petter Hohle, Jena Hwang, Takumi Ikeda, Anton Karl Ingason. Radu Ion, Elena Irimia. Olájidé Ishola, Kaoru Ito, Siratun Jannat, Tomáx Jelínek, Apoorva Jha, Anders Johannsen, Hildur Jónsdóttir, Fredrik Jørgensen, Markus Juutinen, Sarveswaran K, Hüner Kaşıkara, Andre Kaasen, Nadezhda Kabaeva, Sylvain Kahane, Hiroshi Kanayama, Jenna Kanerva, Neslihan Kara, Ritván Karahóga, Boris Katz, Tolga Kayadelen, Jessica Kenney, Václava Kettnerová, Jesse Kirchner, Elena Klementieva, Elena Klyachko, Ame Köhn, Abdullatif Köksal,
Kamil Kopacewiez. Timo Korkiakangas. Mehmet Köse, Natalia Kotsyba. Jolanta Kovalevskaite, Simon Krek, Parameswari Krishnamurthy, Sandra Kübler, Oguzhan Kuyrukęu, Ash Kuzgun, Sookyoung Kwak, Veronika Laippala, Lucia Lam, Lorenzo Lambertine, Tatiana Lando, Septina Dian Larasati, Alexei Lavrentiev, John Lee, Phuong Lå Hång, Alessandro Lenci, Saran Lertpradit, Herman Leung, Maria Levina, Cheuk Ying Li. Josie Li, Keying Li, Yuan Li, Kyung Tae Lim, Bruna Lima Padovani. Krister Lindén, Nikola Ljubešié, Olga Loginova, Stefano Lusito, Andry Luthfi, Mikko Luukko, Olga Lyashevskaya, Terrs Lynn, Vivien Macketanz, Menel Mahamdi, Jean Maillard, Aihek Makazhanov, Michael Mandl, Christopher Manning, Ruli Ma- nurung. Büşra Marşan, Cätälina Märänduc, David Mareček, Katrin Marheinecke. Stella Markantonatou, Héctor Martinez Alonso, Lorena Martín Rodríguez, André Martins, Jan Mašek, Hiroshi Matsuda, Yuji Matsumoto, Alessandro Mazzei, Ryan McDonald, Sarah McGuinness. Gustavo Mendonça, Tatiana Merzhevich, Niko Miekka, Karina Mischenkova, Margarita Misirpashayeva, Anna Missilä, Cätälin Mititelu, Maria Mitrofan, Yusuke Miyao, AmirHos- sein Mojiri Foroushani, Judit Molnár, Amirsaeid Moloodi, Simonetta Montemagni, Amir More, Laura Moreno Romero, Giovanni Moretti, Keiko Sophie Mori, Shinsuke Mori, Tomohiko Morioka, Shigeki Moro, Bjartur Mortensen, Bohdan Moskalevskyi, Kadri Muischnek, Robert Munro, Yugo Murawaki, Kaili Müürisep, Pinkey Nainwani, Mariam Nakhlé, Juan Ignacio Navarro Horñiacek, Anna Nedoluzhko, Gunta Lespore-Berzkalle Manuela Nevaci, Luong Nguyēn Thi, Huyên Nguyên Thj Minh, Yoshihiro Nikaido, Vitaly Nikolaev, Rattima Nitisaroj, Alireza Nourian, Hanna Nurmi, Stina Ojala, Atul Kr. Ojha, Adédayo Olúdkun, Mai Omura. Emeka Onwueg- buzia, Noam Ordan, Petya Osenova, Robert Östling, Lilja Øvrelid, Şaziye Betül Özates, Merve Özşelik, Arzucan Özgür. Balkiz Öztürk Başaran, Terrsa Paccosi, Alessio Palmero Aprosio. Hyunji Hayley Park, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Guilherme Paulino- Passos. Giu- lia Pedonese, Angelika LejjjkkLapiiiska,, Siyao Peng, Cenel-Augusto Perez. Natalia Perkova, Guy Perrier, Slav Petrov, Daria Petrova, Andrea Peverelli, Jason Phelan, Jussi Pitulainen, Tommi A Pirinen, Emily Pitler, Barbara Plank, Thierry Poibeau, Larisa Pono- mareva, Martin Popel, Lauma Pretkalnipa, Sophie Prévost, Prokopis Prokopidis, Adam Przepiórkowski, Tiina Puolakainen, Sampo Pyysalo, Peng Qi. Andriela Raäbis, Alexandre Rademaker, Mizanur Rahoman, Taraka Rama, Loganathan Ramasamy, Carlos Ramisch, Fam Rashel, Mohammad Sadegh Rasooli, Vinit Ravishankar, Livy Real, Petru Rebeja, Siva Reddy, Mathilde Regnault, Georg Rehm, Ivan Riahov, Michael RieBler, Erika Rimkuté, Larissa Rinaldi. Laura Rituma, Putri Rizqiyah, Luisa Rocha, Eirikur Rögnvaldsson, Mykhailo Romanenko, Rudolf Rosa, Valentin Rosca, Davide Rovati, Ben Rozonoyer, Olga Rudina, Jack Rueter, Kristján Rúnnarsson, Shoval Sadde, Pegah Safari, Benott Sagot, Aleksi Sahala, Shadi Saleh, Alessio Salomoni, Tanja

---

Samardžić, Stephanie Samson, Manuela Sanguinetti, Ezgi Sanyyar, Dage Särg, Baiba Saufite, Yanin Sawanakunanon, Shefali Saxena, Kevin Scannell, Salvatore Scarlata, Nathan Schneider, Sebastian Schuster, Lane Schwartz, Djamé Seddah, Wolfgang Seeker, Mojgan Seraji, Syeda Shahzadi, Mo Shen, Atsuko Shimada, Hiroyuki Shirasu, Yana Shishkina, Muh Shohibussiri, Dmitry Sichinava, Janine Siewert, Einar Freyr Sigurdsson, Aline Silveira, Natalia Sil- veira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Simková, Kiril Simov, Maria Skachedubova, Aaron Smith, Isabela Soares-Bastos, Shafi Sourov, Carolyn Spadine, Rachele Sprugnoli, Vivian Stamou, Steinhór Steingrímsson, Antonio Stella, Milan Straka, Emmett Strickland, Jana Strnadová, Alane Suhr, Yogi Lesmana Sulestio, Umut Sulubacak, Shingo Suzuki, Daniel Swanson. Zsolt Szántó, Chihiro Taguchi, Dima Taji, Yuta Takahashi, Fabio Tamburini, Mary Ann C. Tan, Takaaki Tanaka, Dipta Tanaya, Mirko Tavoni. Samson Tella, Isabelle Tellier, Marinella Testori, Guillaume Thomas, Sara Tonelli, Liisi Torga, Marsida Toska, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Utku Türk, Francis Tyers, Sumire Uematsu, Roman Untilov, Zdeňka Urešová, Larraitz Uria, Hans Uszkoreit, Andrius Utka, Filena Vagnoni, Sowmya Vajjala, Rob van der Goot, Martine Vanhove, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Uliana Vedenina, Eric Villemonte de la Clergerie, Veronika Vineze. Natalia Vlasova, Aya Wakasa, Joel C. Wallenberg, Lars Wallin, Abigail Walsh, Jing Xian Wang, Jonathan North Washington, Maximilan Wendt, Paul Widmer, Shira Wigderson, Sri Hartati Wijono. Seyi Williams, Mats Wirén, Christian Wittern, Tsegay Woldemariam, Tak-sum Wong, Alina Wróblewska, Mary Yako, Kayo Yamashita, Naoki Yamazaki, Chunxiao Yan. Koichi Yasuoka, Marat M. Yavrumyan, Arife Betüil Yenice, Olcay Taner Yildiz, Zhuoran Yu, Arlisa Yuliawati, Zdeněk Zabokrtsky, Shorouq Zahra, Amir Zeldes, He Zhou, Hanzhi Zhu, Anna Zhuravleva, and Rayan Ziane, 2022. Universal dependencies 2.10. LINDAT/CLARIAH-CC digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles Universily.
XT1
IOUNT Layer
Figure 6: Accuracy in identifying word classes of different languages across different layers of mBERT.
produces contextual representations h for each of the token ws (i = I,....n). We take the representation hş corresponding to the word wf
as the feature of its word class. The feature of the grammatical relation between wead-depeddent pair of words (whead dep) is given by the difference between their representations:
e (he)
=hhead
hep
(4)
akin to previous works (Hewitt and Manning, 2019;
Chi et al., 2020; Xu et al., 2022). The features and corresponding labels constitute the datasets D = [x, 3/ }= whereby we derive each structural concept k and measure the alignablity between languages (Section 2.1).
Validation of method The middle layers of BERT-like models have shown most effective in encoding syntactic information (Hewitt and Manning. 2019; Chi et al., 2020). We validate this applies to our setting through probing the different layers of mBERT. Besides, to ensure that our method reflects the information about structural concepts encoded in the representation space, we compare the performance in classification with the following baselines: 1) LAYERO, the Oth layer of BBERT, where no contextual information is given; and 2) RAND, a model shares the same architecture as mBERT with its weights randomized For these experiments, we set the maximum rank of the probe model to 768 to maximally extract relevant information encoded in the representations.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR.
A Additional Materials for Correspondence between Structural Concepts in Transformers
Results The 7th and 8th layers of the model are most effective in encoding the grammatical relations (Figure 7). For word classes, performance is relatively consistent from the 3rd to 9th layer except for some low-resource languages like
A.1 Identification of Structural Concepts Based Prototypes
As the performance of RAND is approximately equal across different layers, we consistently select the 7" layer for our analysis.
Representations of structural concepts Given an input sequence l n tokens wiz an LLM

---

2.5-
Layer7
LayerO
Rand
HE NOMNNO
Denstty
1.5
1.0
0.5
0.0
Figure Accuracy in identifying grammatical relations of different languages across different layers of mBERT.
1.0
Accuracy
Figure 9: The distribution of the accuracy in deriving grammatical relations from the 7th layer mBERT, along with two baselines. The x-axis denotes the accuracy, of which the distribution is derived from the results 43 languages. The Wilcoxon test shows that the 7th layer exhibits significantly higher performance (W = 0.0, p 2.27 x 10-13)
Lays?
Layer
3.0
Rand
2.5
2.0
15-
10-
Word Classes, RSA, mBERT
0.5
0.0
1.0
0.8
POS Accuracy
Figure 8: The distribution of the accuracy in deriving word classes from the 7th layer of mBERT, along with baselines. The x-axis denotes the accuracy, of which the distribution is derived from the results in 43 languages. The Wilcoxon test shows exhibits significantly higher performance (W 0.0, p 2.27 x 10 13).
0.7
0.5
7 layer
02
Marathi (mr) and Tamil (ta) (Figure 6). We thus take the 7th layer for our experiments. The comparison with the two baselines (Figure 8 and Figure 9) supports the efficacy of our method in deriving prototypes of structural concepts while reflecting the geometry of LMs.
Figure 10: The alignability between word classes within different languages in mBERT measured by RSA.
Word Classes, PA, mBERT
Moreover, we also observe disparities between languages reflected in the classification of struc- tural concepts. As shown in Figure 6 and Figure the performance on oowrresuuce languages like Marathi and Tamil consistently lags behind, indicat- ing insufficient representation of these languages in LLMs.
1.0
07
A.2 Alignability between Structural Concepts in Different Languages
Details of evaluation We use RSA and Pro- crustes analysis (PA) to measure the alignability between structural concepts in different languages. The RSA between two languages is evaluated through the Spearman's 's rank correlation between
Figure II: The alignability between word classes within different languages in mBERT measured by Procrustes analysis.
chttps://doce /ttpp//cccsss......//////.//./pp wwwpy.crg//aae.eeooooooooo...i org//boc/scipy//eferrntt smmerated/scipy.spateee wwt.spilllppccrustees.. prwcrustes..tteee html

---

Grammation atical Relations, RSA, MBERT
Grammatical Relations, RSA, LLaMA
1,0
0.6
0.8
0.7
0.8
0.5
0.3
0.2
02
Figure 12: The alignability between grammatical rela- tions within different languages mBERT measured RSA.
Figure 14: The alignability between grammatical rela- tions within different languages in LLaMA measured by RSA.
Grammatical Relations, PA, mBERT
Grammatical Relations, PA, LLaMA
1.0
0.9
0.8
h
0.
0.5
0.5
0.3
0.2
02
Figure 13: The alignability between grammatical rela- tions within different languages mBERT measured by Procrustes analysis.
Figure 15: The alignability between grammatical rela- tions within different languages in LLaMA measured by Procrustes analysis.
the lower diagonal portion of their dissimilarity matrices. The fitness of the linear transformation derived from PA is evaluated through the average proportion of explained voriance
random mappings per baseline. We employ the Wilcoxon test to assess whether the alignability between a language pair in LLMs, computed based on our method, significantly higher than these baselines.
Details of baselines structural concepts derived from two languages L1 and Lg, we construct the following three baselines: (i) RP, which randomly swaps each prototypes for another in one of the two languages; (ii) RC, where randomly select sample of each concept instead of their prototypes in language; (iii) RS, where we randomly select K samples in one language. Each baseline thus creates a different mapping between two languages, and we test 100
Given the prototypes for K
lan- guages in mBERT with regard to word classes is shown in Figure 10 (RSA) and Figure 11 (PA). Figure 12 (RSA) and Figure 13 (PA) show the results for grammatical relations in mBERT. In terms of LLaMA, the results for word classes shown in Section 2.3, and the results for grammatical relations are depicted in Figure 14 (RSA) and Figure 15 (PA). The alignability mBERT and LLaMA both significantly higher than the baselines, with < 0.001 for almost all
Results The alignability between
"https://scikittleer wtteeeaannoor//eeeeee /tt.a///nlrrr//ttalm/mmmmllllll generated/skkeamm http://s..aaraammmiii....c r2_score__e_eeee html.

---

0.
PXY: A. B, C.D.EEE.............. O.P.Q.
08
0.7
0.6
WORD: adjective, adposition, adverb, aux- iliary, poaddinating_conjueetiiin determiner, interjection, noun, numeral, particle, pro- noun, proper_ noun, punctuation, subordinat- mng_conjunction symbol, verb, other.
0.5
Language
304
Nethod
Meta
Figure Accuracy M28 in identifying grammatical relations in languages used during meta-training (depicted by bluish lines) and novel languages encountered during meta-testing (depicted by reddish lines) given N sentences.
Languages for meta-training We employ five languages for meta-training, including bg, en, fi, fr, ru. The other languages and datasets used in our experiments are shown in Appendix
C.2 Full Results
language pairs. The only exception arises when comparing the alignability with RC, where the samples are randomly taken from the same class (Table 3).
Figure 17 presents the full results for our meta- learning-based methods across 24 languages.
D Implementation Details
Large language models We use Multilingual and LLaMA-7B (Llama-7b-hf) for all our experi- ments.
B Additional Materials for Aligning Conceptual Correspondence for Cross-Lingual Generalization
BERT
For POS tagging, Table shows our results on 30 low-resource languages compared with UDAPTER. Results on other languages are shown in Table The results for the classification of grammatical relations are shown in Figure 16, Table 8 and Table 9.
Deriving structural concepts from LLMs For all our experiments that derive structural concepts from LLMs through linear transformation, we train the linear probe with batch size of 8 and max sequence length of 128 for 20 epochs, and validate at the end of each epoch. We select the model performing the best on the development set. We use the Adam optimizer with 81 = 0.9, B2 0.999, and weight decay of 1 x 10-6, The learning rate is set to 1 10-4
Additionally, we present the average accuracy and standard deviation for both low-resource and high-resource languages, as shown in Table 4 and Table 5. With an increasing number of available examples, our approach demonstrates consistent improvements and helps mitigate the performance gap across diverse languages.
Learning to align conceptual correspondence Our meta-learning-basee method follows the pro- cedure described above to derive prototypes for each concept. Subsequently, the networks are trained for 50 epochs, with maximum sequence length of 128. During meta-training, given m languages, each epoch consists m 50 training episodes. These episodes are constructed using labeled sentences as the support set and 30 labeled sentences as the query set. The method we use here bears resemblance to Li et al. (2022b). The parameters of the network are optimized through the Adam optimizer, with 31 0.9, B2 0.999, and weight decay of 1 x 10-4 4 The learning
C Additional Materials for Aligning Conceptual Correspondence during In-Context Learning
C.1 Experimental Details
Label forms The label forms used in our experi-
ments are as follows:
UPOS: ADJ, ADP, ADV, AUX, CCONJ, DET, INTJ, NOUN, NUM, PART, PRON, PROPN, PUNCT, SCONJ, SYM, VERB, X.
SHFI. PUNCT, DET, AUX, ADJ, PRON, X, PART, CCONJ, INTJ, NUM, SCONJ, ADV, SYM, VERB, PROPN, ADP, NOUN.
"nttps://hggiingaacecmm bert-bbseemmlliiiingua. hhttps://hugginggeee co/ weo/decaaaaaaaaaeee research/ 1lama-7b-ht

---

mBERT L
Model L
Concept
Measure
Baseline
p-value
POS
RSA
RC
P == 0.040 < 0.05
mBERT ar
nl
POS
RSA
RC
p = 0.030 < 0.05
mBERT ko
de
POS
RSA
RC
- 0.009 < 0.01
mBERT ko de
POS
RSA
RC
= 0.009 < 0.01
mBERT mnr
POS
PA
RC
P = 0.130
mBERT Vi
POS
PA
RC
p 0.002 < 0.005
mBERT ko VI
POS
PA
RC
P 0.616
LLaMA bg
de
POS
PA
RC
p 0.027 < 0.05
LLaMA
bg
fr
POS
PA
RC
p 0.007 0.01
Table Exceptions where the alignability between the language pair exceeds significance threshold of < 0.001.
LR-AVG
LR-STD
HR-AVG
HR-STD
AVG
STD
UDAPTER
58.4
0,2135
97.0
0.0113
70.0
0.2516
M28-0
56.6
0.2118
90.8
0.0480
66.9
0.2379
M28-10
59.0
0.2075
89.2
0.0581
68.1
0.2245
M28-30
61.5
0.1874
90.7
0.0507
70.3
0,2080
M28-50
62.5
0.1834
91.0
0.0478
71.1
0.2032
M43-0
57.9
0.2280
90.9
0.0468
67.9
0.2446
M43-10
60.8
0,2111
89.6
0.0596
69.5
0,2229
M43-30
63.0
0.1942
90.6
0.0541
71.4
0.2079
M43-50
64.4
0.1869
91.1
0.0495
72.5
0.2003
Table The average accuracy (AVG) and standard deviation (STD) with regard to POS tagging measured across the languages used in UDAPTER, where the low-resource languages (LR) are the 30 languages listed in Table 6 and the high-resource languages (HR) include: ar, en, eu, fi, he, hi, it, ja, ko, ru, sv, tr and zh. AVG and STD are computed over all languages. While method lags behind UDA ATEPPTE in terms of high-resource languages, its demonstrates an increasing performance on low-resource ones when provided with additional examples. Moreover, the gap between different languages becomes smaller, especially M28, which does not involve any low-resource languages in meta-traie eetannraining.
rate is set to 5 x 10-5, The hidden layer dropout probability 0.33.
layer dropout probability 0.33.
Note that as the conceptual correspondence holds across languages for concepts with (Sec- tion 2.3) for concepts with more than (or equal to) 20 samples, concepts with fewer than 20 samples in either the source target languages are excluded during meta-training. During meta- testing, examples belonging to these categories are considered as misclassified.
E Data
The data used in all our experiments is from UD v2.10 (Zeman al., 2022)13 We follow the split of training, development and test set in it.
Data for measuring correspondence between structural concepts within different languages In terms of measuring the correspondence between structural concepts within different languages (Section 2), we use 28 languages for mBERT, including af, ar, bg, ca, cy. de, el, en, es, et, eu, fa, fi, fr, ga, he, hi, hu, hy, id, is, it, ja, ko, It, ly, mr, nl, no, pl, pt, ro, ru, sk, sr, sv, ta, te, tr, uk, ur, vi and zh. For LLaMA, we test on 20 languages: bg, ca, da. de, en, es, fr, hr, hu, nl. pl, pt. ro, ru, sl,
Aligning conceptual correspondence during in- context learning For meta-learning during in- context learning, our networks are trained for 100 epochs with each consists of x 10 episodes, where 5 is the number of languages involved in training. The parameters of the network are optimized through the Adam optimizer, with 31 = 0.9, 32 = 0.999, and weight decay of x 10-4 The learning rate set to 5 x 10 The hidden
"httpss//linda" mff, wfff.cun..cm www.ccin...ccccooooce wtt...ccc//pppsttorr/mm.lll/ hamdle/11224/1-111

---

LR-AVG
LR-STD
HR-AVG
HR-STD
AVG
STD
M28-0
53.1
0.1870
84.7
0.0525
62.6
0.2151
M28-10
57.6
0.1648
83.2
0.0667
65.3
0.1848
M28-30
59.9
0.1609
84.9
0.0610
67.4
0.1802
M28-50
60.8
0.1597
85.6
0.0564
68.3
0.1780
M43-0
52.5
0,2080
81.4
0,0717
61.3
0.2222
M43-10
58.6
0.1739
82.6
0.0741
65.8
0.1869
M43-30
60.5
0.1681
84.6
0.0622
67.8
0.1822
M43-50
61.7
0.1662
85.7
0.0559
68.9
0.1799
Table 5: The average accuracy (AVG) and standard deviation (STD) with regard to the classification of grammatical relations measured across different languages, where the low-resource languages (LR) are the 30 languages listed in Table 6 and the high-resource languages (HR) include:
tr and zh. AVG fi, he, hi, it, ja, ko, and STD are computed over all languages. Our method demonstrates an increasing performance on low-resource ones when provided with additional examples. Morcover, the gap between different languages becomes smaller, especially for M28, which does not involve any low-resource languages in meta-training.
1.0
Accuracy
0.5
0.0
bg'
ca
da
ent
e5
h1"
hr
11
1.0
Accuracy
EN
0.5
Mono
Meta
0.0
ja
n1
pl
pt
rut 51
Sr
SV
tr²
zh
Language
Figure 17: The few-shot generalization performance on POS tagging in the monolingual (MONO) and cross-lingual (EN) in-context learning settings, with English the source language. META denotes our meta-learning-basee method, Error bars represent the standard deviation calculated from 10 runs. Languages marked with "4" are not included the pretraining corpus. "t" indicates that the language involved meta-training for META.
sv, uk and zh.
pt, ro, ru, sl, sr, sv, uk and zh, among which zh, fi, hi, ja, tr are not included in the pretraining corpus.
Data for aligning conceptual correspondence for cross-lingual generalization The languages involved in the meta-training of M28 are: ar, bg. ca, de, es, et, eu, fa, fi, fr, he, hi, is, it, ja, ko, Iv. nl, no, pl, pt, ru, sk, sv, tr, uk and zh. For M43, the 15 additional languages used for meta-training include: af, сy, el, en, ga, hu, hy, id, It, mr, st, ta. te, ur and vi. Both are evaluated 72 languages, which involves the 43 languages for meta-training and another 29 languages: aii, akk, am, be, bho, bm, br, bxr, cs, da, fo, gsw, gun, hsb, kk, kmr, koi, kpv, krl, mdf, myv, olo, pem, sa, sl, tl, wbp, yo and yuc.
Detailed language information Table 10 and Table provide detailed information on the languages utilized in our experiments, including their respective language codes, sizes of the UD datasets, and language families.
total of
Data for aligning conceptual correspondence during in-context learning The experiments with in-context learning encompass 24 languages, including bg, ca, da, de, en, es, fr, hr, hu, it, nl, pl,

---

aii"
akk"
am'
be
bho"
hm"
br"
bxr*
cy!
fo
gsw*
gun"
hsb"
kk
kmr
UDAPTER
14.8
20.4
10.9
96.9
63.1
35.8
72.2
65.6
69.7
79.6
65.9
36.3
78.8
83.4
48.8
M28-0
14.6
21.9
20.9
90.3
62.5
32.7
75.4
60.1
69.9
84.6
64.7
12.9
73.5
77.2
42,4
10
22.1
30.0
14.9
89.3
63.3
35.7
79.4
60.5
73.4
85.1
68.8
14.9
75.3
79.0
48.1
M28-30
24.5
34.1
23.8
91.0
64.2
37.9
81.3
59.7
75.2
85.6
70.5
31.6
76.0
79.8
49.2
M28-50
24.3
36.5
29.7
90.7
64.6
40.7
81.1
60.0
76.9
86.3
70.6
29.0
75.8
79.8
48.6
M28-100
44.4
91.8
43.8
82.7
59.6
77.6
33.9
75.8
79.6
M28-200
49.0
92.6
66.2
48.9
83.4
59.8
78.6
87.0
36.5
75.8
79.6
49.1
M43-0
19.0
21.0
7.5
90.3
62.0
33.6
73.9
59.5
88.5
83.8
63.4
16.6
73.9
77.7
43.1
M43-10
30.9
28.1
14.1
89.5
62.4
36.7
80.3
59.8
87.3
84.9
69.0
17.7
74.6
80.2
46.1
M43-30
27.8
35.0
32.0
91.1
63.7
40.0
81.2
59.4
88.1
85.2
70.0
24.8
75.6
80.2
47.9
M43-50
26.9
37.6
41.7
91.6
64.8
42.2
82.3
59.2
88.9
85.7
72.2
29.8
75.5
80.3
48.5
M43-100
39.5
91.9
65.2
45.2
83.1
59.6
89.4
86.5
32.4
75.5
80.1
47.0
M43-200
44.0
92.7
65.7
49.8
83.9
59.5
89.6
86.7
34.1
75.5
80.1
48.1
koi'
kpv*
krl"
mdF*
mr!
myv*
olo"
pem
ta
te
wbp"
yo
yue
UDAPTER
48.9
36.7
78.3
54.7
66.5
52.8
76.6
54.7
42.2
70.3
84.2
78.4
34.1
63.7
66.3
M28-0
44.8
36.8
72.7
50.4
79.3
51.3
37.7
45.2
73.1
73.8
M28-10
46.6
39.3
73.1
51.1
80.1
51.6
66.3
43.8
49.4
77.5
80.2
80.5
54.6
63.1
72.8
M28-30
48.8
41.1
75.0
52.1
80.3
51.9
66.0
55.5
50.8
78.0
81.4
82.0
59.2
64.9
73.8
M28-50
51.0
41.7
76.7
51.7
79.0
52.7
66.0
64.3
52.0
79.5
80.3
83.7
58.8
67.0
75.3
M28-100
43.1
76.5
52.4
81.6
52.6
66.3
71.3
51.3
79.5
82.1
83.3
68.0
75.9
M28-200
45.8
79.5
52.5
81.6
53.7
66.3
76.3
55.2
79.7
70.7
M43-0
47.4
36.6
71.8
49.9
87.5
51.3
65.7
46.0
81.3
90.4
70.0
49.3
58.0
70.4
M43-10
51.9
41.7
72.5
50.3
86.7
51.5
65.9
54.5
48.7
81.9
90.0
78.6
53.9
62.2
72.0
M43-30
53.7
42.6
75.5
52.6
88.6
51.9
65.2
65.7
48.9
81.2
89.2
82.0
54.2
63.9
74.4
M43-50
53.2
42.8
76.0
52.2
88.6
51.6
65.0
70.6
49.8
81.5
90.0
82.8
58.8
66.7
75.2
M43-100
43.2
76.3
52.9
88.8
52.4
65.2
74.2
51.1
81.4
90.7
83.3
67.6
76.6
M43-200
44.4
78.7
53.7
88.8
53.7
77.2
54.0
71.2
Table 6: The zero-shot and few-shot generalization performance on POS tagging for 30 low-resource languages. Languages marked with "*" are included the pretraining corpus. "t" indicates that the language is involved in meta-training for M43. The UD dataset for some languages comprises less than 100 or 200 sentences, and the sho- performance for these languages left unspecified.

---

af
bg
cs!
da*
de
el
en
fa
fi
fr
UDAPTER
96.8
97.0
95.7
97.3
M28-0
89.8
93.0
96.2 95.8
92.1
89.5
93.3
88.4
84.7
94.4
91.0
90.0
92.8
92.0
95.4
M28-10
91.3
89.5
96.2 96.0
92.6
91.1
92.9
90.1
86.2
93.9
91.0
88.7
90.5
91.5
95.7
M28-30
91.6
92.5
96.3 96.5
93.4
91.3
92.9
91.2
88.2
94.5
91.5
89.4
91.5
92.2
95.9
M28-50
91.8
93.5
96.6
96.7
94.0
92.4
93.3
91.6
87.8
94.7
91.7
90.0
93.1
92.1
96.1
M28-100
92.8
93.8
96.7
96.9
94.3
92.8
93.4
91.9
89.0
94.9
92.0
90.2
93.2
92.5
96.1
M28-200
93.6
93.8
96.8
97.1
94.5
92.9
93.8
92.7
89.4
95.1
92.2
90.4
93.4
92.5
96.2
M43-0
93.4
92.5
96.0
95.6
92.4
89.4
93.0
94.6
91.9
94.1
90.5
89.4
92.4
91.4
95.2
M43-10
93.2
89.1
95.5
95.8
93.2
90.6
93.0
94.0
91.0
94.2
91.0
89.1
91.3
91.1
95.4
M43-30
94.3
92.3
96.1
96.4
93.6
91.8
93.0
94.4
92.0
94.8
90.9
89.3
92.3
91.8
95.5
M43-50
95.0
92.8
96.3
96.5
93.9
92.3
93.4
95.0
92.5
94.5
91.4
89.6
92.4
91.8
95.7
M43-100
95.4
93.3
96.5
96.6
94.5
92.6
93.5
95.3
92.7
94.8
91.8
89.7
92.8
92.2
95.8
M43-200
95.7
93.3
96.8
97.1 94.6
92.9
93.7
95.6
93.0
95.2
92.0
90.0
93.2
92.1
96.1
ga¹
he
hi
hut
hy
id
is
it
ja
ko
lt
lv
nl
pl
UDAPTER
97.1
97.4
98.3
97.0
96.5
M28-0
71.4
93.0
92.4 84.0
83.1
82.1
94.2
96.2
93.5
78.8
84.9
90.7
93.2
93.6
94.4
M28-10
73.9
90.2
91.4
86.3
83.9
86.7
93.3
96.5
92.5
72.9
85.0
89.6
91.9
92.6
92.7
M28-30
75.2
91.9
92.0
88.0
84.6
87.6
93.7
96.7
93.0
76.9
86.1
90.5
92.7
93.3
93.6
M28-50
75.9
92.8
92.1
89.4
84.1
88.1
93.8
96.6
93.2
78.3
86.1
90.7
92.8
93.5
94.1
M28-100
76.9
93.2
92.9
89.6
85.3
89.2
94.5
96.8
93.8
79.6
86.6
91.0
93.7
94.0
94.9
M28-200
77.8
93.4
93.2
90.0
85.8
89.8
94.6
96.8
93.8
79.9
86.8
91.2
94.1
94.2
95.5
M43-0
86.8
92.5
92.2
91.6
88.8
91.1
93.4
96.1
93.0
77.8
89.3
90.2
92.1
93.2
94.1
M43-10
86.6
90.9 91.2
91.4
88.4
91.5
92.7
95.9
92.3
71.8
87.4
89.8
91.8
91.0
92.8
M43-30
86.9
91.8
91.8
92.0
89.1
91.5
93.1
96.3
93.0
75.0
89.1
90.7
93.0
92.7
93.6
M43-50
87.0
92.3
92.5
92.4
88.4
91.8
93.5
96.3
93.1
77.4
89.5
90.4
93.0
93.1
94.0
M43-100
87.5
92.9
92.8
92.2
89.3
92.1
93.8
96.6
93.2
78.6
90.2
90.6
93.8
93.6
94.7
M43-200
87.8
93.1
93.1
92.6
89.9
92.4
94.0
96.8
93.4
79.1
90.4
91.0
94.1
94.1
95.4
0о
ru
sk
s/
srl
SV
tr
uk
url
vil
zh
UDAPTER
98.9
98.4
95.1
95.1
M28-0
95.2
94.9
94.7
92.6
88.5
92.2
95.1
85.0
93.8 81.3
67.1
91.5
M28-10
95.0
94.3 93.0
91.6
90.5
94.3
95.1
82.9
92.9
82.8
69.4
89.6
M28-30
95.8
95.1 95.6
92.2
91.7
94.6
95.3
84.3
93.6
84.1
70.2
91.2
M28-50
96.1
95.1
95.0
93.4
92.0
95.2
95.2
84.6
93.8
84.2
71.7
91.7
M28-100
96.4
95.2
95.7
93.5
92.7
95.4
95.8
85.0
94.3
85.8
75.4
92.1
M28-200
96.4
95.6
95.9
94.0
93.1
95.8
95.8
84.9
94.6
86.9
78.7
92.4
M43-0
94.9
94.4
94.7
92.5
89.5
96.2
94.9
84.4
93.5
89.6
86.3 90.6
M43-10
94.5
94.3
94.4
91.2
90.7
95.8
95.3
83.7
93.2
88.4
79.3 89.2
M43-30
95.4
94.6
94.7
93.0
91.8
96.5
95.5
83.8
94.0
89.6
82.8 90.7
M43-50
95.7
94.9 95.6
94.1
92.7
96.4
95.2
83.8
93.7
89.7
85.3 91.1
M43-100
95.9
95.0
96.0
94.2
92.9
97.0
95.4
84.3
94.6
90.1
87.7
91.6
M43-200
96.1
95.3 96.1 94.4 93.4
97.1
95.7
84.7
94.9
90.2
88.5
91.9
Table The zero-shot and few-shot generalization performance on POS tagging
other languages. By default, these languages are used for meta-training. Languages marked with "4" is involved in meta-training for M43 but not for M28. "4" indicates that the language excluded from meta-training for both M28 and M43.

---

aii
akk"
am'
be
bho"
bm"
br"
bxr"
cy²
fo"
gsw"
gun"
hsb"
kk
kmr
M28-0
33.3
20.8
18.2
86.6
53.2
28.7
73.4
35.6
69.0
84.5
66.2
20.9
67.3
71.1
45.6
M28-10
43.7
29.3
47.7
86.2
58.3
37.4
77.3
36.2
71.4
85.3
68.2
27.1
68.2
72.1
48.7
M28-30
49.0
34.8
48.3
86.9
61.8
42.1
78.8
37.1
73.8
86.0
70.5
68.5
49.4
M28-50
49.7
39.9
47.8
87.3
63.3
46.0
79.5
36.9
75.1
85.9
71.7
34.0
68.3
72.8
49.9
M28-100
48.5
87.8
63.9
48.6
80.2
37.0
76.4
87.2
36.4
68.5
72.8
49.6
M28-200
48.4
88.3
65.7
53.7
81.4
37.0
77.1
87.5
37.5
68.5
72.8
49.6
M43-0
40.0
22.0
18.4
86.0
48.0
29.6
73.5
18.1
80.5
84.5
64.0
19.8
60.9
61.7
44.0
M43-10
46.8
28.4
46.7
87.0
55.6
37.3
77.7
36.9
78.9
85.5
69.1
26.6
67.8
72.5
49.0
M43-30
49.0
36.2
47.5
86.6
61.7
40.3
79.3
37.3
81.6
85.8
71.1
30.1
68.5
73.2
49.7
M43-50
51.2
40.0
45.9
87.5
62.9
44.0
79.9
37.3
81.9
86.2
72.6
34.7
68.4
73.2
49.7
M43-100
49.9
88.4
64.4
49.8
80.0
37.3
82.8
86.4
36.8
68.5
73.2
49.4
M43-200
50.2
88.5
65.1
53.8
80.6
37.2
83.1
87.3
38.0
68.4
73.2
49.4
koi
kpv
krl"
mdf"
mr!
myv
olo'
pem
sa"
te
wbp"
yo
yue*
M28-0
48.2
38.5
64.7
49.7
71.5
48.3
44.1
32,5
44.3
65.2
75.5
72.9
53.3
54.5
55.1
M28-10
51.2
40.3
65.7
52.4
76.6
49.3
44.4
35.1
47.2
70.8
74.2
77.8
63.1
59.1
63.7
M28-30
54.1
42.2
68.6
53.1
75.8
51.2
44.4
37.2
48.1
72.9
77.1
80.2
71.6
61.9
66.9
M28-50
54.7
42.7
70.3
52.8
76.1
51.7
44.3
37.4
45.7
75.0
79.1
81.5
73.9
63.1
68.5
M28-100
45.7
68.4
53.5
77.9
52.6
44.2
39.4
48.1
75.0
78.5
81.6
64.0
70.6
M28-200
47.0
69.5
54.2
76.9
53.7
44.2
41.6
51.7
76.0
78.5
66.2
72.3
M43-0
49.3
39.1
63.2
49.4
77.4
48.3
21.8
34.6
43.4
75.6
82.5
73.4
55.6
55.2
56.9
M43-10
52.9
40.3
66.0
51.0
78.7
48.8
43.5
37.2
47.1
75.5
81.7
79.6
66.0
59.6
63.2
M43-30
52.3
43.5
68.5
53.5
77.9
50.7
44.0
38.4
47.2
77.7
81.0
81.9
71.2
60.9
67.4
M43-50
55.2
43.5
69.4
53.3
80.3
51.4
44.0
38.8
47.6
77.1
82.9
82.0
76.1
63.3
69.2
M43-100
44.6
70.7
52.6
80.6
52.4
43.8
40.2
46.0
78.3
82.7
81.6
64.7
70.8
M43-200
45.7
70.8
54.7
81.6
54.0
44.0
42.0
50.0
78.7
83.6
66.9
71.8
Table 8: The zero-shot and few-shot generalization performance the classification of grammatical relations for 30 low-resource languages. Languages marked with "*" are not included the pretraining corpus. "t" indicates that the language is involved in meta- meta-training for M43. The UD dataset for some languages comprises less than 100 200 sentences, and the few-shot performance for these languages is left unspecified.

---

af
ar
bg
ca
cst
da
de
en
es
et
eu
fa
fr
M28-0
78.3
85.7
90.8
89.1
87.7
86.5
89.2
88.1
84.3
89.6
83.9
82.6
88.8
85.1
91.9
M28-10
81.1
84.1
88.7
89.2
88.4
86.4
89.4
91.0
86.6
89.3
83.5
81.4
86.8
84.0
92.2
M28-30
82.4
85.1
90.8
90.1
89.2
87.3
89.9
91.8
88.1
91.0
84.3
82.3
87.6
85.3
92.7
M28-50
83.0
86.2
91.2
90.7
89.4
88.2
90.2
92.2
87.5
91.9
84.5
83.0
89.1
85.4
93.0
M28-100
85.2
86.4
92.0
91.2
89.7
88.3
90.0
92.5
88.8
92.1
84.6
83.9
89.6
86.2
93.3
M28-200
86.0
86.6
92.3
91.9
90.1
88.6
90.2
92.9
89.7
92.5
85.2
84.3
90.2
86.5
93.4
M43-0
85.1
84.0
89.6
88.1
87.6
86.8
88.1
92.3
86.1
88.8
81.9
80.7
81.8
83.5
91.6
M43-10
84.9
84.0
89.4
89.1
88.6
87.3
88.9
92.5
88.8
89.3
82.1
81.0
86.6
83.4
92.0
M43-30
87.0
84.2
90.8
90.0
89.1
87.3
89.9
93.4
89.5
91.7
83.0
81.9
87.5
84.3
92.5
M43-50
87.4
85.7
90.8
90.5
89.2
87.4
90.0
93.8
90.4
91.8
83.4
82.8
88.6
85.4
92.7
M43-100
87.8
85.8
91.6
91.1
89.7
87.9
90.1
94.2
91.3
92.0
84.3
83.2
89.2
85.6
93.2
M43-200
88.3
86.3
92.1
91.6
90.1
88.1
90.2
94.4
91.5
92.3
84.7
83.8
89.8
86.0
93.3
gal
he
hi
hu
hy
id
it
ja
ko
It
lv
nl
pl
M28-0
64.6
83.0
86.7 80.5
77.9
74.0
81.9
93.0
85.4
72.7
79.1
84.0
89.4
90.5
88.6
M28-10
67.8
81.2
84.6 83.0
78.4
77.2
77.8
92.5
84.3
66.8
80.6
83.1
89.1
89.7
87.8
M28-30
70.1
83.4
86.7
84.4
79.2
78.7
81.2
93.7
86.8
69.9
81.3
84.3
89.9
90.6
89.0
M28-50
70.8
84.6
87.1
85.4
80.1
79.7
82.3
93.8
87.4
71.5
80.8
84.4
89.8
90.7
89.3
M28-100
71.4
85.8
87.7
86.2
80.1
81.0
83.7
94.0
89.0
74.9
82.0
85.3
90.5
91.5
90.0
M28-200
72.5
86.0
88.3
86.8
80.8
81.9
84.5
94.3
89.7
76.1
82.3
85.6
91.0
91.6
90.6
M43-0
75.8
80.0
78.9
84.1
80.0
79.1
78.0
92.7
76.3
65.4
80.9
82.8
87.6
88.8 87.9
M43-10
75.5
81.2
84.1
86.5
80.5
81.7
77.5
93.1
81.0
64.8
81.7
82.9
88.8
88.7 87.4
M43-30
77.3
83.2
86.8
87.3
81.5
83.4
80.5
93.3
85.7
69.5
81.9
84.0
89.8
90.8 89.0
M43-50
78.0
84.0
87.4
87.4
81.6
84.5
81.3
93.6
87.3
72.4
83.4
84.2
89.9
90.9 88.6
M43-100
78.0
85.1
87.9
88.0
82.5
85.1
83.0
94.1
87.8
74.3
84.1
85.3
90.4
91.3 89.8
M43-200
79.0
85.7
88.3
88.3
83.0
85.6
83.9
94.5
88.6
75.1
84.4
85.6
91.0
91.5 90.4
pt
sk
s1t
srl
uk
vi
2h
M28-0
89.1
88.8
91.2
90.7
87.5
88.4
91.0
78.4
88.6
75.3
59.5
81.4
M28-10
91.4
88.3
91.5
88.9
88.1
90.0
90.6
76.5
87.9
76.0
65.9
77.8
M28-30
93.0
89.2
92.2
91.0
89.4
90.1
91.3
77.9
88.3
77.5
68.4
81.6
M28-50
93.4
89.4
92.0
91.2
89.8
91.5
91.6
78.7
89.4
77.7
69.3
83.8
M28-100
93.7
90.4
92.1
92.4
90.5
91.8
91.9
80.0
89.9
80.0
70.7
85.2
M28-200
93.8
90.6
92.7
92.9
90.8
92.0
92.1
80.2
90.5
80.9
72.8
86.1
M43-0
87.6
87.6
90.4
90.4
87.1
90.7
89.7
75.0
87.6
73.0
68.6
76.2
M43-10
90.4
87.9
91.2
90.6
87.6
91.5
90.1
76.6
88.2
81.4
72.9
74.2
M43-30
92.3
89.2
92.0
91.2
89.9
92.2
90.9
77.1
89.5
82.6
75.4
81.7
M43-50
92.9
89.4
91.8
91.6
89.9
92.5
91.2
78.4
89.5
83.1
75.9
83.1
M43-100
93.4
89.9
92.5
93.0
90.0
93.1
91.8
78.9
90.2
83.4
77.4
84.5
M43-200
93.9
90.4
92.6
93.2
90.8
93.5
92.0
79.8
90.4
84.0
78.5
85.4
Table The zero-shot few generalization performance on classifying grammatical relations for other languages. By default, these languages are used for meta-training. Languages marked with "4" is involved in meta-training for M43 but not for M28, "g" indicates that the language is excluded from meta-training for both M28 and M43.

---

Language
Abbr.
Language Family
UD Treebanks
Train
Test
Arabic
PADT
6,075
680
Bulgarian
bg
IE, Palto-Slavvis
BTB
8,907
1,116
Catalan
ca
Romance
AnCora
13,123
1,846
German !
de
IF. Germanic
GSD
13,814
977
English
en Germanic
EWT
12,543
2,077
Spanisht
IE. Iomance
GSD
14,187
426
Estonian
Uralic.Finnic
EDT
24.632
3,214
Basque
eu Basque Indo-Iraann
BDT
5,396
1,799
Persian
PerDT
26,196
1,455
Finnish
fi Uralic.Finnic
TDT
12,217
1,555
French fr Romancee
GSD
14,449
416
Hebrew he Aito-Asiatic.Semin
HTB
5,241
491
Hindi
hi
IE. Indo-Iranian
HDTB
13,304
1,684
Icelandic
is
IE, Germanic
Modem
5,376
768
Italiant
it
IE, Romance
ISDT
13,121
482
Japanese
ja
Japonic
GSD
7,050
543
Korcan
ko
Koreanic
Kaist
23,010
2,287
Latvian
lv
IE. Baato-SSlavic
LVTB
12,521
2,325
Dutch
nl
IE. Germanii
Alpino
12,289
596
Norwegian
IF. Germanic
Nynorsk
14,174
1,511
Polish
pl
Balto-Slavic
PDB
17,722
2,215
Portuguese+
pt
Romance
GSD
9,615
1,200
Romanian
ro
IE Romance
RRT
8,043
729
Russian
IE. Balto-Slavic
GSD
3,850
601
Slovak
sk
IE. Bal.Balltolii Slavie
SNK
8,483
1,061
Swedish
IE. Germanic
Talbanken
4,303
1,219
Turkish
tr
Turkic.Oghuz
BOUN
7,803
979
Ukrainian
uk
IF. -llBooo--Slavkk
IU
5,496
892
Chinese (Mandarin) zh Sino- Aino-Tibetan.Sinitte
GSDSimp
3,997
500
Table 10: Languages and UD Treebanks involved in the meta-training of M28, which typically high-resource
languages. English is employed as the source language. Languages marked with "f" are included in the experiments in-context learning. The phylogenetic information obtained from Glottolog (Hammarström al., 2022). stands for the Buropen family.

---

Language
Abbr.
Language Family
UD Treebank
Train
Test
Afrikaans af
Germanii
AfriBooms
1,315
425
Assyrian"
aii
A Afro-Asiatic.Semiie
AS
0
57
Akkadian akk
Aro-Asiatic.Semiie
PISANDUB
0
101
Amharic am
A Aro-Asiatic.Semiiii
ATT
0
1,074
Belarusian be
IE, BaltooSSlavic
HSE
22,853
1,077
Bhojpuri' bho
IE. IndoIraannn
ВHTB
0
357
Bambara" bm
Mande. Western Mande
CRB
0
1,026
Breton
IE. IE,Celtic
KEB
0
888
Buryat* bxr
Mongolic-Khine Khitan. Mongolii IE. Balto-Slavic
BDT
908
Czech
PDT
68,495
10,148
Welsh
cy
IE, IE,Celtic
CCG
976
953
Danish
da
IE. Germanic
DDT
4,383
565
Greekt
IE. Greek
GDT
1,662
456
Faroese*
IE. Germanic
OFT
0
1,208
Irish
ga
E.CCllic
IDT
4,005
454
Swiss German"
gsw
IE, Germanic
UZH
0
100
Mbya Guarani
gun
Hupian.aawttiGGuaran
Thomas Dooley
0
98
1,046
Croatian
IE. Balto-Slavik
SET
6,914
1.136
Upper Sorbian"
hsb
IE, Balto-Slavvi
UFAL
23
623
Hungarian ft
hu
Uralic. Hungarinnn
Szeged
910
449
Armenian
hy I.Armmnicc
ArmTDP
1,974
277
Indonesian
Austronesian.Malee Malayo-Polynesian
GSD
4,482
557
Kazakh
kk Turkic.Kipchal Kipchak
KTB
31
1,047
Kurmanji*
kmr IndonooIranial
MG
20
734
Komi-Permyak
koi
Uralic.Permian Permian
UH
0
100
Komi- Domii-yrrin*
kpv
Uralic.Permian
Lattice
0
663
Karelian"
krl
Tralic.Finnile
KKPP
0
228
Lithuanian
It mdf mr
IE, Balto-SSllvik Uralic.Mordwe IE, Balto-Slavik Mrodic.Mordvin Uralic.Finni IE. Germanic IE. Indo-aaa
ALKSNIS
2,341
684
Moksha
JR
0
342
Marathi
UFAL
373
47
Erzya* myv
JR
0
1714
Livvi*
olo Naija* pem
KKPP
19
106
NSC
7,278
972
Sanskrit
sa
UFAL
0
230
Slovenian
IE. Balto-Slavii
SSJ
10,903
1,282
Serbian sr
IE, Balto-Slavik Dravidian.South
SET
3,328
520
Tamil ta
TTB
400
120
Telugu te
Dravidian.Soutt
MTG
1,051
146
Tagalog
tl
Austronesian. IF. Indo-aaannnn
TRG
0
128
Urdu ur
UDTB
4,043
535
Vietnamese vi
Austroasiatic. Vietic
VTB
1,400
800
Warlpiri wbp Pama-NYungan. Desert Nyungic
UFAL
0
55
Yoruba"
yo Atlantic-Congo. Volta-Congo
YTB
0
318
Cantonese
yue
Lino-TibetanSSinite
HK
0
1,004
Table 11:
The remaining languages and their UD treebanks that are employed in our experiments. Languages marked with "t" are involved in the meta-training M43, "*" indicates that language is not included in the pretraining corpus of mBERT. "z" shows that the language is included in the experiments of in-context learning. The phylogenetic information is obtained from Glottolog (Hammarström et al., 2022). IE stands for the Indo-European family.


Reviews:
[{"review_id": "OACwyhXjb5", "reviewer": "Reviewer_m4pp", "summary": "", "questions": "1. How about the performance of static word embedding for concept alignment as you did in section-2? See point 2 in Reason to Reject.\n2. Line 304-308: why do you think a language-specific linear mapping can achieve a similar effect that orthogonal transformation gets?\n3. Can you briefly describe the meta-learning setting in sec-3.1, e.g., what is the objective? Which meta-learning approach is used? During the meta-testing time, whether cross-lingual label signals are leveraged? (I guess so according to lines 327-329). \n\nI will be happy to raise my score if questions get answered convincingly.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments. \n\n**Question#1**: This point is intriguing and merits consideration. However, previous work (e.g., \\[1\\]) has shown that structural information like grammatical relations and word classes (POS) can be decoded from static word embeddings (produced by, e.g., skip-gram and C\\&W models) to some extent. Our results in Appendix A.1 Fig. 8 \\& 9 also show that structural information can be partly derived from the 0-th layer of mBERT even in the absence of contextual information, consistent with \\[2\\]. \n\nIn addition, we would like to clarify some issues raised in **the second point in Reasons To Reject**. \n\n- The baselines in Section 2 are constructed to validate the **significance of alignability** between structural concepts in different languages, which is akin to previous work in validating the significance of the results measured by methods like RSA and Procrustes analysis (e.g., \\[3, 4, 5\\]). \n\n- It is the correspondence between structural concepts in different languages that we aim to leverage for cross-lingual transfer. Given a source-target language pair and sufficient labeled data, we may directly learn a linear (orthogonal) transformation to align the conceptual correspondence, allowing classification of samples in the target language based on the prototypes in the source language. As the conceptual correspondence holds across languages, we rely on meta-learning to learn to align the conceptual correspondence in a few-shot manner, thereby generalizing to target languages with minimal labeled data. Our approach enables generalization to languages unseen during meta-training, particularly benefiting low-resource languages that are distant from high-resource ones.\n\n**Question#2**: Given the results from Procrustes analysis in Section 2, the prototypes in different languages can be aligned through an orthogonal transformation, which is indeed a linear transformation that preserves the lengths of vectors and angles between them. Through an orthogonal transformation that aligns the prototypes in a source and a target language, the samples in the target language can be classified according to their distances to the prototypes in the source language. We use a linear mapping to approximate the orthogonal transformation, akin to previous work that learns to align word embeddings in different languages \\[6, 7, 8, among others\\].\n\n**Question#3**: In Section 3.1, we propose a meta-learning-based method to learn to align the conceptual correspondence between languages, which effectively facilitates generalization to languages not encountered during meta-training. We first derive the prototypes in the source language through **labeled** sentences in its training dataset. During meta-training, a training episode consists of a support set with $N$ **labeled** sentences and a query set with $M$ **labeled** sentences in a certain (target) language. The details can be found in Appendix D Line 1294-1307. The language-agnostic function $f_{\\phi}$ is optimized over the entire training procedure, and the language-specific function $g_{\\alpha}$ is learned separately for different languages. During meta-testing, $f_{\\phi}$ is kept fixed and $g_{\\alpha}$ is trained from scratch using the available examples (i.e., a support set with $N$ **labeled** sentences in $L_{T}$) in the few-shot setting. The method here bears resemblance to \\[9\\]. We appreciate your comments and will incorporate the descriptions of the experimental setting into our manuscript to improve clarity.\n\nWe are open to discussing our paper further if there are any remaining questions or concerns.\n\n\\[1\\] Qian, P., Qiu, X., & Huang, X. (2016). Investigating Language Universal and Specific Properties in Word Embeddings. ACL.\n\n\\[2\\] Chi, E. A., Hewitt, J., & Manning, C. D. (2020). Finding Universal Grammatical Relations in Multilingual BERT. ACL.\n\n\\[3\\] Kriegeskorte, N., Mur, M., & Bandettini, P. A. (2008). Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 4.\n\n\\[4\\] Patel, R., & Pavlick, E. (2021). Mapping language models to grounded conceptual spaces. ICLR.\n\n\\[5\\] Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., & Søgaard, A. (2021). Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color. CoNLL.\n\n\\[6\\] Mikolov, T., Le, Q. V., & Sutskever, I. (2013). Exploiting similarities among languages for machine translation. arXiv:1309.4168.\n\n\\[7\\] Lample, G., Conneau, A., Ranzato, M. A., Denoyer, L., & Jégou, H. (2018). Word translation without parallel data. ICLR.\n\n\\[8\\] Schuster, T., Ram, O., Barzilay, R., & Globerson, A. (2019). Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing. NAACL-HLT.\n\n\\[9\\] Li, W. H., Liu, X., & Bilen, H. (2022). Cross-domain few-shot learning with task-specific adapters. CVPR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good writing skills, although a little bit unclear in some places.\n2. Sufficient experimental demonstration. A good effect on cross-lingual transfer is shown.", "reasons_to_reject": "1. The setting of meta-learning in sec-3.1 is not described clearly. \n\n2. Part of the baselines is not convincing to me: For the experiments in 2.2, they conduct 3 baselines by randomly choosing other concepts or samples. Can we look at concepts deduced from static word embeddings, which do not have any structural information like word2vec or fasttext. Then, let’s say how better alignability of structural concepts deduced from LLMs (contextual embedding) can be achieved compared to it. It may be impossible for classification but for alignments it might be fine. I am really curious about this, which is related to the main contribution of this paper as I see. If it is questionable, hard to say where the transfer exactly comes from.", "questions_to_authors": "1. How about the performance of static word embedding for concept alignment as you did in section-2? See point 2 in Reason to Reject.\n2. Line 304-308: why do you think a language-specific linear mapping can achieve a similar effect that orthogonal transformation gets?\n3. Can you briefly describe the meta-learning setting in sec-3.1, e.g., what is the objective? Which meta-learning approach is used? During the meta-testing time, whether cross-lingual label signals are leveraged? (I guess so according to lines 327-329). \n\nI will be happy to raise my score if questions get answered convincingly.", "paper_topic_and_main_contributions": "In this paper, they explicitly align conceptual correspondence (syntactic aspect) across languages to enhance cross-lingual transfer. They conduct experiments on 43 languages, where they show that: 1) a high degree of alignability for syntactic concepts can be deduced from LMs, 2) a proposed meta-learning-based method, targeting align concepts explicitly, can enhance cross-lingual transfer for downstream syntactic tasks, like word class and grammatical relations identification. 3) for in-context learning, the transfer can be achieved as well. The experiments are sufficient for their claim, although there are a little few places I think can be made clearer.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cTexKluOvt", "reviewer": "Reviewer_TY8m", "summary": "", "questions": "- how did you tune the values for h and m for the language-agnostic function? Given figure 2, it seems that even much lower values for `m` could work?\n- Given the proposed approach, how can we ensure if e.g. RSA is high due to model's internal multilingual alignment, or due to having the transformation matrix A which could strengthen the alignment?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments. Before answering the questions, we would like to elucidate some central points of our paper, which may improve the understanding of our following response.\n\n- In this paper, we propose to explicitly establish alignment between concepts of which there is **correspondence** across diverse languages, thereby facilitating cross-lingual generalization, particularly towards low-resource languages. Take spacial concepts like \"left,\" \"right,\" \"top,\" and \"bottom\" as an example. While these concepts are represented differently across languages, it is possible that the structures of relations between these concepts within different languages is alignable, providing knowledge that can be transferred among languages. We use the structural concepts defined in Universal Dependencies as a testbed for our analyses.\n\n- We show that the structural concepts represented by LLMs can be defined based on their prototypes. We further demonstrate that the relational structures of the concepts within different languages are highly alignable, which indicates that the correspondence between the concepts is captured by LLMs (Section 2). \n\n- As the correspondence between concepts holds across languages, we propose a meta-learning-based approach that learns to align the conceptual correspondence in a few-shot (/zero-shot) manner, facilitating generalization particularly to low-resource languages (Section 3). \n\nBelow are our answers to the questions.\n\n**Question#1**: The value of $m$ is determined based on the findings in Section 2.3. Specifically, $m = 32$ for word class identification and $64$ for grammatical relation, where the probe accuracy approaching its peak performance. $h$ is selected to be larger than $m$. We examine cases where $h \\in \\left\\\\{128, 256, 512 \\right\\\\}$ and observe little improvements by increasing $h$ beyond $256$.\n\n**Question#2**: Rather than assessing the LLMs' internal alignment, we use RSA to measure whether correspondence between structural concepts is encoded in LLMs. The matrix $A$ is a simple linear transformation exclusively trained to derive features of structural concepts from LLMs, which enables classification of samples according to their distances to prototypes for each concept. RSA is employed to evaluate the degree to which the relational structures of the prototypes within different languages are alignable. \n\nIn addition, we would like to clarify some issues raised in **Reasons To Reject**.\n\nRegarding the first point:\n\n- **Figure 5 and Figure 3** are meant to illustrate the performance gain achieved by our proposed meta-learning-based method. This shows that the structural concepts are effectively aligned through our method, and it effectively generalizes to languages unseen during meta-training, as mentioned in Section 4.3 Line 531-534 and Section 3.3 Line 381-386. \n\n- Based on typological information, **Udapter** effectively exploits cross-linguistic similarities to implicitly transfer knowledge between languages. Through adapter-based tuning, (which has shown to achieve comparable performance to fine-tuning methods,) it achieves SOTA performance on high-resource languages like English, as well as languages similar to high-resource ones. By contrast, we focus on the generalization to low-resource languages that are distant from high-resource ones. By explicitly leveraging conceptual correspondence between languages, our approach offers particular advantages in terms of these languages, while solely relying on representations derived from pretrained LLMs. The results support the efficacy of our approach (Line 363-371).\n\nWe appreciate your comments and will revise our manuscript concerning the figures and the table mentioned here to enhance clarity.\n\nRegarding the second point: \n\n- RSA and Procrustes analysis are two complementary methods with distinct metrics. Consequently, complete uniformity between the results is not anticipated. RSA is a non-parametric method to compare the geometry of two representation spaces. In our context, for two languages, RSA computes the rank correlation between the relational structure of the concepts within each language. By contrast, Procrustes analysis evaluates the potential for **linear** alignment between two spaces by finding the optimal orthogonal transformation that matches corresponding representations (of concepts) in the two spaces. It is possible that the relational structure within two spaces are similar but they cannot be aligned linearly. Experiment results derived from both methods suggest that the correspondence between structural concepts across languages is encoded in LLMs.\n\nWe are open to discussing our paper further if there are any remaining questions or concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow\n- The observation of syntactic alignment between structural concepts for multiLMs in different scales seem promising. There also has been a thorough study by the authors with two metrics over two models at different scales to demonstrate the (relative) alignment.\n- The proposed method can get competitive results to SoTA without updating the multiLM parameters, and especially has promising performance for low-resource languages. It's also shown that the proposed meta-learning approach is quite data efficient (compared to FT), and few samples in the target low-resource languages considerably improves generalization.", "reasons_to_reject": "- Some of the results in the paper doesn't seem to be a fair comparison. For instance, in Figure 5, Meta-training is seemingly performed on many languages, while `En` and `Mono` using only a single language in training data. Similar concern for Figure 3 as well. It's also evident than when a model like UDapter is studied in Table 1 (which has a more fair training data), the performance is even a bit better than the proposed meta-learning approach.  \n- Procrustes and RSA values in the figure 1 seems \"contradictory\" for the mBERT model. There's no explanation in the paper why this inconsistency occurs", "questions_to_authors": "- how did you tune the values for h and m for the language-agnostic function? Given figure 2, it seems that even much lower values for `m` could work?\n- Given the proposed approach, how can we ensure if e.g. RSA is high due to model's internal multilingual alignment, or due to having the transformation matrix A which could strengthen the alignment?", "paper_topic_and_main_contributions": "This paper explores the possibility to improve cross-lingual generalization by aligning structural concepts like word classes and grammatical relations across different languages. By setting a linear transformation (probe) on top of the multiLMs like mBERT and LLaMA, they show that there's alignment between structural concepts across different language.\nGiven the observed (weak) alignment, the authors try to strengthen the alignment by proposing a meta-learning based approach. The proposed meta-learning method seems data-efficient and is competitive to the SoTA (and even outperforms it for certain low-resource languages).\nFinally they analyse LLaMA performance for POS tagging in the in-context learning (ICL) setting, where they argue that although in-domain performance is quite high, the generalization to other languages is not high. Therefore they use the proposed meta-learning approach once more but in the ICL setting to help the model improve alignment of structural concepts across languages. Their method achieves considerable gains in generalization compared to the baseline ICL when the proposed method is applied.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "avZXQNWi5k", "reviewer": "Reviewer_McsH", "summary": "", "questions": "- A: I’m not sure whether LLaMA is a good model for multi-lingual analysis (most pre-training data is in English), maybe a multi-lingual LLM such as BLOOM might be more suitable?\n- B: It might be interesting to analyze the correlations between alignability and language distances, has this been considered in the experiments?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. Before answering the questions, we would like to elucidate some central points of our paper, which may improve the understanding of our following response.\n\n- This paper aims to improve cross-lingual generalization to truly low-resource languages that are distant from high-resource ones. This presents a big challenge because i) there lacks sufficient data to support learning and ii) the substantial cross-linguistic differences render it difficult to apply the knowledge acquired from high-resource languages to these low-resource ones.\n\n- To this end, rather than leveraging the cross-linguistic similarities captured by LLMs, we propose to explicitly establish alignment between concepts of which there is **correspondence** across diverse languages. Take spacial concepts like \"left,\" \"right,\" \"top,\" and \"bottom\" as an example. While these concepts are represented differently across languages, it is possible that the structure of relations between these concepts within different languages is alignable, providing knowledge that can be transferred among languages. \n\n- To investigate its feasibility, we present two approaches: the first is a method to measure the degree to which the correspondence between concepts in different languages is captured by LLMs (Section 2), and the second is a meta-learning-based approach that learns to align the conceptual correspondence between languages in a few-shot (/zero-shot) manner, facilitating cross-lingual generalization particularly towards low-resource languages (Section 3). We use the structural concepts defined in Universal Dependencies as a testbed for our analyses.\n\nBelow are our answers to the questions.\n\n**Question#1**: While the majority of LLaMA's pretraining data is in English, it involves Wikipedia dumps spanning 20 languages and exhibits capability of cross-lingual transfer (Section 4.3). Our analysis of its representations suggests that the conceptual correspondence between languages is captured by it (Section 2.3). Moreover, we also extend our analysis to languages not involved in its pretraining data for comparison (Section 2.3 \\& Section 4.3). The impact of distribution of languages in pretraining data is an interesting topic and we leave exploration of this for future work.\n\n**Question#2**: We have discussed factors potentially influencing the alignability of structural concepts in Section 2.4 Line 260-279. Our emphasis in this paper lies in the investigation of conceptual correspondence rather than similarities and differences between languages. The subtle differences between languages that impact the alignability between languages is an intriguing topic that warrants deeper future exploration.\n\nAdditionally, we would like to clarify some issues raised in **Reasons To Reject**.\n\nOur approaches rely solely on representations derived from LLMs, which makes them model-agnostic and readily adaptable to various models and concepts.\n\n- Regarding the first point, we use the structural concepts outlined in UD including word classes and grammatical relations as a testbed. Through this, we demonstrate the viability to improve generalization to truly low-resource languages by explicitly aligning conceptual correspondence between languages. Our results support the efficacy of our methods. It is intriguing to generalize our method to other contexts like different modalities and semantic facets of language, as discussed in Section 6 (Line 637-647). However, considering space limitations, exploring these broader contexts falls outside the scope of this paper. \n\n- Regarding the second point, our selection includes two extensively used Transformer-based models that are pretrained on multiple languages without explicit cross-lingual information. These two models are trained via distinct yet widely adopted pretraining objectives: masked language modeling and language modeling. The results of our experiments with the models support the efficacy of our approaches. We leave the exploration of alternative models for future work.\n\nWe are open to discussing our paper further if there are any remaining questions or concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work provides a detailed and comprehensive analysis on the cross-lingual alignments of structural concepts in pre-trained LMs and might inspire future work on better understanding of LMs.\n- The paper is well-structured and easy to follow.\n", "reasons_to_reject": "- The analysis is in some way limited; only two types of syntactic structures are included. It will be more interesting to see more analyses on semantic concepts as well as how the explicit alignments can help further downstream tasks.\n- Only two pre-trained LMs (mBERT and LLaMA) are analyzed; some more models should be included to demonstrate that the findings generalize well to different models.\n", "questions_to_authors": "- A: I’m not sure whether LLaMA is a good model for multi-lingual analysis (most pre-training data is in English), maybe a multi-lingual LLM such as BLOOM might be more suitable?\n- B: It might be interesting to analyze the correlations between alignability and language distances, has this been considered in the experiments?\n", "paper_topic_and_main_contributions": "This work investigate how structural concepts are encoded in pre-trained LMs, and whether and how to make them aligned across different languages. POS tags and grammatical relations are used as the concepts for analyzing. The authors first show that the structural concepts are implicitly aligned through vanilla pre-training. Moreover, a meta-learning-based approach is adopted to bring improvements through explicit alignments. Finally, the analysis is extended to the in-context learning setting, where aligning with demonstration examples brings further gains.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is consensus amongst the reviewers that the work is sound and well written. The majority of the reviewers agree that the experimental design and analysis is sufficiently comprehensive, and that all the claims made in the paper are well-supported. The general excitement is moderate, with the text of the reviews (and reasons to accept) not particular championing for the idea and its novelty itself. Overall, the positives of the paper outweigh the weaknesses which were not addressed in the rebuttal, and the paper has _good technical soundness_ and _ambivalent excitement_.

The following is a summary of the strengths, weaknesses and scores across the three reviews:

**Strengths:**

- Paper is well written (**m4pp**, **TY8m**, **McsH**)
- Experimental design is sufficient in backing the claims made in the paper (**m4pp**)
- Adaptation of proposed technique (meta-learning approach) to in-context learning is especially pertinent, and shows the efficacy of the proposed methodology in low-resource settings (**TY8m**)
- Analysis of cross-lingual alignment is comprehensive (**McsH**, **m4pp**)

**Weaknesses:**

- Baselines are limited and too simple (**m4pp**)
	- Rebuttal somewhat addresses this weakness
- Generalization of the findings in general is not sufficiently shown (**McsH**)
	- e.g. two models, one of which was not explicitly trained on non-english

**Scores in decreasing order of confidence:**

|      | Soundness | Excitement | Reproducibility | Confidence |
|------|-----------|------------|-----------------|------------|
| m4pp | 4         | 3          | 4               | 4          |
| McsH | 3         | 3          | 4               | 3          |
| TY8m | 3         | 4          | 4               | 2          |
Decision: Accept-Findings