Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
Authors: Jirui Qi
Raquel Fernández
Arianna Bisazza
Venue: EMNLP 2023 Main (2023)
Keywords: Model Consistency
Multilinguality
Knowledge Incorporation
Large-scale Pre-trained Language Model
Model Evaluation
Knowledge Probing

Abstract: Multilingual large-scale Pretrained Language Models (PLMs) have been shown to store considerable amounts of factual knowledge, but large variations are observed across languages. With the ultimate goal of ensuring that users with different language backgrounds obtain consistent feedback from the same model, we study the cross-lingual consistency (CLC) of factual knowledge in various multilingual PLMs.
To this end, we propose a Ranking-based Consistency (RankC) metric to evaluate knowledge consistency across languages independently from accuracy. Using this metric, we conduct an in-depth analysis of the determining factors for CLC, both at model level and at language-pair level. Among other results, we find that increasing model size leads to higher factual probing accuracy in most languages, but does not improve cross-lingual consistency. Finally, we conduct a case study on CLC when new factual associations are inserted in the PLMs via model editing. Results on a small sample of facts inserted in English reveal a clear pattern whereby the new piece of knowledge transfers only to languages with which English has a high RankC score. All code and data are released at https://github.com/Betswish/Cross-Lingual-Consistency.

Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
Jirui Qi 1 Raquel Fernández', Arianna Bisazza! Center for Language and Cognition, University of Groningen Institute for Logic, Language and Computation, University of Amsterdam qi.qi, a. ..bisazza)@rug nl raquel. fernandez@uva.n nl
Abstract
en:
The capital of People's Republic of Chins La capital de República Popular China Vón cûa Trung Quóc la
High
GLC Pairs
vi:
Multilingual large-scale Pretrained Language Models (PLMs) have been shown store considerable amounts of factual knowledge, but large variations are observed across lan- guages. With the ultimate goal of ensuring that users with different language backgrounds oh- tain consistent feedback from the model, we study the cross- mross-lingual consistency (CLC) of factual knowledge in various multilingual PLMs. To this end, we propose Ranking- based Consistency (RankC) metric to evaluate knowledge consistency across languages inde- pendently from accuracy. Using this metric, we conduct an in-depth analysis of the deter- mining factors for CLC, both model level and at language-pair level. Among other re- sults, we find that increasing model size leads higher factual probing accuracy in most lan- guages, but does not improve cross-lingual con- sistency. Finally, we conduct a case study on CLC when new factual associations inserted in the PLMs via model editing. Results on small sample of facts inserted in English reveal a clear pattern whereby the new piece of knowl- edge transfers only to languages with which English has high RankC score.!
Uisoss
bu:
Kina fövöross
et:
mputtbouuo Aaiih Argooporic Tnς Kivoç diai
on:
Beijing
Same Correct
(high CLC):
Pekin (Beijing) of
PLM
(high CLC):
Bác Kinh(Beijnn) of
Carisle
(Carlele) X
Bipon
(Bamna)
en: The original language of The Godfather
}
High
es:
idioms original EI padrino es Ngön ngü gōe cua Bö già
CLC
vi:
Pairs
Lisons
hu: A Kereszaae eredeli ryelre apxish yλúoor You O Novóç cival
en:
Balian
Same
PLM
68
vi
(high CLC):
(high CLC):
finng Y
Htlliano
(Italian)
(Italian)
X
Wrong
hu
(low CLO)
orosz
(Russian)X X
(low CLC)
(Engish)
Figure I: Motivating example: Some languages share consistent knowledge in the multilingual PLM BLOOM- 3b, while others do not.
the extent to which factual knowledge generalizes across languages, revealing yet another facet of language inequality in modern NLP technologies (Hupkes et al., 2022).
Assessing factual knowledge across languages, however, is not trivial endeavor. Ensuring com- parability of the results requires that single set of 'universal' facts is queried in all languages, but the choice of that set likely to be biased to specific world regions that are more represented in popular knowledge bases like Wikidata. Conversely, facts that are more relevant in other regions of the world (e.g., information about locations important per- sons in given region) less likely ocear in the benchmarks, which makes it hard interpret the results of such evaluations.
1 Introduction
Large-scale Pre-trained Language Models (PLMs) have demonstrated powerful capabilities in tasks where factual knowledge plays an important role (Roberts et al., 2020; Qin et al., 2022). While most previous work on probing factual knowledge in PLMs has focused on English (Davison et al., 2019; Bouraoui et al., 2020; Shin et al., 2020; Brown et al., 2020; Alghanmi et al., 2021; Peng et al., 2022), a few notable studies have extended the evaluation
In this work, we take different stance: instead of measuring the amount of factual knowledge en- coded by PLM in each language, we focus its consistency across languages. As illustrated in Fig- ure 1, the multilingual BLOOM-3b model (Scao
number of other languages (Jiang et al., 2020; Kassner al., 2021; Yin al., 2022). The results of these studies show a large variation in
All code and data released at https: //g /////giittuu wwt.thu..coo com/ Betswish/Cross- Liwggaassss--n-a---- Soaaaal-Conssstnncy
Pht //www/ wiw.wwiwiiii wikidata.or.

---

al., 2022) outputs consistently correct comple- tions of the first prompt when queried in English, Spanish, and Vietnamese, but not in Hungarian and Greek. The model also outputs consistent, though wrong, answers to the second query in English, Spanish, and Vietnamese (but not in Hungarian and Greek), suggesting the first three languages share relevant knowledge representations within the model.
present a case study with a state-of-the-art model editing technique based on neuron interpretability (Meng et al., 2022), providing preliminary evidence that CLC is predictive of whether a fact inserted in language X will transfer to language Y.
Related Work
Probing factual knowledge in PLMs Since first proposed by LAMA (Petroni et al., 2019), prompt- based probing has become the main technique to assess factual knowledge in PLMs (Davison et al., 2019; Bouraoui et al., 2020; Shin et al., 2020; Brown et al., 2020; Alghanmi et al.. 2021; Peng et al., 2022). Given the knowledge represented in a tuple (subject, relation, object), query q is formed by filling the subject into relation-specifie tem- plate, which is fed into the PLM. If the prediction in line with the object, the model is considered to possess this knowledge. For instance, given a can- didate set of city names, when queried with 'The capital of People's 's Republic of China is _', the PLM is considered to capture this piece of knowl- edge if it gives the highest probability to the correct answer 'Beijing,, among all candidates.
The study of cross-lingudl consisteney (CLC) is important for at least two reasons: Firstly, true knowledge of a fact implies encoding of its mean- ing regardless of a given surface form (Ohmer et al., 2023). Thus, if model knows that the city of Bei- jing is the capital of China, it should return the same answer when asked the same question in dif- ferent languages. From a practical point of view, CLC is crucial to ensure users have similar ex- perience when interacting with the same model in different languages. Secondly, studying CLC im- portant understand whether and how knowledge acquired in a language gets implicitly transferred to another within multilingual PLMs. Besides sci- entific relevance, this has practical implications for the incorporation of external knowledge into multi- lingual PLMs. In fact, while prolific line of work has focused on model editing way to insert new factual associations in PLMs in various data- and computation-effiiinni
Multilingual probing of factual knowledge Next to a multitude of works focusing on English, few notable studies have probed factual knowl- edge multilingually by translating the English prompt-object pairs into a number of languages. X-FACTR (Jiang et al., 2020) and MLAMA (Kass- ner et al., 2021) indicate strong variations between the amount of knowledge in different languages, due to the size of their training corpora. Apart from English and a handful of other high-resource European languages, very low (i.e. < <10%) prob- ing accuracies are reported overall. Another rele- vant work, GeoMLAMA (Yin et al., 2022), probes specifically commonsense knowledge that is sus- ceptible to vary across different regions, leading to the rather surprising finding that the best lan- guage to probe knowledge about certain country (e.g. China) is often not the native language of the given country (e.g. Chinese). The main focus of all these studies is on assessing the amount of factual knowledge encoded for each language, rather than on understanding how such knowledge percolates among languages.
manners (De Cao et al., 2021; Hou et al., 2022; Meng et al., 2022), no one, to our knowledge, has looked yet at how this affects the factual knowledge in languages other than the one to which editing is directly applied.
We conduct the first in-depth study of CLC of factual knowledge in multilingual PLMs and make the following contributions: (i) We propose a novel Ranking-based Consistency (RankC) met- ric which assesses knowledge consistency indepen- dently from accuracy. (ii) We filter the existing unbalanced datasets (Jiang et al., 2020; Kassner et al., 2021) to form a multi-parallel CLC bench- mark, Balanced Multilingual LAnguage Model Analysis (BMLAMA), which has the same set of prompts translated into all languages. (iii) We ap- ply the new metric to BMLAMA to assess CL.C in various encoder-only, decodcr-only, and encoder- decoder PLMs, including XLM-RoBERRTa-arree mar--arree and BLOOM series. We analyze num- ber of language properties that correlate with CLC and provide new insights into how factual knowl- edge percolates among languages. Finally (iv) we
Self-consistency Self-consistency refers to a PLM's ability to output the same answer to seaning-preserved paraphrases of the same query.

---

Self-consistency of English PLMs has received attention across different tasks (Li ct al., 2019; Mitchell et al., 2022; Wang et al., 2023). Fierro and Søgaard (2022) extend the study of self-consistency to multilingual PL.Ms, by measuring it separately in each language. Their results show poor self- consistency in all languages.
score calculation depends on the type of model (encoder-only, encoder-decoder, or decoder-only) and on the way a candidate word gets segmented into subwords (see details in Appendix B). After being sorted by ranking scores, the candidate set for qi is represented as {c]...., c,";, where oj has the highest prediction probability and ct has the lowest. Note that the existing multilingual datasets for knowledge probing (X-FACTR (Jiang et al.. 2020) and MLAMA (Kassner et al., 2021)) have different numbers of queries in different languages, which is problematic for measuring consistency.
Cross-lingual consistency To our knowledge, we are the first to conduct a systematic analysis of cross-lingual consistency of factual knowledge in multilingual PLMs, that is the extent to which a PLM returns the same answer to the same ques- tion asked in different languages. As part of their probing study, Jiang et al. (2020) compute the ra- tio of correct predictions that overlap between two languages within mBERT (cf. Sect. 3.1). They report low ratios overall, with peak of only 34% in the most similar pair (English-Dutch), but do not further investigate the factors that determine con- sistency. Moreover, they limit this analysis to one (encoder-only) model while we also examine an encoder-decoder and a series of decoder-only mod- els (cf. Sect. 5.1). As another difference, we take more holistic view of consistency, whereby predic- tions that are incorrect but refer to the same entity across languages should also be considered con- sistent. Interestingly, concurrent work by Ohmer et al. (2023) proposes to use the cross-lingual con- sistency of a model's predictions as a means to assess its understanding of meaning beyond spe- cific word forms. They showcase their approach in two language understanding tasks (paraphrase iden- tification and natural language inference). Despite the different scopes, their evaluation of ChatGPT using English, German, and Chinese translations reveals limited consistency in the model responses, which is in line with our factual probing findings (cf. Sect. 5) and further indicates this problem persists in very large-scale, last-generation PLMs.
3.1 Prior Work: Correct Predictions Overlap
Based the predictions ot and d' (i.e. first ele- ments of the sorted candidate lists) of each q and de Jiang et al. (2020) compute the average overlap- ping ratio of correct predictions as follows:
10 1(c) o, = 1(c) o.l.d =
COverlap(1,")
(1)
where 1(-) is an indicator function, and o; and o's are the correct answer for qu and of respectively. Because their benchmark contains different amounts of queries in different languages, they filter the query set for each language pair (1, by discarding samples that are not available in cither or l':
Qi. Q: filter(Qi, Qv) |Q7 1Q71
(2)
Since the filtering is done on each language pair separately, this results in different query sets, which limits the comparability of their results across lan- guage pairs that have very different filtered sets.
3.2 This Work: RankC Metric
To ensure comparability between different lan- guage pairs, we instead require that all queries in our benchmark and their corresponding candidates are translated across all languages. Thus, for any language pair (1, I"), the length of the query sets is always equal |Q:l = QQrl. and so is the number of candidates for the i-th query N; = -N-
3 Measuring Cross-Lingual Consistency
Task definition Each language E L has a set of queries (i.e. prompts) defined as Qi- For each query q E Qi, there are N; corresponding can-
didates (Kassner et al., 2021; Yin et al., 2022). For example, the query 'Steve Jobs worked for has 10 candidates: Apple, Nintendo, Google, WWE, Alexandria, Germany, Yahoo, Berlin, BBC, Microsoft. Each query fed the PL.M, and the re- turned probabilities are used to compute a ranking score for each of the candidate words. The specific
Based on these assumptions, we propose novel Ranking-based Consistency (RankC) metric ef- fectively assess the cross-lingual consistency of knowledge in PLMs, independently from accuracy. Instead of merely focusing on the correct predic- tions, we take the rankings of all candidates into

---

consideration. RankC is inspired by the Mean Av- erage Precision at K (MAP@ K) metric for infor- mation retrieval (Schutze al., 2008). Differently from vanilla MAP@ K, in RankC K varies across queries. The value for di equals N₁, the num- ber of its candidates. Given languages I and l", the consistency score between the two languages is de- fined as the mean value for the consistency of all translated query pairs (qi, 2() (Qi, Qv))
Property
BMLAMA-17 BMLAMA-53
# Languages
17
53
# Relations
41
30
Queries
6792*17
3070"53
# Candidates (Avg)
9.71
9.56
Table 1: Statistics the Balanced Multilingual LAMA (BMLAMA) benchmark used in analysis.
Σ Qui === consist(q, of)
scores is discussed in Appendix E.
RankC(I,
(3)
We performed an empirical comparison of RankC to the previously used metric (COverlap, cf. Eq. 1) on the same dataset. The results in Ap- pendix show that nearly all language pairs with high COverlap score also receive a high RankC score. Additionally, RankC reveals some new high- consistency pairs that had a low COverlap score due to low probing accuracy.
Qdl
The consistency for each query pair is calculated by weighted averaging j functions, which output the overlapping ratio among the candidates with top-j highest probabilities':
N:
consist(gi, 9 = Σ 5=1
wj * Paj
Experimental Setup
Paj - ... ..t.....nnn (c (e
Datasets As explained in Sect. 3.2, RankC re- quires that the queries and their candidates are translated into all evaluated languages. We there- fore extract from X-FACTR (Jiang al., 2020) and MLAMA (Kassner et al., 2021) all queries that fulfill this criterion. We call the resulting multi- parallel dataset Balanced Multilingual LAnguage Model Analysis (BMLAMA), and release it in two versions: BMLAMA-1 17 including 6.7k queries 17 languages (close to X-FACTR which includes 23 languages), and BMLAMA-53 including 3k queries in 53 languages (same MLAMA), The detailed statistics are shown in Table 1.
The weight wy for each jaj is defined below.
Ranking-based weights Intuitively, candidates with higher ranking should have more influence on the consistency score. To achieve this, RankC adopts weighted average over all P@js, where P.j with a smaller j is given a higher weight wj to emphasize the effect of candidates with high probabilities. However, the predicted probabilities cannot be directly used since they are different for the candidates of and g/. To solve this issue, introduce normalized weights based on softmax over the value j:
Models Previous work multilingual knowl- edge probing (Jiang al., 2020; Kassner al., 2021) focused on encoder-only PLMs, such as mBERT (Devlin et al., 2019) or XLM-RoBERTa (Liu et al., 2019). However, since decoder-only PLMs have become mainstream in the current NL.P era, our experiments also include the decoder-only BLOOM series (560m, 1.1b, 1.7b, 3b parameters) (Scao et al., 2022) and the encoder-decoder mT5- large (1.2b) (Xue al., 2021), in addition the encoder-only XLM-RRBEERTa-aagg (354m).
Ni-j
W
N Ek
No
(5)
where N₂ is the number of candidates for queries
di and of* Combining equations 3. 4 and 5, the RankC metric becomes:
E
Rdl
P@j 2K= Ne-K
N
i=
RankC(1,1")
Qil
1 Pāj (00
(6) A RankC computation example given in Ap- pendix D and the interpretation of high/low RankC
*e recall that candidates (c)...,, c have been sorted by their PLM probabilities from highest lowest (cf. Sect. For alternative ranking- based weighting schemes, see Appendix C.

---

XLM- XLM-ROBERTB RoBERTa-lerge
BLCOM-1.70
50
mTS large
BLCOM-1.1b
BLOOM-3b
BI COM-560m
40
30
20
10
AVG en
fr
nl
es
ru
ja
zh
ko
Vi
el
hu
he ca ar
uk fa
Figure 2: Factual knowledge probing accuracy (%) various multilingual PLMS, measured on BMLAMA-17.
BLOOM-3b Avg: 25%
mT5-large
XLM-RoBERTa-arrge
Avg: 32%
Avg: 32%
80
40
h1
en T r
05 TU
zhko
hu
ho
fr nl zh o hu
en
ch
hu
Figure 3: Knowledge consistency (RankC %) between language pairs in the PLMs, with darker shading denoting higher-consistency pairs. Green bars: Probing accuracy of each language.
5 Main Consistency Results
5.1 Consistency in Different PLMs
Figure 3 presents RankC results for the three PLMs. The first observation is that average consistency rather low in all models, and lowest in BLOOM- 3b (25%). This negative result is in line with the low overlap ratio of correct predictions observed by Jiang et al. (2020) on mBERT.
Before looking at consistency, we present in Figure 2 the factual probing accuracy results of the three PLMs on BMLAMA-17. We first notice that the encoder-only ALLMRoBBERTa-aage and encoder- decoder - models outperform the whole decoder-only BLOOM series in terms of average probing accuracy. The trend across languages is similar among the three models, however, BLOOM stands out with an English accuracy that is much higher than all other languages. Regarding model size (BLOOM series, green bars), we find that in- creasing the number of parameters leads to slight but consistent improvements in factual probing ac- curacy, which is in line with previous work (Petroni al., 2019).
We now zoom in on the comparison among dif- ferent language pairs, which is made possible by the new RankC metric and the balanced dataset BMLAMA. Here, we find that the European lan- guages English, French, Dutch, Spanish, and Cata- lan share a considerable amount of knowledge in m and pattern applies to BLOOM-30 with the exception of Dutch, which is expected as this language was not included in this model's training corpus. ? In ad- dition, Vietnamese and Turkish achieve remarkable consistency with the above European languages in all PLMs. A common feature of these languages is that they all use the same script (Latin). Another notable high-consistens pair is that of Russian-
A similar
Our XLM- S.MM-RBBRRTa-laree results are consistent with those reported by Jiang et al. (2020) on X- FACTR, which demonstrates the reliability of our multi-parallel dataset BMLAMA.
"The experiments in this section use BMLAMA-I7 as it en- ables better visualization. Results for BMI BMLAMA-53. are given in Appendix H and discussed in the analysis of Section 6.
Pairs of language to itself (e.g. en-en) have always a RankC of 100% and are excluded from the average. Phttps: http///gggggaacccee/ee/ co/ co/biosciiiiieeeee wewpiiieeeeeblommee.

---

100
BLOOM-3b Avg: 25%
BLOOM-1.7b Avg: 25%
tations within a multilingual model. We may also expect that languages spoken in the same world re- gion have higher chances of encountering mentions of the same entities and events in the training data. BL BLOOM-560m
BLOOM-1.1b
Avg: 24%
Avg: 23%
90
50
ko
40
20
en os ja zhlovii of nute caarkki UkTa on
nu 20
huho If Ta on cs 2hko huho oa arukia an ft es jo-2hloov of hulhe DB uliTa
Figure 4: Cross-lingual consistency (RankC) shows little fluctuation among PLMs of different scales. The average probing accuracy of the four models is 25.97%, 24.77%, 22.93%, and 14%, respectively.
Ukrainian, two languages that use the same script (Cyrillicc) and are also closely related.
These observations suggest that various lan- guage properties influence the CLC of multilingual knowledge. We examine number of such proper- ties in Section 6.1.
To answer this question, we obtain four types of typological similarity (syntactic, genetic, geo- graphic, and phonological) from lang2vec (Littell et al., 2017); an open-source library that provides pre-computed similarities based on various typo- logical databases. Next, we compute the Pearson correlation coefficient (Cohen et al., 2009) between RankC scores and the typological similarity of all language pairs in BMLAMA.
5.2 Effect of Model Size
As observed above (green bars Figure 2) and by previous work (Petroni et al., 2019), the ability to retrieve correct knowledge grows with model size when other factors are fixed. We ask whether this is also the case for CLC. However, the BLOOM results in Figure 4 show only a minor variation in average RankC (+2%) when moving from the smallest the largest model in our series, i.e. 5-fold increase in parameters. While this pattern cannot be safely generalized to other models, does suggest that cross-lingual consistency may remain a problem in very large-scale PI.Ms.
Results are shown in Table 2 for both BMLAMA-17 17 and the smaller but more multilin- gual BMLAMA-53. 10 For BMLAMA-17, we find that RankC has a moderate correlation with genetic similarity, weak correlation with geographical similarity, but significant correlation with syn- tactic similarity. As expected, no correlation is observed with phonological similarity. The corre- lation results on the more comprehensive dataset BMLAMA-53 are similar except for syntactic sim- ilarity which acquires a weak positive correlation. Somewhat surprisingly, genetic and geographical similarities see their correlations slightly decreased in this larger dataset, which might be due to the presence of noise in the typological vectors of low- resource languages.
6 Factors of Cross-Lingual Consistency In this section, we examine the different degrees of CLC that are observed in different language pairs, and the factors that may explain such differences.
6.1 Typological Similarity
Typological features have been proven useful to model fine-grained similarities among languages and guide transfer learning techniques for a variety of multilingual NLP tasks (Ponti et al., 2019; Üstün al., 2022). Could such features also explain some of the variance observed in the consistency of fac- tual knowledge within multilingual PLMs? For ex- ample, we may expect that languages with similar grammar and word order, or with related vocabu- laries share a higher degree of linguistic represen-
An important characteristic of genetically related languages is that they tend to have many words in
"The similarity score for language pair ranges from to where represents maximal similarity. For examplc, German Polish have geographical similarity of because they are spoken in neighboring regions
genetic similarity of only 0. 14 because they belong to very different branches of the Indo- European family. Genetic similarity based the distance of two languages the Glottolog tree (Hammarström al., 2017).
*See Appendix for additional results on models with 7h
RankC and genetic similarity values on BMLAMA-53 are given in Appendix H.
parameters.

---

BMLAMA-17
Syntactic
Genetic
Geographical Phonological
Voc(BMLAMA)
Voc(Flores)
BLOOM-3b
0.08 (4e-01) 0.52 (8e-11) 0.37 (8e-06) -0.05 (6e-01)
0.70 (2e-21)
0.49 (2e-09)
marelarge
0.12 (2e-01) 0.50 (4e-10) 0.43 (2e-07) 0.00 (1e-00)
0.71 (2e-22)
0.68 (5e-20)
RoBERTa
-0.04 (6c-01) 0.42 (4c-07) 0.34 (4c-05) -0.09 (3c-01)
0.80 (8e-32)
0.79 (lc-30)
BMLAMA-53
Syntactic
Genetic
Geographical
Phonological
Voc(BMLAMA)
Voc(Flores)
BLOOM-3b
0.14 (2e-07) 0.40 (2e-53) 0.15 (Ic-08) 0.05 (6e-02)
0.69 (3c-195)
0.61 (2e- 140)
mTS-large
0.20 (2e-13) 0.31 (2e-32) 0.23 (3e-18)
-0.02 (4e-01)
0.74 (8e-243)
0.65 (2e-168)
RoBERTa
0.22 (4c-16) 0.28 (5c-26) 0.21 (8c-15) 0.00 (1c-00)
0.70 (5c-205)
0.63 (7c- 156)
Table 2: Pearson correlation (with p-value) between language-puirwise knowledge consistency (RankC) and various typological similarities (left) with subword vocabulary overlap (right).
common or with a common ancestor. Thus, the moderate correlation of RankC with genetic sim- ilarity. combined with the weak correlation with syntactic and geographic similarity, suggests that vocabulary overlap may be more important factor for CLC than having similar grammar and word order being spoken in nearby regions.
1.0
Correlation (with p-value)
0.8
0.61 (2e-140)
RankC
0.6
0.4
0.2
0.0
Subword Vocabulary Overiap (Flores, BLOOM-3b)
Figure 5: Linear regression between subword vocabu- lary overlap (Flores) and CLC measured on BMLAMA- 53 for BLOOM-3b. For results m mT5-large and XLMM- MOBERTa-large see Appendix H.
6.2 Subword Vocabulary Overlap
Based on the above observations, we investigate whether a crude measure of vocabulary overlap could also good predictor of CLC. Specifically, we extract the vocabularies of strictly parallel cor- pora in our evaluated languages and measure their pairwise overlap:
similarity. This suggests that factual knowledge may be primarily percolating across languages in rather shallow way (through the shared use of some subword embeddings) and conversely, may be hampered in the absence of such an- chors even when languages are related. For in- stance, the highest-consistengg pair in BLOOM-3b is Ukrainian-Russian, which lies close in the lan-
(V V(I U()) c [0,
(7)
We consider two corpora: BMLAMA itself and Flores-200 (Costa-jussà et al., 2022). The former is expected to be very relevant, but the resulting cor- relation may be less generalizable as it is the same corpus on which consistency itself measured. By contrast, the latter tences translated from English into 200 languages for the purpose of MT evaluation. Because we are interested in the extent to which different languages make use of the exact same word representations, we segment the corpora with the model's 's tokenizer before measuring vocabulary overlap, which makes this metric -eeeededpp.d !I
guage tree (genetic similarity: 0.8) and shares a vast number of subword vocabulary overall (vocab- ulary overlap: 0.76). However, when querying the working place of David Cameron, BLOOM-3b pre- diets the correct answer in the Russian query ('Lon- don') ) but a wrong answer Ukrainian (" ('Moscow") ) This suggests that the correct knowledge did not transfer from Russian to Ukrainian due to the lim- ited subword overlap between these two queries (0.17). When vocabulary overlap is measured on Flores (last column of Table 2), correlation is lower but still significantly positive, indicating our find- ings are not limited to our benchmark. The correla- tion between cross cross-lingnns lingual knowledge consistency and vocabulary overlap is clearly illustrated in Fig- ure 5.
mix-domain set of 2k sen-
Shown in Table 2 (right), the Pearson correla- tion scores on BMLAMA prove that subword vo- cabulary overlap has a significantly strong impact on the cross-lingual consistency of knowledge in the PLMs, overshadowing the effect of genetic
11 See Appendix H for the vocahulary overlap scores of all language pairs in BMLAMA-53.
The strong dependence of CLC on shallow vo- cabulary overlap partly explains why increasing

---

the model size does not have a positive effect (cf. Section 5.2). We speculate that larger subword vo- cabularies may actually lead to lower consistency as the chances of sharing parts of words between any two languages decrease. We leave a further investigation of this hypothesis to future work.
RankC
Pre-editing Correct Wrong
Post-editing Correct Wrong
Lang with en
Steve Jobs worked for Apple/Microsoft
en
0.95
0.05
0.19
0.81
cs
52 (high)
0.93
0.07
0.12
0.88
vi
49 (high)
0.99
0.01
0.24
0.76
7 Case Study:
Cross-Lingual Consistency and Knowledge Incorporation
hu
26 (low)
0.95
0.05
0.81
0.19
el
24 (low)
0.99
0.01
0.91
0.09
Previous work (Jiang et al., 2020; Kassner al., 2021; Artetxe et al., 2022) and our probing results indicate the amount of knowledge in low-resource languages is limited. Simply training new PLM on larger non-English corpora is time-consuming and the cost is not affordable by most universi- ties and other research institutions (Ding et al., 2022). A promising solution is to incorporate ex- ternal knowledge through fine-tuning methods (Hu et al., 2022) or directly editing the PLM's weights in a very targeted way (De Cao et al., 2021; Meng et al., 2022). To make the process feasible in multi- lingual scenarios and avoid unexpected effects, it is important to understand whether and how inserting piece of knowledge in one language affects other languages within the PLM, including the most and least susceptible languages. In this section, we conduct first case study on this problem and its interplay with CLC.
IBM Connections is created by IBM//dobe
en
0.93
0.07
0.38
0.63
es
52 (high)
0.96
0.04
0.36
0.64
vi
49 (high)
0.96
0.04
0.31
0.69
hu
26 (low)
0.99
0.01
0.85
0.15
el
24 (low)
0.99
0.01
0.68
0.32
Sandy Bridge was product of Intel/Samsung
en
0.99
0.01
0.40
0.60
es
52 (high)
0.98
0.02
0.22
0.78
vi
49 (high)
0.99
0.01
0.09
0.91
hu
26 (low)
0.93
0.07
0.60
0.40
el
24 (low)
0.93
0.07
0.55
0.45
Table 3: Normalized logits for predicting the candidates
before and after model editing BLOOO-33 in English.
with low RankC' (Hungarian and Greek). These languages are also diverse in terms of script and relatedness to English. Six queries are picked by ensuring that the PLM selects the originally correct answer as highly probable before editing. We also ensure that, for each edited knowledge, the sub- ject and object entities are the same tokens across all languages. This eliminates the concern that, e.g., Spanish and Vietnamese get consistent predic- tions with English simply because of the lexical co-occurrence of their subject and object tokens in the evaluated queries. For the evaluation, we follow the setup in Meng al. (2022) and nar- row down candidate sets into two words - one correct and one wrong. The latter is the editing target of ROME. Fed with each query, the PLM calculates logit values for the correct and wrong answers, namely logit and logitw respectively. These logits vary vastly among different languages. To focus the relation between the original and edited fact, normalize the logits following the logitz:
Rank-One Model Editing (ROME) Proposed by Meng et al. (2022), this state-of-the-art model editing technique based on neuron interpretability outperforms several other editing techniques both in terms of specificity and generalization. In short, this technique directly modifies weights in the early feed-forwe forworrdddd layers of the PLM, where a factual as- sociation has been located via causal interventions.
Counterfactual knowledge Following Meng et al. (2022), we consider the task of inserting coun- terfactual knowledge into the PLM, such the factually wrong 'Steve Jobs worked for Microsoft'. Because such factual associations were never ob- served during pre-training, this approach avoids the risk of inserting facts that were already considered likely by the model.
Case study We examine BLOOM-3b since ROME is currently only applicable decoder-only models. English is chosen the source language in which the fact is inserted. As target languages, we choose two that have high consistency (RankC) with English (Spanish and Vietnamese) and two
previous work (Sarti al., 2023) logit logit
and logity 12
logite: logit
12The raw probability values are provided in Appendix K.

---

Table 3 shows the results for three queries. A very clear pattern emerges: when fact is in- serted in English, it propagates consistently to the high-CLC languages (i.e. Spanish and Vietnamese). Conversely, the low-CI.C languages (Hungarian and Greek) are much less affected and still oulput higher probabilities for the correct answers even after model editing. The three remaining queries given in Appendix J show the same pattern.
Union's Horizon 2020 research and innovation pro- gramme (grant agreement No. 819455).
Limitations
Due to restriction of our GPU resources, we could not test models larger than BLOOM-7.1b Extend- ing our analysis to larger-scale models in future work is encouraged to see if the same conclusions reached. Nevertheless, results in Figure 4 indicate that the average CLC grows extremely slowly with the increment of model scale.
Despite the small size of our study, the results suggest that CLC is not only a by-product of the ex- isting knowledge in PLMs, but also represents the susceptibility to perturbation of a language when incorporating new knowledge in other languages. We see this as promising direction to enhance the benefits of model editing in multilingual scenarios.
The facts included in BMLAMA, while sup- posed to be universals, are likely to be more rele- vant to the Western world, which can introduce a bias in the evaluation. We inherit this problem from the benchmarks BMLAMA is built upon. Fixing this issue is not trivial especially in comparative work where probing the exact set of facts across languages is a requirement, and should be given attention in future work.
8 Conclusion
We analyzed the cross-lingual consistency (CLC) of factual knowledge in multilingual large-scale PLMs. We proposed a new metric, RankC, to quantify consistency independently from accuracy, and applied it to a cross- linnusulll balanced factual knowledge benchmark. Our comprehensive analy- sis shows that (i) average CLC is low across differ- ent PLMs and not visibly affected by model size; (ii) CLC of different language pairs within a PLM correlates significantly with genetic similarity, but the correlation with vocabulary overlap notably stronger; (iii) novel facts inserted language via model editing are more likely to propagate to languages having higher CLC scores with X.
Ethics Statement
Since BMLAMA data is derived from previous works X-FACTR and MLAMA, queries in BM- LAMA are also likely to encounter gender and racial bias issues, which are inevitable in the source Wikidata. However, this paper mostly focuses on the consistency between knowledge in different languages rather than the specific knowledge in a single language.
References
Taken together, our findings suggest that fac- tual knowledge may primarily propagate across languages in a shallow way, that is, through shared word embeddings rather than through deeper ingguage-agnosti representations. While this re- sult has negative implications on the generalization abilitics of modern PLMs, it could also be turned into an advantage to design efficient methods for knowledge incorporation in multilingual scenarios.
Israa Alghanmi, Luis Espinosa Anke, and Steven Schockaert. 2021. Probing pre-trained language mod- els for disease knowledge. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNLP 2021, pages 3023-3033, Online. Association for Computational Linguistics.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott. Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pa- sunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jef- frey Wang. Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. 2022. Efficient large scale language modeling with mixtures of ex- perts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1699-1 1732, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Acknowledgements
This publication is part of the project LESSEN with project number NWA. 1779.138.220 20. 183 of the research program NWA-ORC 2020/21 which is (partly) fi- nanced by the Dutch Research Council (NWO). We also gratefully acknowledge the computational resources provided by the High-Performance Com- puting cluster Hábrók. RF is supported by the Euro- pean Research Council (ERC) under the European
Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capa-

---

bilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3563-3574, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Constanza Fierro and Anders Søgaard. 2022. Factual consistency of multilingual pretrained language mod- els. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3046-3052, Dublin, Ireland. Association for Computational Linguistics.
Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from bert. In Proceedings of the AAAI Conference on Artificial Intelliencee volume 34, pages 1466.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference Natu- ral Language Processing (Volume I: Long Papers), pages 38 3816-3830, Online, Association for Computa- tional Linguistics.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901..
Harald Hammarström, Robert Forkel, and Martin Haspelmath. 2017. Glottolog 3.0. Max Planck Insti- tute for the Science of Human History.
Israel Cohen, Yiteng Huang. Jingdong Chen, Jacob Ben- esty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coeffi- cient. Noise reduction in speech processing, pages 1-4.
Yifan Hou, Wenxiang Jiao, Meizhen Liu, Carl Allen, Zhaopeng Tu, and Mrinmaya Sachan. 2022. Adapters for enhanced modeling of multilingual knowledge and text. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3902-30 29223997,,00 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi. Janice Lam. Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling numan-centered machine translation. arXiv preprint arXiv:2207.0467 04.672.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2022. Knowledgeable prompt-tuning porating knowledge into prompt verbalizer for text classification. In Proceedings of the 60th Anmual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2225-2240, Dublin, Ireland. Association for Computational Lin- guistics.
Incor-
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pre- trained models. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN) pages 1173- Hong Kong, China, Association for Computational Linguistics.
Dieuwke Hupkes. Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Chris- tos Christodoullppolls, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari. Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2022. State-of-the-art gen- eralisation research in NLP: a taxonomy and review. CoRR.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6491 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-FACTR: Multilingual factual knowledge retrieval from pre- trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 5943-5959, On- line. Association for Computational Linguisties.
Jacob Devlin. Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Nora Kassner, Philipp Dufter, and Hinrich Schüizee 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 3250-3258 Online. Association for Computational Linguistics.
Ning Ding, Yujia Qin. Guang Yang, Fuchao Wei, Zong- han Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et 2022. Delta tuning: A comprehensive study of parameter efficient meth- ods for pre-trained language models. arXiv preprint arXiv:2003.06904.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Sriku- mar. 2019. A logic-driven framework for consistency

---

of neural models. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLP), pages 3924-3935, Hong Kong. China. Association for Computational Linguistics.
Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language P Processing (EMNLP-IJCNLP), pages 2463-2473 Hong Kong, China, Association for Computational Linguistics.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenete vectors. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2. Short Papers, pages 8-14, Valencia, Spain. Association for Computational Linguistics.
Edoardo Maria Ponti, Helen O' O'Horan, Yevgeni Berzak, Ivan Vulié. Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling Lan- guage Variation and Universals: A Survey on Typo- logical Linguistics for Natural Language Processing, Computational Linguistics, 45(3):559-66 601.
Yujia Qin, Yankai Lin, Jing Yi. Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. Knowledge inheritance for pre-trained language models. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3921-3937 Seattle, United States. Association for Computational Linguistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 1907. I 1692.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in gpt. Advances in Neural Information Pro- cessing Systems, 17:19999
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Procexsing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Eric Mitchell, Joseph Noh, Siyan Li, Will Arm- strong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022. Enhancing self- consistency and performance of pre-trained language models through natural language inference. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 1754- 1768, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Os- kar van der Wal. 2023. Inseq: An interpretability toolkit for sequence generation models. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 421-435, Toronto, Canada. Association for Computational Linguisties.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai- ley Schoelkopf. Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almuharak, Samuel Al- banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generaliza- tion through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié. Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Mathiis Gallé, al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05//
Toronto. Canada, Association for Computational Linguistics.
Hinrich Schutze, Christopher D Manning, and Prab- hakar Raghavan. 2008 Introduction to information retrieval. Cambridge University Press.
Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. 2023, Separating form and meaning: Using self- consistency quantify task understanding across multiple senses. CoRR.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Hao Peng, Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. 2022. COPEN: Probing conceptual knowledge in pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natu- rul Language Processing, pages 5015-5035, Abu Dhahi, United Arab Emirates. Association for Com- putational Linguistics.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint MxXiv:2302.13977
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and

---

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning language models, In The Eleventh International Conference on Learning Representations.
B Prompt-based Probing
Given a query q which carries the subject word and the mask slot(s), the ranking score of each word in the candidate sets is calculated by the follow- ing prompt-based probing methods (Kassner et al., 2021; Yin et al., 2022).
Linting Xue. Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou. Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ler of the Association for Computational Linguistics:
Encoder-only Fed into the PLM M, each candi- date word is tokenized into multiple sub-words $1,..., Snt For instance, 'chopsticks' is split 'chop', 'stick', and 's' in BERT. The ranking score of each candidate e the average log probability of each sub-word occurring at their corresponding mask slot:
Human Language Technologies, pages 483-499 On- line. Association for Computational Linguistics.
Da Yin, Hritik Bansal, Masoud Monajatipoor, Liu- nian Harold Li, and Kai-Wei, Chang. 2022. GeoM- LAMA: Geo-diverse commonsense probing multi- lingual pre-trained language models. Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 2039 2059, Abu Dhahi, United Arab Emirates. Association Com- putational Linguistics.
scorCene (c) =
log PM ([MASK]: =
=8
[MASK] sca 9)
(8)
Encoder-Decoder Given one mask slot, encoder-decoder PLMs able to output sequence of words. Therefore, the ranking score for each word in the candidate is represented as
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gert- jan van Noord, 2022. UDapter: Typology-based Lan- guage Adapters for Multilingual Dependency Parsing and Sequence Labeling, Computational Linguistics, 48(3)5555-592.
SCOT SCoreeneder (c) = log PMM[ = =clg)
(9)
GenBench Evaluation Card
log PM 4)
Figure shows the evaluation card for our experi- ments on BMLAMA according to the generaliza- tion taxonomy of Hupkes et al. (2022).
Decoder-only Decoder-only PLMs abolish the use of mask slots in their pre-training and only perform next-token predictions. Therefore, the ranking score of each word is defined as the out- put probability of the sub-word sequence of q(c). where the candidate word is inserted into the mask slot q. Formally, the score defined as
Motivation
Practical
Cognitive
Intrinsic
Fairesss
Generalisation type
Compo- sitional
Cross
Cross
Cross Domain
Robust- ness
Σ log PM(g(ch(((c)sa) (10)
Structural
score
Task
Language
Shift type
where / is the length of sub-word sequence of q(c).
Covariate
Label
Full
Assumed
C Ranking-Based Weight
Shift source
Naturally
Partitioned
Generated shift
Fully generated
RankC uses weighted average to give more impor- tance to candidates with a higher ranking (Eq. 3):
occuring
natural
Shift locus
RaukC(t, = Σ QQi
consist of)
Train test
Finetune train-test
Pretrain train Pretrain test
'Qi
In alternative the softmax-normalizz weights wy introduced in Eq. 5, we also considered weights based on 1 --norm:
Table 4: LAMA.
Evaluation card for experiments on BM-
N
wi 1 -k)
(11)

---

2-norm:
The consistency between qi and of is then:
(N: - -
consis(g: 9.)
w * P@j
W/
(12)
N, (N; -;)²2 E
j=1
= 0.677 *
+
0.21 * 2 + 1.009 *
1
0.88
softmax
norm1
norm2
Averaging the consistency between all translated query pairs, the CLC between English and Spanish is represented as the RankC score.
Σ QQl f==1 consist(g,, 9f)
RankC(1,1))
(13)
|Q1l
Figure 6: Comparison of different ranking-basse based weight- ing schemes on BLAMA-17. for BLOOM-3b.
E Range of RankC Values
In practice, if model outputs the same answers (the ones with the highest probabilities) for all queries in two languages, the P@l values for each query are constantly and the RankC value be- tween these two languages should be equal or larger
Experimental results in Figure 6 illustrate that the softmax weighting scheme achieves the best visualization by highlighting the most consistent pairs. Therefore, we selected it as RankC weighting scheme in all our experiments.
than I P1+P
Eis wn where wi is the [Q:]
Q11
weight for the candidates with the highest proba- bility, which varies with the length of the candi- date sets. However, since the average candidate is about 10 for BMLAMA, W1 can be approximated by e0+..+ee for nearly all queries. The RankC score under this 'all-same-answer hypothesis is then
D RankC Computation Example
To get a clearer view of the RankC metric, we pro- vide an example of computing the score between the Aiglish-Spanist pair. For simplicity, the num- ber of candidates is narrowed to 3 for each query.
When probed with the original language of The Godfather in English (4,) and Spanish (g/), the PLM predicts the ranking scores for the candidates of the two queries. Based on the scores, the candi- date sets are sorted as (Italian, English, Russian] for English and (italiano, ruso, inglés) for Spanish.
RankC(1, Qii +e. 63%.
Q1
Therefore, two languages are considered highly consistent if their average RankC score around 63% or larger.
P01 of these two sorted candidate sets is } since although incorrect, the answers with the highest probabilities are consistent. The weight wy for it o² is calculated as -ा 0.67, which highlights the effect of the top-1 predictions.
F Comparison of RankC and COverlap
Figure 7 compares the proposed RankC scores COverlap scores Jiang al. (2020), both com- puted on BMLAMA-17. The pairs with consis- tency greater or equal to the average of all language pairs in each PLM are considered high-consistency pairs and marked with squares. We find that nearly all high-consistency pairs found by the previous metric are successfully retrieved by our RankC metric. Despite the three missing pairs, RankC mines total of 7 pairs that are underestimated by OnlyCorrect, where one of the languages has low probing accuracy. This demonstrates the ef- fectiveness of the RankC metric at disentangling consistency from accuracy.
The value for P@2 is 1 since 'Italian' is the only
consistent answer appearing in top-2 predictions of both sets. The weight w2 is क 0.24.
3 because (translations of) The value for P@3 is 3 the first 3 elements in the English candidate set are all covered by the first 3 clements in the Spanish candidate set. However, it is multiplied by low weight u3, equaling 2 -e 0.09. In practice. the average length of candidate sets BMLAMA is around 10. Given E (1....,7 10). the weights for Pāj(j 2 6) are lower than 0.004.

---

Model
CLC Ace
en
fr
nl
65
ja
zh
ko
vi
el
hu
he
tr
ca
ar
uk
fa
RoBERTa
32.0 32.5 46.8 37.3 39.4 38.6 26.7 29.4 23.4 31.1 37.1 27.3 31.3 26.3 37.0 33.2 29.7 28.7 29.8
mT5-large
32.3 33.4
47.4 36.8 40.5 41.6 33.4 22.2 18.9 32.0 34.1 24.8 30.3 29.5 41.8 36.6 34.6 32.9 30.0
LLaMA-7b
28.5 28.5
51.7
31.3 41.5 38.6 34.5 19.9 15.2 19.0 22.3 20.4 28.4 23.5 23.2 34.3 23.1 35.5 22.3
BLO0M-7.1b
26.3 27.8
56.4 35.7 27.0 45.3 22.3 20.4 28.6 18.2 41.9 20.4 16.8 23.3 21.1 33.1 24.7 22.6 15.2
BLOOMZ-3b
24.3
23,9
52.3 37,5 20.6 39.8 16.5 18.6 24.0 17.2 37.6 15.5 16.9 16.1 16.5 24.0 21.3 18.2 13.2
BLOOM-3b
25.2
26.0
54.7 34.6 23.4 41.0 22.1 20.9 25.4 19.4 37.7 20.1 16.2 20.2 20.3 28.3 21.4 22.7 13.3
BLOOM-1.77
24.8
24.8
51.5 31.8 22.7 38.7 24.0 20.1 23.3 17.0 35.9 20.3 16.2 19.3 20.0 25.1 19.9 20.4 14.8
BLOOM-I.Ib
23.9
22.9
48.1
28.0 20.9 36.5 22.6 18.9 21.6 16.1 31.3 19.3 16.1 17.7 17.1 22.2 17.8 21.9 13.8
BL0OM-560m
23.1 21.1
42.2 25.9 18.1 29.7 23.1 18.1 19.1 16.6 26.7 19.1 16.6 19.4 16.0 19.6 14.9 21.0 13.2
Table 5: Overall probing results. Probing accuracy rises with the increment of model size in BLOOM series.
100
BLOOM-30 Avg: 25%
BLOOM-3o Avg: 21%
Dataset
Voc(BMLAMA)
Voc(Flores)
DC
BMLAMA-17
0.83 (3c-06)
0.79 (2c-05)
BMLAMA-53
0.76 (4c-18)
0.75 (6e-18)
20
Table 6: Pearson correlation (with p-value) between language-pairwist knowledge consistency (RankC) and vocabulary overlapping when only considering the sup- ported languages of BLOOM-3b.
m75-large Aug: 32%
mT5-large Avg: 92%
sc
all BMLAMA languages in our main CLC evalua- tion. As suggested by previous work (Blevins and Zettlemoyer, 2022), this could be due to the actual presence in the training data of some non-ofiicially supported languages.
42
XLM-RoBERTe--aree
XLM-RoBERTe-Irgge
To ensure the presence of non- non-supported lan- guages did not affect our analysis of BLOOM, we further calculate the correlation scores between CLC and word-overlapping when only considering the supported languages. As shown in Table 6, the correlation scores between CLC and sub-word vo- cabulary overlap become even higher, confirming our finding that such overlap is a key predictor of consistency across languages.
Aug.
Avg: 33%
RankC (Ours)
COverlap (Previous)
Figure 7: Comparison between RankC and COverlap BLOOM-3b, mT5-large, and XLM-RoBERTa--aars Blue squares: High-consisteney pairs retrieved by both metrics. Red squares: High-consistency pairs mined by only metrie. Green bars: Probing accuracy.
H Additional Results on BMLAMA-53
Figure shows the cross-lingual consistency of language pairs in BMLAMA-53, on BLOOM-3b, mTS-large, and LLM--oBBERTa-lrree Overall, the average consistency is in line with the results on BMLAMA-17, with only a 2% decrease on XLM- RoBERTa-large due to the increasing number of languages.
G Effect of Non-Supported Languages in BLOOM
BLOOM does not officially support all languages in BMLAMA. Despite that, found the factual knowledge probing accuracy of the non-supported languages to be considerably higher than randomly guessing (cf. Table 5) and thus decided to include
Figure 10 shows the pairwise genetic similarity scores in BMLAMA-53. For instance, the Russian- Ukrainian pair here achieves 80% genetic similarity while also shows high CLC in all PLMs.

---

100
BLOOM-3b Avg: 25%
mTS-large Avg: 32%
KLM-RoBERTa-large Avg: 30%
80
80
40
20
Figure 8: RankC scores between language pairs in BMLAMA-53.
100
Vocabulary Overlapping (BLOOM-3b)
Vocabulary Overlapping (mT5-large)
Vocabulary Overlapping (XLM-RoBERTTa-arge)
80
Figure 9: Subword overlapping for languages in BMLAMA-53, varying between PLMs due to different tokenizers.
100
Genetic Similarity
80
60
1.0
Correlaionn (with p-vilue)
0.8
0.65 (2e- 168)
RankC
0.4
0.2
20
Subword Vocabulary Overiap (Flores, mT5-large)
1.0
Corralation (wth p-value)
0.63 (7c-155)
RankC
0.6
0.4
Figure 10: Genetic similarity between language pairs
BMLAMA-53, calculated by lang2vec (Littell al., 2017) based on the distance of two languages in the Glottolog tree (Hammarström et al., 2017).
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Subword Vorabulary Overlap (Flores, XLM-RoBERTaaaaggee
Figure 11: Linear regression between subword vocabu- lary overlap (Flores) and CLC measured on BMLAMA- 53 for mT and MMMM---erta-aaar
Figure 9 shows the subword vocabulary over- lap scores in BMLAMA-53. We observe that high subword vocabulary overlapping leads high RankC scores, such as sr- rr-uubb--ukbg-uu/// wkbsbsss-kbbgsssskkkkkk sr/sk- se pairs in BLOOM-3b, sl-hr/sv-da/idmm pairs in

---

n'T'sllagee and gl-es/id-ms/pt--l pairs in XLM- RoBERTa-large.
Additional Cases of Counterfactual Knowledge Incorporation
Figure shows the correlation berween subword vocabulary overlap (Flores) and CLC (RankC) computed with BMLAMA-53 for mT5- large and XLM-RoBERTa--arge.
RankC
Pre-editing
Post-editing
Lang
with
Correct Wrong
Correct
Wrong
Steve Johs, who was employed by Apple/Miccosoftt
I Knowledge Consistency on More Models
en
0.93
0.07
0.37
0.63
es
52 (high)
0.93
0.07
0.34
0.66
We conducted additional experiments on more mod- els than those included in the main body: namely, BLOOM-7.1b, a larger model in the BLOOM se- ries; BLOOMZ-3b (Muennighoff et al., 2023), branch of BLOOM-3b fine-tuned with human in- struction; LLaMA-7b (Touvron et al., 2023), one of the cutting- edge PLMs trained on much more tokens (i.e. I.4T) than BLOOM (i.e. 366B). Av- crage CLC and probing accuracy for all PLMs are presented in Table 5.
vi
49 (high)
0.91
0.09
0,40
0.60
hu
26 (low)
0.82
0.18
0.75
0.25
el
24 (low)
0.99
0.01
0.97
0.03
Elon Musk worked for Tesla/Chanel
en
0.99
0.01
0.43
0.57
es
52 (high)
0.99
0.01
0.18
0.82
vi
49 (high)
0.99
0.01
0.40
0.60
hu
26 (low)
0.98
0.02
0.58
0.42
el
24 (low)
0.96
0.04
0.95
0.05
BLOOM-7.IB lb: Probing accuracy raises 1.8% relative to BLOOM-3b, while the average CLC still grows relatively slowly (1.1%) in the BLOOM series.
Sandy Bridge is created by Intel/Samsung
en
0.99
0.01
0.33
0.67
52 (high)
0.99
0.01
0.20
0.80
BLOOMZ-3b: both average CLC and probing accuracy are lower than BLOOM-3,, suggesting human instruction fine-tuming has a mildly negative effect on factual knowledge. Nevertheless, average cross-lingual consistency (CLC) still remains less affected, and the heatmaps are nearly the same BLOOM-3b since they use the exact same to- conclusion that CLC is property independent of accuracy and mostly relies on the subword overlapping.
vi
49 (high)
0.99
0.01
0.14
0.86
hu
26 (low)
0.98
0.02
0.55
0.45
el 24 (low)
0.89
0.11
0.86
0.14
Table 8: Normalized logits for predicting the candidates before and after model editing BLOOM-3b in English, for three additional queries.
kenizer. This further strengthens
Three extra queries are shown in Table 8. The languages with high RankC scores with English (i.c. es, vi) also change synchronously when incor- porating new knowledge into BLOOM-3b. Mean- while, the languages with low RankC scores are less affected, in line with our findings in Section 7. Note that the first query ('Steve Jobs, who was employed by _") is paraphrase of the first query shown in the main body (Table 3). The similar result obtained with the two paraphrases suggests that the specific prompt has little influence on the effect of model editing on different languages (Gao et al., 2021). Similar results are also observed in the two cases where the relation is fixed ("_ worked _o) and the model is queried about different subject entities (i.e. Steve Jobs in Table and Elon Musk in Table 8).
LLaMA-7b: Probing accuracy only slightly out- performs BLOOM-7.1 lb having a similar number of parameters. For average CLC, LLaMA-7b reaches 28.5% while BLOOM-7.11 reaches 26.3%. In ad- dition, we calculate the Pearson correlation scores between CLC and vocabulary overlapping of these two models. Results in Table also support our finding that CLC positively correlates with the over- lapping of sub-word vocabulary.
BMLAMA-17
Voc(BMLAMA) Voc(Flores)
LLaMA-7b
0.60 (2c-14)
0.52 (1e-10)
BLOOM-7.16
0.69 (4e-20)
0.48 (3e-09)
Table 7: Pearson correlation (with p-value) between Language-pairvis knowledge consistency (RankC) and vocabulary overlapping I.LaMA-bb and BL.OMM-
K Raw Logits Before/After Model Editing
For replicability, Table K shows the raw (unnormal- ized) logits of the model editing case study.
7.1b.

---

RankC
Pre-editing
Post-editing
Lang
with en
Correct Wrong
Correct
Wrong
Steve Jobs worked for Apple/Microsoft
en
4.1e-1
2.2c-2
1.4e-3
6.1e-3
es
52 (high)
4.0c-1
2.9c-2
1.2e-3
9.0e-3
vi
49 (high)
6.9e-1
9.0e-3
1.3e-3
4.1e-3
hu
26 (low)
8.1e-5
4.1e-6
2.2e-5
5.3e-6
el
24 (low)
8.4e-3
5.1e-5
6.0e-4
5.9e-5
IBM Connections is created by IBM/Adobe
en
1.6e-2
1.3e-3
3.0c-4
5.0e-4
es
52 (high)
3.9e-2
1.7e-3
8.0e-4
1.4e-3
vi
49 (high)
1.3e-1
5.1e-3
1.6e-3
3.6e-3
hu
26 (low)
9.3e-2
4.0c-4
2.2e-3
4.0c-4
24 (low)
7.3e-2
1.1e-3
3.2e-3
1.5e-3
Sandy Bridge was product of Intel/Samsung
en
2.0e-1
1.5e-3
2.0e-4
3.0e-4
52 (high)
1.le1
2.5e-3
2.0e-4
7.0c-4
vi
49 (high)
5.7e-1
1.6e-3
6.0e-4
6.0e-3
hu
26 (low)
5,3e-6
4.0c-7
9.6e-7
6.4c-7
(low)
2.6e-3
2.0e-4
9.0c-6
7.4e-6
Steve Jobs, who was employed by Apple/Microsoft
en
3
2.9e-2
9.0e-4
1.5e-3
52 (high)
5.9e-1
4.7c-2
1.4e-2
2.8e-2
49 (high)
5.3e-1
5.4e-2
1.2e-2
1.8e-2
(low)
1.0c-4
2.2e-5
5,9e-5
1.9e-5
24 (low)
2.3e-3
2.0e-5
2.0e-4
6.3e-6
Elon Musk worked
Tesla/Chanel
1.0e-1
2.0e-6
3.0e-4
4.0e-4
52 (high)
L.le1
4.9c-6
2.0c-4
9.0e-4
49 (high)
2.9c-1
1.0e-6
2.0e-4
3.0e-4
hu
26 (low)
4.7c-6
8.9e-8
1.1e-6
8.3e-7
el
24 (low)
1.7e-8
6.7e-10
1.3e-8
6.0e-10 10
Sandy Bridge created by Intel/Samsung
en es
2.2e-1
1.3e-3
1.0c-4
2.0e-4
52 (high)
6.4c-1
1.4e-3
2.0e-4
8.0c-4
vi
49 (high)
4.0e-1
1.5e-3
2.0e-4
1.2e-3
hu
26 (low)
3.0e-4
7.3e-6
1.8e-6
1.4e-6
el
24 (low)
4.0e-5
4.9e-6
1.3e-5
2.1e-6
Table 9: Raw (unnormalized) logits for predicting the candidates before and after model editing BLOOM-3b in English, for all the examined queries.


Reviews:
[{"review_id": "3TzYYWzjAl", "reviewer": "Reviewer_Ukqz", "summary": "", "questions": "-  Line 227-228: Translate what to what? how to ensure the translations are correct and unbiased across languages?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort in reviewing our paper!\n\n1)Regarding your question about the translation, the probing queries and candidates for each query were originally written in English and then translated to other non-English languages for probing multilingual factual knowledge. \n\nAs for your concern about the correctness and bias issue, our BMLAMA is derived from X-FACTR (Jiang et al., 2020) and mLAMA (Kassner et al., 2021), where all translated queries were manually checked and corrected to ensure each query contains the slots for the subject and object exactly once. Moreover, according to Kassner et al. (2021), manual calibration of the machine translated queries (e.g. fixing grammatical errors or revising by native speakers) had little effect on the probing performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is the first paper that conducts an in-depth analysis of factual consistency across languages and makes good use of it to understand how knowledge gets propagated or not across languages and the underlying factors behind that.\n- Comprehensive analysis that spans multiple state-of-the-art PLMs including encoder-only, decoder-only, encoder-decoder and a representative set of typological languages to draw conclusions.  \n- I think the way they designed this evaluation for factual consistency should be taken as standardized for any paper working on a similar problem.", "reasons_to_reject": "I couldn't come up with any reasons to reject it! This is solid work in my opinion :)", "questions_to_authors": "-  Line 227-228: Translate what to what? how to ensure the translations are correct and unbiased across languages?", "paper_topic_and_main_contributions": "This paper proposes a cross-lingual factual consistency metric that goes beyond accuracy. Using that metric, the author provide an in-depth analysis that looks at different factors, and spans multiple models and languages. They shed light on how factual knowledge propagate across languages and expose new implications that could be directly leveraged to improve the transferability of MLMs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- make Figures 3 and 4 more readable. The font for the numbers is too small."}, "extra_text": {}}, {"review_id": "fcBT73cHYl", "reviewer": "Reviewer_xdFB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the helpful comments!\n\n1)We agree about the overstatement of the dataset contribution. This was an oversight on our part and will be removed from Section 1. We will make it clear that BMLAMA is a balanced version of existing factual knowledge benchmarks, as we already did in the rest of the paper.\n\n2)We appreciate your suggestions about involving more decoder-only models. We have managed to evaluate BLOOMZ-3b and LLaMA-7b on the BMLAMA-17 benchmark.\n\n2.1) For BLOOMZ-3b: The probing accuracy (23.87%) is lower than BLOOM-3b (25.97%), suggesting human instruction fine-tuning has a mildly negative effect on factual knowledge. Nevertheless, average cross-lingual consistency (CLC) remains 33%, and the heatmaps are nearly the same as BLOOM-3b, since they use the exact same tokenizer. This further strengthens our conclusion that CLC is a property independent of accuracy and mostly relies on the subword overlapping.\n\n2.2) For LLaMA-7b: To ensure a fair comparison between LLaMA and BLOOM, we have also carried out experiments on BLOOM-7.1b, which contains a similar number of parameters. In terms of average factual knowledge accuracy, LLaMA-7b (28.51%) only slightly outperforms BLOOM-7.1b (27.82%). For average CLC, LLaMA-7b reaches 36% and BLOOM-7.1b reaches 34%.\nIn addition, we calculate the Pearson correlation scores between CLC and vocabulary overlapping of these two models. \n\n————————————————————————\n\n| $\\quad$ $\\quad$ $\\quad$ $\\quad$ $\\quad$$\\quad$| Voc (BMLAMA) | $ $ Voc (Flores) $ $ |\n\n| $\\quad$LLaMA-7b$\\quad$$\\quad$| $ $ $ $ $ $ 0.60 (2e-14) $ $ $ $ | $ $ 0.52 (1e-10) $ $|\n\n| $\\quad$BLOOM-7.1b$\\quad$| $ $ $ $ $ $ 0.69 (4e-20) $ $ $ $ | $ $ 0.48 (3e-09) $ $|\n\n————————————————————————\n\nThus, the results on LLaMA also support our finding that CLC has a positive correlation with the overlapping of sub-word vocabulary. \n\nWe will add these new results to Table 2 in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation is clear and interesting. Cross-lingual consistency of factual knowledge seems important for understanding the implicit knowledge in multi-lingual pre-trained models.\n- Well-organized and Well-written paper\n- The proposed new metric could promote the research on this topic. The predicted probabilities are also considered in the metric, making it reasonable.", "reasons_to_reject": "- I think the authors exaggerate their dataset contribution in Section 1. Since the proposed benchmark datasets only drive from existing ones with some minor adaptions (translation?).\n- For decoder-only models, in addition to BLOOM, I also recommend the authors conduct experiments on BLOOMZ and LLaMA. (1) For BLOOMZ, it can further analyze the influence of fine-tuning with human instruction (STF) on cross-lingual consistency. (2) For LLaMA, it could be better to conduct experiments on the open-source cutting-edge LLMs and convince the conclusions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the cross-lingual consistency of factual knowledge in multi-lingual pre-trained models. The authors propose a new metric to quantify the cross-lingual consistency (named RankC). To construct the benchmark datasets for probing multi-lingual pre-trained models with the same number of queries across different languages, they also modified existing X-FACTR and MLAMA to BMLAMA-17 and BMLAMA-53, respectively. The conclusion of paper includes (1) cross-lingual consistency is at a low level for existing mainstream multi-lingual pre-trained models (XLM-RoBERTa, mT5 and BLOOM); (2) the degree of cross-lingual consistency does not increase with the increase of parameters; (3) cross-lingual consistency of a language pair is relevant to the genetic similarity; (4) the influence of model editing in one language may transfer to other languages which have a high-level cross-lingual consistency with it.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qVBkpaVpwy", "reviewer": "Reviewer_tJxb", "summary": "", "questions": "**A)** With regards to my comment in the previous section, is there a specific reason why experiments were conducted for BLOOM on all potential language pairs, even when the model does not officially support certain languages? How much would the results be impacted if only supported language pairs would be considered?\n\n**B)** Would you consider incorporating a brief discussion on the interplay between consistency and accuracy? The concept of quantifying consistency independent of accuracy is intriguing. However, addressing the question, \"Does consistency of knowledge across languages imply its accuracy?\" could potentially enrich the context and further highlight the significance of your results.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing valuable feedback and suggestions!\n\n1)Regarding your question about the official supported languages of BLOOM, we decided to include all languages because we found the factual knowledge probing accuracy of the non-supported languages to be considerably higher than randomly guessing (cf. Table 4 in the Appendix). This could be due to the actual presence in the training data of some non-officially supported languages, or to the LM’s ability of making correct factual knowledge predictions even for unseen languages (for instance, because named entities are recognized by the LM anyway).\n\nTo fully answer your question, we conducted additional experiments by only considering the officially supported languages , as you suggested. \n\nThe CLC scores in BLOOM-3b still vary vastly between different language pairs, from 0.23 (ar-vi) to 0.52 (en-es).\n\nen$\\quad$100\n\nfr $\\quad$ 49$\\quad$100\n\nes$\\quad$ 52$\\quad$48$\\quad$100\n\nzh$\\quad$ 34$\\quad$30$\\quad$30$\\quad$100\n\nvi $\\quad$ 49$\\quad$37$\\quad$40$\\quad$29$\\quad$100\n\nca $\\quad$39$\\quad$39$\\quad$42$\\quad$25$\\quad$32$\\quad$100\n\nar$\\quad$ 27$\\quad$32$\\quad$33$\\quad$27$\\quad$23$\\quad$30$\\quad$100\n\n$\\qquad$en$\\quad$ fr$\\quad$ es$\\quad$zh$\\quad$ vi$\\quad$ ca$\\quad$ ar\n\nMeanwhile, the correlation scores between CLC and sub-word vocabulary overlap (table below) become even higher, confirming our finding  that such overlap is a key predictor of consistency across languages. We will add these results to our paper:\n\n——————————————————————\n\n| $\\quad$ $\\quad$ $\\quad$ $\\quad$ $ $ | Voc (BMLAMA) | Voc (Flores) |\n\n| BMLAMA-17 | $ $ $ $ $ $ 0.83 (3e-06) $ $ $ $ $ $ | 0.79 (2e-05) |\n\n| BMLAMA-53 | $ $ $ $ $ $ 0.76 (4e-18) $ $ $ $ $ $ | 0.75 (6e-18) |\n\n——————————————————————\n\n2)We agree that the interplay between consistency and accuracy is an interesting question that can be discussed more openly in the paper.\n\nAccording to our proposed accuracy-agnostic metric, there is not a direct connection between CLC and accuracy. However, if a model outputs correct answers (the ones with the highest probabilities) for all queries in two languages, the P@1 values for each query are constantly 1 and the CLC between these two languages should be equal or larger than $(\\sum{w_1*P@1})/|Query| = \\sum{w_1}/|Query|$, where $w_{1}$ is the weight for the candidate with the highest probability. Since there are ~10 candidates for nearly all queries, $w_1$ can be approximated as $e^9/(e^0+e^1+e^2+e^3+e^4+e^5+e^6+e^7+e^8+e^9)$, and the CLC score under this 'all-correct' hypothesis is $\\sum{w_1}/|Query| \\approx 63$%.\n\nThe other situation where CLC can reach 63% is that the $\\textbf{same}$ wrong predictions are made for all incorrectly predicted queries, which is very unlikely in practice.\n\nTherefore, this number can be seen as a threshold. If CLC between two languages reaches 63% on BMLAMA, they are very likely to make the $\\textbf{same}$ and correct predictions for most queries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While this study primarily focuses on the cross-lingual consistency of factual knowledge rather than its correctness, it could indirectly enhance the latter as well. This is because it does not just identify factual inaccuracies in a particular language, but also investigates the reasons behind them. The conclusions of this paper point out that merely incorporating knowledge in one language and presuming its effective transfer to other languages is insufficient. This widely-held belief in cross-lingual NLP is rigorously questioned in this paper, which should encourage further research on this topic.", "reasons_to_reject": "BLOOM officially supports only a fraction of the languages in BMLAMA-17 and BMLAMA-53, specifically 7 out of 17 and approximately 15 out of 53 respectively. This may significantly influence the cross-lingual consistency results for BLOOM, as well as the subsequently derived conclusions.", "questions_to_authors": "**A)** With regards to my comment in the previous section, is there a specific reason why experiments were conducted for BLOOM on all potential language pairs, even when the model does not officially support certain languages? How much would the results be impacted if only supported language pairs would be considered?\n\n**B)** Would you consider incorporating a brief discussion on the interplay between consistency and accuracy? The concept of quantifying consistency independent of accuracy is intriguing. However, addressing the question, \"Does consistency of knowledge across languages imply its accuracy?\" could potentially enrich the context and further highlight the significance of your results.", "paper_topic_and_main_contributions": "In this paper, the authors quantify the level of consistent factual information shared among languages in three distinct models. They differentiate between the consistency and accuracy of the knowledge, assessing the amount of shared knowledge between different language pairs without considering its correctness. The study presents a new metric, RankC, allowing to measure knowledge consistency independently of accuracy. Additionally, a new benchmark dataset, BMLAMA, is introduced, which allows prompt-based factual knowledge probing across 53 languages, using an equal number of prompts across languages. In addition, the paper investigates the impact of model type, model size, typological similarity and vocabulary overlap on cross-lingual consistency of factual knowledge.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "“Bloom” $\\rightarrow$ “BLOOM”\n\n “xlm-RoBERTa”  $\\rightarrow$ “XLM-RoBERTa”"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that this is an excellent submission. It is the first study of factual consistency across languages, and provides valuable analyses that are deep and comprehensive.
Decision: Accept-Main