Is Knowledge in Multilingual Language Models Cross-Lingually Consistent?

Authors: Xi Ai, Mahardika Krisna Ihsani, Min-Yen Kan
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multilingual Models, Fact-checking, Cross-lingual Knowledge Consistency, Self-consistency, Model Parity

Abstract:
Few works study the variation and cross-lingual consistency of factual knowledge embedded in multilingual models. However, cross-lingual consistency should be considered to assess cross-lingual transferability, maintain the factuality of the model’s knowledge across languages, and preserve the parity of language model performance. We are thus interested in analyzing, evaluating, and interpreting cross-lingual consistency for factual knowledge. We apply interpretability approaches to analyze a model’s behavior in cross-lingual contexts, discovering that multilingual models show different levels of consistency, subject to either language families or linguistic factors. Further, we identify a cross-lingual consistency bottleneck manifested in middle layers. To mitigate this problem, we try vocabulary expansion, additional cross-lingual objectives, and adding biases from monolingual inputs. We find that all these methods boost cross-lingual consistency to some extent, with cross-lingual supervision offering the best improvement.

Decision: reject
Reviewer scores: 5, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730392722234

thread_type:
official_review

<thread_object>
creator: reviewer_kczx

<content>
summary:
This paper investigates the cross-linguistic consistency of multilingual models. To do this, the authors use the zero-shot cloze-task retrieval format enabled by the mLAMA dataset. However, the task is slightly modified because the prompts are mixed in several languages: the model must not complete the prompt “The capital of England is ___” but “The capital of Grande-Bretagne is ___”. So the words “England” and “Grande-Bretagne” here co-refer to the same underlying concept and should be treated identically by a cross-linguistically consistent model. To measure this consistency, the authors propose several metrics, such as accuracy at rank 1, weighted accuracy at rank 5, but also more precise metrics such as attention analysis or the IG² score.

The XLM-R and mT0 families of models are studied. The main findings are that the models possess a cross-linguistic consistency that is impacted by geographical proximity, language family and writing script. The authors then propose a more in-depth analysis, looking at the consistency score layer by layer and identifying a bottleneck in the intermediate layers.

Finally, the last part of this paper focuses on attempts to improve this consistency in the light of previous findings. To this end, the authors propose several techniques: biasing the model by removing attention or modifying FFN activations, but also by expanding the vocabulary or adding a supervised word alignment training objective. These last two methods are interesting as a direction for future work.

strengths:
- The theme of this paper is very interesting and important.
- Many metrics are introduced, enabling in-depth analysis.
- Use of causal methods to improve and verify their findings a posteriori.

weaknesses:
- Figure presentation: labels/titles should be larger, e.g. Figure 2 shows the same legend and title 3 times, which could be unified in the same graph (or as a table).
- The task definition section is understandable, but I think it needs some rework. In particular, it could benefit greatly from examples as notations are introduced. Finally, I had trouble understanding the baseline: here again, an illustrated explanation would be welcome.
- There are no error bars, yet the effects sometimes seem small (e.g. Figure 2). For example, the differences noted in Figure 5 are subjective and could be more clearly measured. Without clearer statistics, my conclusions from Figure 6 are opposed to those of the authors: I have the feeling that the interventions haven't changed anything, or even degraded consistency!
- Related work is a little light and could be fleshed out further

questions:
1) Have you controlled for the fact that similar languages will sometimes have the same token for proper nouns, thus artificially skewing consistency? For example, “Paris” is spelled the same in French and English, but differently in Chinese.
2) Why not use the baseline for Figures 2, 7 and 10?
3) Why did you choose a code-mixed set-up? It's an out-of-distribution scenario, which can lead to strange behavior that's not representative of reality?
4) Why did you just look at the sum of the IG²s, thus losing information at neuron level, when you could have looked directly at whether the same neurons activated (using an overlap coefficient, for example)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730473404239

thread_type:
official_review

<thread_object>
creator: reviewer_mfk7

<content>
summary:
The submission asks whether the masked language models encode factual knowledge consistently across different languages. In the experiments, the authors propose to use a popular world knowledge dataset: mLAMA. Specifically, they use prompts with subjects translated into German, Indonesian, Tamil, and Arabic. Then, they compare the output distributions of the models for prompts in different languages, interpreting similar ranking of tokens as evidence of knowledge consistency.

Subsequent analysis uses a range of interpretability methods to identify the layers and attention patterns that are important for knowledge consistency. Furthermore, the study shows that particular design choices: larger subword vocabulary and using word alignment objectives in multilingual training benefit consistency across languages.

strengths:
- The work presents a clear-cut research question about multilingual knowledge consistency and proposes a relatively simple experimental setting to address it.  

- The experimental results are followed by a detailed analysis to identify the components and attention mechanisms contributing to knowledge sharing across languages.


- The authors compare XLM-Roberta results with models of similar architecture but differing in specific aspects: XLM-V with a larger vocabulary and XLM-Align using word alignment objective. The survey concludes that these design choices improve cross-lingual consistency.

weaknesses:
- The paper makes the wrong assumption about English and Indonesian being related languages. In fact, the languages are significantly typologically different, but they both are written with Latin script.
My suspicion is that the proposed experimental setting can inadequately boost consistency for languages using Latin script because named entities (from mLAMA) in such languages have the same or similar orthographical forms. To address the concern, the authors could analyze the effect of writing script on consistency in more detail and consider alternative task formulation, e.g., with the whole prompt translated (and transliterated) into the target language.  
[The concern was addressed by the analysis of the transliteration's role added during the rebuttal period]

- Chaotic notation makes the setting hard to understand. Especially section 2 contains multiple instances of notation that are not properly defined (e.g., {...}^{l1} or I^{\S^{l1}}). The methodology description should be clarified, I suggest including a specific example from a dataset. (Improved in the rebuttal version).

- The work only considers masked language models, while most newer models are causal. I do not agree with the justification for discarding causal LMs referring to their “inherent hallucination problem” (sic). Please note that masked language models have also been shown to be prone to hallucinations, e.g. by Myaney, Narayan et al. 2020.

questions:
- Why do you claim that scaling is not a promising strategy for consistency based on the observed bottlenecks in mid-upper layers?  I do not see evidence of that based on the experimental setting.


- How do you treat the cases when the prediction in the original language is wrong? Should we then assign a positive score to the model prediction that repeats wrong predictions in another language?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698220437

thread_type:
official_review

<thread_object>
creator: reviewer_brtt

<content>
summary:
The paper explores knowledge consistency across languages in multi-modal encoder type LLMs by testing on a cloze task with  parallel triples in 52 languages focused on referencing tasks. The paper uses several metrics: consistency metrics, consistency evolution, subset-object attention, and a slightly modified version of a metric called IG^2 to measure levels of cross-lingual consistency. The paper finds that multi-lingual models show different levels of consistency depending on the "differences" among pairs of languages, based on geographical distance, language family and use of similar or different scripts. To obtain better consistency in knowledge across languages, they experiment with some mitigation techniques and find that adding monolingual examples to bias, and adding multi-lingual vocabulary and pre-training with aligned words improve results.

strengths:
Knowledge consistency across multiple languages is important if multi-lingual language models are to be used with confidence in tasks across languages. This paper uses existing metrics to compute knowledge inconsistency in language pairs. The results clearly show that language family and the scripts used matter in the measures of consistency. Similar languages show more consistency, and dissimilar languages shows less consistency. Languages using similar scripts, (i.e., likely similar family and geographical close languages) show higher consistency also. The paper also develops ways to reduce inconsistent knowledge using monolingual biases and appropriate pre-training.

weaknesses:
The fact that knowledge consistency across languages is higher if they are in the same language group or use the same script seems somewhat self-evident, although it is nice to find experimental evidence for the same. The paper works with one multi-lingual reference-oriented code-switched dataset. It also experiments with only one encoder-only model, although two sizes of it.

questions:
What are the types of references the dataset has? Some examples will be nice. Do some reference types work better than others? Some examples will be nice in the appendix. What is the alignment objective? Is there no need for analysis of subject-verb attention  and/or object-verb attention? Can any meaningful comparison be performed with methods published in other relevant papers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728332594

thread_type:
official_review

<thread_object>
creator: reviewer_ahwz

<content>
summary:
This paper investigates cross-lingual factual consistencies for language models, in particular for multilingual encoder-only models like XML-r and multilingual encoder-decoder models like mT0.

Their investigation proceeds along a number of dimensions, including distances between languages as well as a number of mitigation strategies. While the mitigation strategies offer some improvements, cross-lingual supervision directly leads to the largest benefits.

strengths:
Addressing cross-lingual factual consistency is an important problem, which is investigated from several angels.

weaknesses:
My main problem with this paper is the choice to trigger/prompt cross-lingual knowledge through code-switching. This seems a very unnatural choice, given that the models were never exposed (unless by coincidence) to code-switched data. I have serious doubts that the outcomes tell us a lot about the cross-lingual capabilities of these models. Why not use Translations of cloze prompts to measure consistency? I am aware of the fact that code-switching has been used to train multi-lingual models (e.g., mRASP) and that it is beneficial to induced shared representations across languages, but I don't see why this would sense for evaluation.

questions:
While you look into differences between languages, to what extent does the amount of data used during pre-training impact cross-lingual consistency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732219578747

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General

comment:
General:

Thanks for all the valuable comments. We have uploaded a rebuttal revision. 

Major Updates:
1. We polish and proofread the paper based on comments.
2. Following reviewers KcZx and MFk7, we clarify task definitions and provide our input templates (sec 2.1).
3. We address typos in Table 1.
4. Following reviewer aHwz, we analyze a new model fine-tuned on code-mixed dataset (sec 4.4). Overall, code-switching training offer significant gains. 
5. Following reviewer brtt, we provide our preliminaries on mt5 in comparison to mt0, which can be used to discuss the potential mitigation for encoder-decoder models. Overall,  mt5 and mt0 are similar in cross-lingual consistency.
6. Following reviewer MFk7, we add a new section to Appendix (A 4.4) to study transliteration. Overall,  the evaluation task does not inadequately boost consistency for languages using Latin script because word alignments resulting in cross-lingual consistency are the main factor in the experiments. 
6. Following KcZx,  we add error bars to all figures.

Due to time constraints, we only add results on 4 similar and dissimilar languages for new models suggested by reviewers. We will add the overall result (based on the three factors) to our next revision, which requires us to run experiments for all 53 languages. 

We hope our rebuttal revision can mitigate the concerns. Further questions and comments are always welcome.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732501768941

thread_type:
official_comment

<thread_object>
creator: area_chair_ldeg

<content>
title:
Respond  to the author's rebuttals

comment:
Dear Reviewers,

Thank you for your efforts in reviewing this paper. We strongly encourage you to review and respond to the author's comments to promote a more dynamic exchange of ideas.

Thank you for your collaboration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732985287094

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Near Deadline

comment:
Dear reviewers,

Thank you very much for your time. If you have future questions, please let us know. We would appreciate it if you could reassess our paper based on our responses and the rebuttal revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates cross-lingual factual consistency in multilingual language models, focusing on encoder-only models like XML-R and encoder-decoder models like mT0. The study examines how these models maintain knowledge consistency across different languages, using a cloze task with parallel triples in 52 languages, leveraging the mLAMA dataset. 

However, there are noteworthy concerns. 1)  The novelty of the paper is limited, as the cross-Lingual consistency of factual knowledge in pre-trained language models has been studied in previous works.  2) Clarifying the motivation behind using code-switching to trigger or prompt cross-lingual knowledge is essential. 3) Including experiments on decoder-based language models would further strengthen and validate the conclusions.

Addressing the aforementioned questions will render the paper more comprehensive and impactful.