Cross-Lingual Knowledge Editing in Large Language Models (2024)

Abstract: Knowledge editing aims to change language models’ performance on several special cases (i.e., editing scope) by infusing the corresponding expected knowledge into them. With the recent advancements in large language models (LLMs), knowledge editing has been shown as a promising technique to adapt LLMs to new knowledge without retraining from scratch. However, most of the previous studies neglect the multi-lingual nature of some main-stream LLMs (e.g., LLaMA, ChatGPT and GPT-4), and typically focus on monolingual scenarios, where LLMs are edited and evaluated in the same language. As a result, it is still unknown the effect of source language editing on a different target language. In this paper, we aim to figure out this cross-lingual effect in knowledge editing. Specifically, we first collect a largescale cross-lingual synthetic dataset by translating ZsRE from English to Chinese. Then, we conduct English editing on various knowledge editing methods covering different paradigms, and evaluate their performance in Chinese, and vice versa. To give deeper analyses of the crosslingual effect, the evaluation includes four aspects, i.e., reliability, generality, locality and portability. Furthermore, we analyze the inconsistent behaviors of the edited models and discuss their specific challenges.1

heading: 1 Introduction
text: The goal of knowledge editing is to adjust language models’ behaviors within an expected scope (i.e., editing scope) and retain out-of-scope model performance ideally (Yao et al., 2023). Along with the dynamic changes in the world, knowledge editing could help models forget outdated knowledge and adapt to the new counterpart without retraining from scratch. As the example shown in Figure 1 (a), the number of Honkai-series games increases
∗Corresponding author. 1Data and codes are available at https://github.
com/krystalan/Bi_ZsRE

heading: Knowledge Editing
text: to four after the release of Honkai: Star Rail (on April 26, 2023). However, if we ask a model that has been trained before the date, the model might only know three Honkai-series games. In such a situation, knowledge editing could help the model efficiently update this new knowledge, and give the right answer after editing.
Despite many efforts devoted to this research field (De Cao et al., 2021; Mitchell et al., 2022b; Dong et al., 2022; Dai et al., 2022; Meng et al., 2022; Mitchell et al., 2022a; Huang et al., 2023b; Meng et al., 2023; Zheng et al., 2023; Wang et al., 2024a,b; Ma et al., 2024; Zhang et al., 2024b,a; Wang et al., 2024c), current knowledge editing studies typically focus on monolingual scenarios, where language models are edited and evaluated within the same language, c.f., Figure 1 (a). Meanwhile, the rapid advancements in large language models (LLMs) have led to the widespread adoption of multi-lingual settings, allowing language
11676
modeling ability can be shared across different languages (Zhao et al., 2023; Wang et al., 2023a, 2022b; Liang et al., 2023a,b; Wang et al., 2023b, 2022a). For example, LLMs such as LLaMA (Touvron et al., 2023a), ChatGPT (OpenAI, 2022), and GPT-4 (OpenAI, 2023) are designed to operate under multilingual setting. Under this background, the performance of a source-language edited model on other languages is still unknown. As shown in Figure 1 (b), a research question (RQ) arises, when we utilize source-language samples to edit a multi-lingual LLM, can the model reflect consistent behaviors when faced with a different target language?
To answer the RQ, in this paper, we explore knowledge editing in cross-lingual scenarios, and study the effects of source-language editing on a different target language. Specifically, we automatically translate the knowledge editing data from English to Chinese via cutting-edge LLMs (i.e., ChatGPT and GPT-4). After carefully comparing existing datasets, we finally choose ZsRE (Levy et al., 2017) which is originally a question answering (QA) dataset and is further widely used in knowledge editing (De Cao et al., 2021; Meng et al., 2022; Mitchell et al., 2022a). More recently, Yao et al. (2023) collect a number of QA pairs that need deep reasoning based on ZsRE, and the data could be used to evaluate the portability of knowledge editing methods beyond simple paraphrasing. Therefore, we also translate these QA pairs to give a deeper understanding of cross-lingual knowledge editing performance. The translated data together with the original ones is denoted as Bi-ZsRE. Then, we conduct English/Chinese editing on several open-sourced multi-lingual LLMs (LLaMA, LLaMA2, Baichuan and Baichuan2), and evaluate their behaviors in Chinese/English in terms of reliability, generality, locality and portability. Our experiments involve seven knowledge editing methods covering three main-stream paradigms pointed by Yao et al. (2023), i.e., memory-based, metalearning and locate-then-edit methods. The experimental results reveal that (1) the language modeling gaps across different languages influence the efficiency of knowledge editing; (2) it is still hard for existing knowledge editing methods to transfer the edited knowledge from one language to another in a multi-lingual LLM; (3) when editing LLMs in a language, the model performance on the irrelevant examples in other languages could also be
influenced, resulting in low locality. This presents a significant challenge for multi-lingual LLMs in maintaining consistent behaviors across different languages.
Our main contributions are concluded as follows: • To our knowledge, we are the first to explore
the cross-lingual effect of knowledge editing in LLMs. We achieved this by translating the ZsRE dataset and studying the cross-lingual effect from English (Chinese) to Chinese (English). • We conduct experiments on various knowledge editing methods and multi-lingual LLMs. Our results indicate that it remains challenging for multi-lingual LLMs to generalize the edited knowledge to other languages. • In-depth analysis of the inconsistent behaviors exhibited by the edited models and their specific challenges provide us with a deeper understanding of the cross-lingual effect in knowledge editing.

heading: 2 Related Work
text: Knowledge Editing Methods. The goal of knowledge editing is to alter the behavior of LLMs within an expected scope (i.e., editing scope) without negatively impacting performance out of the scope. According to a comprehensive survey on knowledge editing (Yao et al., 2023), there are three mainstream knowledge editing paradigms: (1) Memorybased methods keep the original model parameters unchanged while employing another model to influence the model’s behaviors. SERAC (Mitchell et al., 2022b) utilizes a scope classifier to evaluate whether new input is close to the stored editing examples, and further influences the model behaviors based on the retrieved editing examples. TPatcher (Huang et al., 2023b) and CaliNET (Dong et al., 2022) add extra trainable parameters into the FFN layers of LLMs to edit model performance. IKE (Zheng et al., 2023) uses context-edit facts to guide the model in generating edited facts. (2) Meta-learning methods employ a hyper network to learn the weight updates of LLMs to edit the models. KE (De Cao et al., 2021) makes use of LSTM networks to predict the weight update for each new input. MEND (Mitchell et al., 2022a) transforms the gradient of fine-tuned language models by employing a low-rank decomposition of gradients. (3) Locate-then-edit methods first identify parameters corresponding to specific knowledge and then update these parameters. Among them,
KN (Dai et al., 2022) specifies a key-value pair in the FFN matrix that embodies the knowledge and then proceeds to update the corresponding parameters. ROME (Meng et al., 2022) leverages causal mediation analysis to locate the edit area, and update the whole parameters in the FFN matrix. MEMIT (Meng et al., 2023) directly updates LLMs with many memories, and thus, facilitating thousands of edits to be executed simultaneously.
More recently, Xu et al. (2023) introduce the cross-lingual model editing task and design a language anisotropic editing method. However, the proposed method is applied to mBERT (Devlin et al., 2019) (a classical pre-trained multi-lingual NLU model), making the cross-lingual effect still unknown in generative LLMs.
Knowledge Editing Datasets. ZsRE (Levy et al., 2017) is a question answering dataset whose queries require models to answer the questions based on the information within the queries. COUNTERFACT (Meng et al., 2022) evaluates whether the edited model can provide counterfactual answers when asked about the corresponding factual knowledge. MQUAKE (Zhong et al., 2023) aims to assess whether edited models correctly answer questions where the answer needs reasoning based on the edited facts. Cheng et al. (2023) propose MMEdit, a multi-modal knowledge editing benchmark dataset. PersonalityEdit (Mao et al., 2023) is proposed to edit personality traits for LLMs. Li et al. (2023) propose ConflictEdit and RoundEdit to investigate the potential pitfalls associated with knowledge editing for LLMs. Eva-KELLM (Wu et al., 2023) evaluates the edited model from reasoning with the altered knowledge and cross-lingual transfer. Though Eva-KELLM provides a subset for cross-lingual knowledge editing, the data has not yet been made public.2 Besides, this work does not conduct experiments on any knowledge editing methods, leaving the cross-lingual effect still not known in the knowledge editing research field.

heading: 3 Bi-ZsRE
text: In this section, we first discuss the details of data collection, including data sources, translation process as well as quality control (§ 3.1). Then, we give the data statistics of Bi-ZsRE (§ 3.2), and finally provide the task overview of cross-lingual knowledge editing (§ 3.3).
2October 15, 2023

heading: 3.1 Data Collection
text: Data Sources. ZsRE (Levy et al., 2017) is a Question Answering (QA) dataset whose queries require models to answer the questions based on the information within the queries. Following previous data settings (Yao et al., 2023; Wang et al., 2023c), it contains 163,196 training samples and 19,086 validation samples. Each sample involves a question and a corresponding answer for editing LLMs. To evaluate the generality of edited models, a rephrased question is also provided. Besides, each sample also associates with an unrelated QA pair (selected from the NQ dataset (Kwiatkowski et al., 2019)) to evaluate the locality. Recently, Yao et al. (2023) provide a test set with 1,037 samples for a more comprehensive evaluation of knowledge editing, where each test sample additionally contains a QA pair to assess LLMs’ portability to reason based on the edited fact. To control the cost of translation, we randomly selected 10,000 training samples and 3,000 validation samples, which together with all test samples are further translated.
Translation Process. We use gpt-3.5-turbo and gpt-4 to translate the above knowledge editing data from English to Chinese. In particular, considering the trade-off between quality and cost, training samples and validation samples are translated by gpt-3.5-turbo, while test samples are translated by gpt-4. The translation is conducted based on the OpenAI’s official APIs3 with zero temperature. The used translation prompt is shown as follows:
Please translate the following JSON data from English to Chinese and keep the format unchanged: [JSON data]
where each sample is organized in JSON format and further translated at the sample level.
Quality Control. To further ensure the translation quality of the test samples, we also employ three translators to correct the translations of gpt-4. All translators are native Chinese and are fluent in English. As a result, there are about 6.0% of samples are corrected while the remaining are unchanged. All corrected samples are further checked by a data expert who has rich experience in translation annotations. Finally, all translated data and original data are denoted as Bi-ZsRE.
3https://platform.openai.com/docs/ api-reference/chat/object

heading: 3.2 Data Statistics
text: Table 1 lists the data statistics of Bi-ZsRE, covering two languages, English (En) and Chinese (Zh), across three subsets. For English samples, the average question lengths are 11.28, 11.19, and 11.43 tokens in the training, validation, and test subsets, respectively, while the counterparts in Chinese are 10.86, 10.94, and 11.48. Besides, the average length of portability questions is longer than that of original questions, rephrased questions or locality questions, thus portability questions may involve more intricate reasoning based on the edited knowledge. To correctly answer the portability questions, the edited model should absorb the knowledge rather than simply memorize the word replacement.

heading: 3.3 Task Overview
text: Knowledge Editing. Given a language model pθ and an edit descriptor ⟨xe, ye⟩, the goal of knowledge editing is to create an edited model p′θ satisfy the following requirements:
p′θ(x) = { ye x ∈ Xe pθ(x) x /∈ Xe
(1)
where Xe denotes a broad set of inputs with the same semantics as xe. The edited model should also satisfy the following four properties: (1) Reliability measures the average accuracy on the edit case. When receiving xe as input, the edited model p′θ should output ye. (2) Generality evaluates the average accuracy on the equivalent cases as the edit case. For instance, when receiving a rephrased text of xe, the edited model p′θ is also expected to output ye. (3) Locality assesses the accuracy of the edited model on the irrelevant samples. When the input x is out of the edit scope Xe, p′θ(x) should be the same as pθ(x) ideally. (4) Portability measures the robust generalization of the edited model via a portability question that needs reasoning based on
the edited knowledge. When receiving the portability question as input, the edited model p′θ is expected to output the golden answer to demonstrate the model indeed learns the knowledge rather than memorizing superficial changes in wording. Cross-Lingual Knowledge Editing. Given a multi-lingual language model pmθ and an edit descriptor in a source language ⟨xse, yse⟩, the goal of cross-lingual knowledge editing is to create an edited model p′mθ satisfy the following requirements:
p′mθ(x s) =
{ yse x
s ∈ X se pmθ(x s) xs /∈ X se (2)
p′mθ(x t) =
{ It(yse) x
t ∈ It(X se ) pmθ(x t) xt /∈ It(X se ) (3)
where xs and xt denote the input text in the source language s and a different target language t, respectively. X se indicates the edit scope in the source language. It(·) transforms the input text from its source language into the target language t with the same meaning, i.e., translation. Therefore, in addition to learning edited knowledge in the source language, the model p′mθ should also reflect consistent behaviors when querying in a different language. The cross-lingual knowledge editing also needs to satisfy the four properties, i.e., reliability, generality, locality and portability. Different from the monolingual scenario, all test samples (except reliability samples) in the cross-lingual scenario are in both the source and the target languages, respectively. For example, an English edited model will be evaluated by a Chinese generality sample to indicate its cross-lingual generality.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Metrics. To evaluate the edited model in terms of reliability, generality, locality and portability,
different questions, which pair with the golden answers, are input to the edited model. Thus, we follow previous QA studies (Rajpurkar et al., 2016; Yang et al., 2018) and adapt exact match (EM) and F1 as two evaluation metrics: (1) EM measures the percentage of predictions that match the golden answers exactly. (2) F1 measures the average overlap between the prediction and the golden answer. We treat the prediction and ground truth as bags of tokens, and compute their F1.
Baselines. Following Yao et al. (2023); Wang et al.
(2023c), we adopted 7 methods as baselines: (1) Directly fine-tuning (FT) the language models with L∞ constraint; (2) SERAC (Mitchell et al., 2022b) utilizes a scope classifier to evaluate whether new input is close to the stored editing examples, and further influences the model behaviors based on the retrieved editing examples; (3) IKE (Zheng et al., 2023) uses context-edit facts to guide the model in generating edited facts; (4) MEND (Mitchell et al., 2022a) transforms the gradient of fine-tuned language models by employing a low-rank decomposition of gradients; (5) KN (Dai et al., 2022) specifies a key-value pair in the FFN matrix that embodies the knowledge and then proceeds to update the corresponding parameters; (6) ROME (Meng et al., 2022) leverages causal mediation analysis to locate the edit area, and updates the whole parameters in the FFN matrix; (7) MEMIT (Meng et al., 2023) directly updates LLMs with many memories, and thus, facilitating thousands of edits to be executed simultaneously.
Backbones. Considering the English and Chinese abilities, we adopt the following four LLMs in the experiments: (1) Chinese-LLaMA-Plus-7B4 is created based on LLaMA-7B (Touvron et al., 2023a) with Chinese vocabulary extension and continual pre-training. (2) In the similar way, Chinese-LLaMA-2-7B5 is created based on LLaMA-2-7B (Touvron et al., 2023b). (3) Baichuan-7B6 and (4) BaiChuan2-7B7 are two LLMs that support both English and Chinese. Table 3 lists the above LLMs’ performance on CEval (Huang et al., 2023a) and MMLU (Hendrycks et al., 2021) to show their Chinese and English abilities, respectively. Baichuan2-7B performs the best among the four backbones in both two evaluation benchmark datasets. Implementation Details. All experiments are conducted on a single NVIDIA A800 GPU (80G). The implementation of all baselines is employed by EasyEdit (Wang et al., 2023c) with the default settings. The hyper-parameters of each method can be found in the corresponding GitHub repository.8
4https://github.com/ymcui/ Chinese-LLaMA-Alpaca
5https://github.com/ymcui/ Chinese-LLaMA-Alpaca-2
6https://github.com/baichuan-inc/ Baichuan-7B/
7https://github.com/baichuan-inc/ Baichuan2
8https://github.com/zjunlp/EasyEdit/

heading: 4.2 Results & Analyses
text: Table 2 shows the experimental results on ChineseLLaMA-Plus-7B and Chinese-LLaMA-2-7B.
Monolingual Analysis. Compared with the other three properties, portability is more challenging for knowledge editing methods to achieve. As we can see, IKE achieves the best performance in terms of portability among all baselines. However, the best EM score in portability is still less than 60.0, showing it is non-trivial to absorb the edited knowledge for most editing methods. As for reliability which directly evaluates the model performance on the edited knowledge, we find that FT and KN obtain limited performance in this property, thus failing to edit knowledge in LLMs. For example, KN (En) and KN (Zh) only achieve 4.63 and 3.09 F1 reliability with the Chinese-LLaMA-Plus-7B backbone, while the counterparts of FT (En) and FT (Zh) are 20.46 and 9.54. Given the above analyses, we next compare the cross-lingual knowledge editing performance on SERAC, IKE, MEND, ROME and MEMIT methods.
Inconsistent Behaviors in Reliability. When using different languages to edit LLMs, there might be performance gaps in terms of reliability. For example, SERAC (En) achieves 73.84 F1 while SERAC (Zh) only achieves 27.05 F1 on Chinese-
tree/main/hparams

heading: Reliability

heading: Reliability
text: LLaMA-Plus-7B. A similar situation could also be found in MEND, ROME and MEMIT. When using English samples to edit LLMs via these four methods, the achieved reliability is significantly higher than using the Chinese samples. This is because the language modeling ability of different languages might be different in a single integrated multi-lingual LLM. Many LLMs show their strong English ability perhaps due to the highquality English data dominating the pre-training corpora (Touvron et al., 2023a,b). The language modeling gaps of different languages might influence the efficiency of knowledge editing in different languages. Moreover, we find that when editing LLMs via IKE using different languages, the achieved reliability scores are similar. For example, IKE (En) and IKE (Zh) achieve the same F1 score (99.90) in terms of reliability on Chinese-LLaMAPlus-7B. This indicates that not all knowledge editing methods are sensitive to the choice of editing languages. Given the strong in-context learning ability of LLMs, IKE can efficiently edit them with demonstration samples in different languages.
Inconsistent Behaviors in Generality. It is intuitive that when using one language to edit LLMs, the generality in this language is significantly higher than in others. For example, SERAC (En) achieves 50.86 F1 in English generality but only performs 19.26 F1 in Chinese generality (with the
Chinese-LLaMA-Plus-7B backbone). In contrast, SERAC (Zh) achieves better Chinese generality than English (67.41 F1 vs. 14.67 F1). Almost all knowledge editing methods have this phenomenon. This finding also indicates that the cross-lingual performance of knowledge editing is still limited. It is hard for existing knowledge editing methods to transfer the edited knowledge from one language to others in multi-lingual LLMs, and reflect consistent behaviors when querying with different languages.
Cross-Lingual Influence on Locality. When editing LLMs in a source language, the locality in other languages could also be influenced. When editing Chinese-LLaMA-Plus-7B, MEND (En) achieves 88.96 F1 and 91.75 F1 in English and Chinese locality scores, while the counterparts in MEND (Zh) are 89.19 and 78.17. Ideally, when editing LLMs in a source language, its performance on irrelevant target-language samples should remain unchanged. Previous work typically only studies locality in the same language and neglects the cross-lingual locality. We also find that though IKE works well in terms of reliability and generality, its locality is generally less than that of SERAC, ROME or MEMIT. The low-level locality makes its usefulness need to be carefully verified in real applications.

heading: Limited Portability in both Languages. As
text: shown in Figure 2, when editing multi-lingual LLMs in English or Chinese, their portability per-
formance in both languages is extremely limited compared with other properties. This finding indicates that most existing knowledge editing methods only memorize the superficial changes in wording rather than absorbing the edited knowledge. This phenomenon shows that sharing knowledge across different languages is tricky. As a result, the currently edited LLMs reflect inconsistent behaviors on the edited knowledge in different languages.

heading: Knowledge Editing Performance on Baichuan.
text: Table 4 shows the knowledge editing performance on Baichuan-7B and Baichuan2-7B. The results show similar situations to those of ChineseLLaMA-Plus-7B and Chinese-LLaMA-2-7B, verifying the generality of the previous phenomena and our analyses.

heading: 4.3 The Influence of Training Scale
text: Due to the high cost of translating all ZsRE training samples (163K), we randomly select and translate 10K samples via gpt-3.5-turbo (Section 3.1). We further conduct experiments to investigate the model performance when the training set is limited. Specifically, we randomly choose 100, 500, 1K, 3K and 5K training samples to conduct experiments. Among all baselines, SERAC, IKE and MEND are three knowledge editing methods that need additional training. Thus, we use different numbers of training samples to edit LLMs via these methods, and evaluate their performance in terms of reliability, generality, locality and portability. Figure 3 shows the results using Chinese-LLaMA-2-7B as
an example backbone. As we can see, the reliability, generality and portability of SERAC typically increase with the number of training samples, especially SERAC (Zh). The English locality of SERAC is stable while the Chinese locality is sensitive to the training data. As for IKE and MEND, their performances are relatively stable in terms of all metrics. We also find that the portability of IKE and MEND may decrease or remain the same as the number of training samples increases. Therefore, simply adding more training samples in these two methods cannot increase the ability of the edited models to absorb and reason the edited knowledge. Future work could explore more effective methods or design more reasonable training paradigms to let LLMs go beyond memorizing the superficial changes in wording.

heading: 5 Conclusion
text: In this paper, we first explore the cross-lingual effect of knowledge editing in large language models. To achieve that, we automatically construct Bi-ZsRE dataset by translating the previous ZsRE dataset from English to Chinese. Based on BiZsRE, we conduct experiments on various knowledge editing methods, and study the cross-lingual effect from English to Chinese and vice versa. Our results indicate that it is still hard for existing knowledge editing methods to transfer the edited knowledge from one language to another in a multilingual LLM. We also analyze the inconsistent behaviors of the edited models and discuss their spe-
cific challenges to provide a deeper understanding of the cross-lingual effect in knowledge editing.

heading: Ethical Considerations
text: In this section, we consider the potential ethical issues of our work. In this paper, we propose the Bi-ZsRE dataset which is collected based on the publicly-available datasets, i.e., ZsRE (Levy et al., 2017) and portability QA pairs provided by Yao et al. (2023). Therefore, Bi-ZsRE might involve the same biases and toxic behaviors exhibited by these datasets. Besides, we obtain our Bi-ZsRE dataset by translating these datasets, and their corresponding license is the MIT License which is granted to copy, distribute and modify the contents.
During manually correcting the results of machine translation, the salary for each annotator is determined by the average time of annotation and local labor compensation standard.

heading: Limitation
text: While we show the cross-lingual effect in knowledge editing, there are some limitations worth considering in future work: (1) Bi-ZsRE only involves English and Chinese, and future work could extend our Bi-ZsRE to more languages and give more comprehensive analyses w.r.t different language families. (2) The backbones used in our experiments are several LLMs with 7B parameters. Future work can extend our analyses to other LLMs with more parameters (e.g., 13B and 70B).

heading: Acknowledgement
text: This work is partially supported by the National Natural Science Foundation of China (Grant No. 62206056). We thank anonymous reviewers for their constructive suggestions and comments.

heading: Future Work
text: While we show the cross-lingual effect in knowledge editing, there are some limitations worth considering in future work: (1) Bi-ZsRE only involves English and Chinese, and future work could extend our Bi-ZsRE to more languages and give more comprehensive analyses w. r. t different language families. (2) The backbones used in our experiments are several LLMs with 7B parameters. Future work can extend our analyses to other LLMs with more parameters (e. g. , 13B and 70B)