Retrieval-Augmented Multilingual Knowledge Editing (2024)

Abstract: Knowledge represented in Large Language Models (LLMs) is quite often incorrect and can also become obsolete over time. Updating knowledge via fine-tuning is computationally resource-hungry and not reliable, and so knowledge editing (KE) has developed as an effective and economical alternative to inject new knowledge or to fix factual errors in LLMs. Although there has been considerable interest in this area, current KE research exclusively focuses on monolingual settings, typically in English. However, what happens if the new knowledge is supplied in one language, but we would like to query an LLM in a different language? To address the problem of multilingual knowledge editing, we propose RetrievalAugmented Multilingual Knowledge Editor (ReMaKE) to update knowledge in LLMs. ReMaKE can be used to perform model-agnostic knowledge editing in a multilingual setting. ReMaKE concatenates the new knowledge retrieved from a multilingual knowledge base with users’ prompts before querying an LLM. Our experimental results show that ReMaKE outperforms baseline knowledge editing methods by a significant margin and is scalable to real-word application scenarios. Our multilingual knowledge editing dataset (MzsRE) in 12 languages, the code, and additional project information are available at https://github. com/weixuan-wang123/ReMaKE.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 335–354 August 11-16, 2024 ©2024 Association for Computational Linguistics
Knowledge represented in Large Language Models (LLMs) is quite often incorrect and can also become obsolete over time. Updating knowledge via fine-tuning is computationally resource-hungry and not reliable, and so knowledge editing (KE) has developed as an effective and economical alternative to inject new knowledge or to fix factual errors in LLMs. Although there has been considerable interest in this area, current KE research exclusively focuses on monolingual settings, typically in English. However, what happens if the new knowledge is supplied in one language, but we would like to query an LLM in a different language? To address the problem of multilingual knowledge editing, we propose RetrievalAugmented Multilingual Knowledge Editor (ReMaKE) to update knowledge in LLMs. ReMaKE can be used to perform model-agnostic knowledge editing in a multilingual setting. ReMaKE concatenates the new knowledge retrieved from a multilingual knowledge base with users’ prompts before querying an LLM. Our experimental results show that ReMaKE outperforms baseline knowledge editing methods by a significant margin and is scalable to real-word application scenarios. Our multilingual knowledge editing dataset (MzsRE) in 12 languages, the code, and additional project information are available at https://github. com/weixuan-wang123/ReMaKE.

heading: 1 Introduction
text: Large Language Models (LLMs) are being used as sources of factual knowledge for search engines and other downstream tasks. Despite their considerable progress, knowledge generated by LLMs can be incorrect or become obsolete in a changing world. Pre-training from scratch or fine-tuning LLMs to adapt them to new knowledge is computationally expensive and not guaranteed to work. Knowledge editing (KE) methods (Zhu et al., 2020; Cao et al., 2021; Mitchell et al., 2022b; Zheng et al.,
2023) have been proposed as effective and economic alternatives to fine-tuning when specific factual knowledge needs to be added or updated. KE involves either updating the parameters of a model (Dai et al., 2022a; Mitchell et al., 2022a; Meng et al., 2022, 2023; Dai et al., 2022b) or adding extra components to an LLM (Mitchell et al., 2022b; Zheng et al., 2023; Dong et al., 2022; Hartvigsen et al., 2022). For example, KE can be used to correct the answer to this question “Who is the foreign secretary of the UK?” from “James Cleverly” (true until mid November 2023) to “David Cameron”, who had recently been appointed to the post.
Despite significant interest in this problem, current research on KE predominantly concentrates on a monolingual setting, where both the injected knowledge and the subsequent queries to the LLM are in English (Mitchell et al., 2022a; Meng et al., 2022, 2023; Mitchell et al., 2022b; Zheng et al., 2023). Companies serving a multilingual customer base need to consider the multilingual KE case, where KE is done in one language and this propagates to queries and answers in all other languages. While Wang et al. (2023a) explored the applicability of knowledge editing to the English-Chinese cross-lingual scenario, their primary focus was to highlight the challenges rather than develop a functional KE approach in a multilingual setting.
Drawing inspiration from in-context learning (ICL), in-context knowledge editing (IKE) uses prompts to edit factual knowledge. It is noted that IKE is so far the only method demonstrating positive results in the cross-lingual KE task setting (Wang et al., 2023a). However, IKE requires explicit provision of new knowledge every time an LLM is used, confining its practicality and scalability in real-world applications. In addition, IKE suffers when irrelevant facts are included in the prompt (Wang et al., 2023c) especially in scenarios where a substantial number of facts are being edited.
335
In this paper, we propose Retrieval-Augmented Multilingual Knowledge Editor (ReMaKE) that integrates multilingual retrieval from a knowledge base with in-context learning. ReMaKE concatenates the retrieved knowledge from an external database with a user query to create the prompt. The proposed multilingual retriever grounds ReMaKE to the retrieved accurate and up-to-date information highly relevant to user queries, therefore effectively mitigating the contextual interference due to irrelevant context. In this way, the generated prompts are able to guide the LLMs in producing accurate responses associated with the injected knowledge. ReMaKE leverages a knowledge base’s ability to scale to further enhance IKE’s knowledge editing performance in real-world application scenarios where large volumes of edits are in scope. Figure 1 shows the architecture of the proposed retrieval-augmented multilingual knowledge editor. Our main contributions are listed below:
• Multilingual knowledge editing: ReMaKE extends the scope of knowledge editing practices across language boundaries. Given that the multilingual knowledge base and multilingual retriever operate independently to a specific LLM, ReMaKE is a plug-and-play tool applicable to any LLM. It is scalable, capable of editing a large number of knowledge. Experiments show ReMaKE outperforms baseline methods by a significant margin in the average accuracy score (up to +40.53%).
• Multilingual editing dataset: We build a machine-translated multilingual knowledge
editing dataset (MzsRE) in 12 languages: English, Czech, German, Dutch, Spanish, French, Portuguese, Russian, Thai, Turkish, Vietnamese, and Chinese using the zsRE testset (Levy et al., 2017). The dataset is available to the community from our accompanying GitHub repository.

heading: 2 Related Work
text: Knowledge editing: Monolingual knowledge editing methods can be categorized into four main paradigms (Yao et al., 2023; Zhang et al., 2023): 1. Hypernetwork editors (Cao et al., 2021; Mitchell et al., 2022a; Hernandez et al., 2023) re-frame knowledge editing as a learning-to-update problem with the help of gradient shift, which is predicted by extrinsic editors. While the research scope extends beyond a single editing, the success rate of edits diminishes remarkably when more edits are executed simultaneously. 2. Locate-and-edit editors (Dai et al., 2022a; Meng et al., 2022, 2023; Dai et al., 2022b; Xu et al., 2023; Chen et al., 2023) are used to locate the parameters related to factual knowledge and subsequently modify them. It is worth noting that this method requires an errorprone analytic step to identify parameters. It is model-specific and not efficient, as the locations are unique for each LLM. 3. Plug-in editors (Cao et al., 2021; Mitchell et al., 2022a; Hernandez et al., 2023) can be used to add extra components to generate predictions about new knowledge without impacting on the parameters of the LLMs. Although this method is less influenced by unrelated inputs, it often cannot achieve precise editing. 4. Prompt-
based editors like IKE (Zheng et al., 2023) use ICL to attach knowledge to the context of a prompt. Compared with other KE methods, IKE achieves a considerably stronger editing performance, together with far fewer side effects. However, IKE needs explicit provision of new knowledge every time, limiting its practicality and scalability in realworld applications.
All above-mentioned editors are limited to a monolingual setting. For cross-lingual editing, Xu et al. (2023) and Chen et al. (2023) require training and manipulating model parameters. They rely on the masking mechanism, limiting the scope of application to a one-word output scenario. Most KE approaches are designed to process single editing and, therefore, have limited practicality. Our proposed editor, ReMaKE, addresses these problems and scales multilingual KE to cover a considerable amount of knowledge.
Retrieval-augmented in-context learning: Offthe-shelf search engines are often used to enhance retrieval-augmented ICL (Gao et al., 2021; Shi et al., 2023; Liu et al., 2023), attaching semantically similar examples to the context to improve the performance of LLMs in few-shot learning. In cross-lingual scenarios, a search engine first uses a low-resource language input sample as a query to find the most semantically similar high-resource language sample in the corpus. The retrieved highresource language sample is concatenated with the input sample to form a prompt for an LLM. For instance, Nie et al. (2023) retrieves semantically similar cross-lingual sentences as prompts to improve the performance of sentiment classification for lowresource languages. While ICL can be used to support cross-lingual tasks, using it to edit knowledge across language boundaries has not been explored. KE in a cross-lingual setting remains a challenge (Wang et al., 2023a). To bridge this gap, ReMaKE leverages ICL and knowledge retrieval to perform cross-lingual KE.

heading: 3 Retrieval-Augmented Multilingual Knowledge Editing
text: ReMaKE enables knowledge in an LLM to be edited in one language and subsequently queried in multiple languages. ReMaKE consists of two stages: multilingual knowledge retrieval and multilingual in-context editing.

heading: 3.1 Multilingual Knowledge Retrieval
text: We propose a simple multilingual retrieval model to search for the most relevant knowledge stored in the knowledge base for a query. As shown in Figure 1, the proposed retrieval model initially maps a query and knowledge base entries to a shared multilingual semantic space. We train a classifier on top of these semantic representations to determine if the knowledge is semantically related to a query. The classifier is based on a sentence transformer (i.e., XLM-R).
More specifically, we finetune the multilingual retrieval model fθ with a binary classification head on the multilingual parallel dataset constructed by translating our English training dataset using Google Translate. We use the separator token 
 to concatenate the sentence x and its corresponding translation I(x) to format the input, predicting whether they are semantically related (related: fθ(x, I(x)) ≥ 0.5 vs. unrelated: fθ(x, I(x)) < 0.5). Negative examples are constructed by pairing unrelated sentences between languages.
Once trained, the multilingual retriever fθ is used to map a query xl1 in language l1 to knowledge kl2 in language l2. From a knowledge base Kl2 = {k0l2 , .., kil2 , ..., kKl2 }, we use the retriever fθ to score each knowledge item for the query and either select the most related knowledge, or an empty R(xl1):
kl2 = R(xl1) =
{ ki
∗ l2
fθ(xl1 , k i∗ l2 ) ≥ 0.5
None fθ(xl1 , k i∗ l2 ) < 0.5
(1)
where i∗ = argmaxifθ(xl1 , k i l2 ) is the index that maximizes the relevance fθ(xl1 , k i l2 ).
The multilingual retrieval model gives a solid performance on our test set with an average retrieval accuracy 99.31%, as shown in Figure 6 (Appendix A.2). ReMaKE can be extended to accommodate a more efficient and performant Information Retrieval model for real-world deployment. We leave this extension as one of our future endeavors.

heading: 3.2 Multilingual In-context Editing
text: As shown in Figure 2, ReMaKE can perform zeroshot and few-shot editing. In zero-shot editing (ReMaKE-zero), the retrieved result (“new knowledge” in Figure 2) is concatenated with a user’s query (“test input” in Figure 2) to form a prompt (“zero-shot prompt” in Figure 2) for an LLM to predict the output P (yl1 |xl1 , kl2).
In few-shot editing (ReMaKE-few-bi), bilingual examples S = {(s1l1 , s1l2), ..., (s q l1 , sql2)} are prepended to the “new knowledge” and the “test input”, where sjl1 and s j l2
are the same statement in languages l1 and l2, corresponding to the “Q_es: ... A_es: ...” and “Q_en: ... A_en: ...” (“few examples” in Figure 2). In few-shot editing, we concatenate “few examples”, “new knowledge”, “test input” as a prompt (“few-shot prompt” in Figure 2). The goal of predicting an edited knowledge is P (yl1 |xl1 , kl2 , S). For the few-shot setting, we follow Zheng et al. (2023) in selecting examples using an unsupervised method from the corpus based on their cosine similarity to the inputs (with the all-MiniLM-L6-v2 model1). The selected examples are included in the context to perform in-context learning. ReMaKE leveraging a searchbased strategy can significantly outperform that based on a random selection (see our experiment in Appendix A.8).
1https://huggingface.co/sentence-transformers/allMiniLM-L6-v2

heading: 4 Metrics, Data and Model

heading: 4.1 Metrics
text: Following Wang et al. (2023a), we evaluate multilingual knowledge editing with the following four metrics: (1) Reliability evaluates the average accuracy of an LLM on all edited instances. In other words, we update the LLM with new knowledge and test the accuracy of querying the LLM for this new knowledge. (2) Generality measures the average accuracy of an LLM for the paraphrased inputs for all edited instances. It indicates ReMaKE’s effectiveness under the prompting frame bias (Wang et al., 2023c) induced by paraphrasing. (3) Locality assesses the average accuracy of an LLM in response to queries on irrelevant semantics after knowledge editing. It tests the knowledge editors’ ability to update only the desired knowledge without affecting other knowledge in the model. (4) Portability estimates the average accuracy of an LLM for questions requiring reasoning after knowledge editing. Questions are constructed to test an LLM’s ability to provide answers requiring it to reason. Portability can indicate whether KE can effectively adapt knowledge to support reasoning.

heading: 4.2 Data Construction
text: Zero-Shot Relation Extraction (zsRE) (Levy et al., 2017) is a monolingual question-answering test set containing 1,038 samples widely used in the knowledge editing task. There is a question-answer pair for each knowledge where the answer is an alternative counterfactual prediction (Cao et al., 2021). The counterfactual answer is expected to be generated by the post-edited LLMs. Additionally, a paraphrased question, an unrelated question, and a portability question are provided to evaluate the generality, the locality, and the portability of the editing. An example of editing knowledge in Spanish and testing in English for four metrics is shown in Table 1. We translate the zsRE from English (EN) to ten languages: Czech (CS), German (DE), Dutch (NL), Spanish (ES), French (FR), Portuguese (PT), Russian (RU), Thai (TH), Turkish (TR), and Vietnamese (VI) with Google Translate and use the Chinese (ZH) zsRE test set (which was also machine translated from zsRE) from (Wang et al., 2023a) to construct the multilingual zsRE test set (MzsRE). As there are multiple counterfactual answers for the same question in zsRE, we de-duplicate MzsRE to 743 items to ensure each question corresponds to a single unique an-
swer, avoiding conflicting entries in the knowledge base. Table 5 (Appendix A.1) lists the statistics of MzsRE.

heading: 4.3 Base LLMs
text: Two representative multilingual LLMs are selected as backbones for us to test various KE methods in the experiments: LLaMA2-7b and BLOOMZ-7b1mt, where LLaMA2-7b2 (Touvron et al., 2023) is a foundation model and BLOOMZ-7b1-mt3 (Muennighoff et al., 2023) is an instruction-finetuned model. We translate a random sample of 10,000 instances from the zsRE training dataset into the other 11 languages and finetune an XLMRoBERTa-base4 (Conneau et al., 2020) on this multilingual dataset to develop our multilingual retriever.

heading: 4.4 Implementation Details
text: All experiments are conducted on a single NVIDIA A-100 GPU (80G). The implementation is based on the EasyEdit (Wang et al., 2023b) framework.

heading: 4.5 Baseline
text: We choose three top-performing KE methods based on the LLMs in Wang et al. (2023a) as our baselines for the experiments. IKE (Zheng et al., 2023) uses in-context learning for knowledge editing, where the prompt consists of one explicit piece of knowledge in the editing language, one query in the test language, and a certain number of examples in the editing language (16 in this case, following the setting in Wang et al. (2023a)). We also test a memory-based KE method SERAC (Mitchell et al., 2022b)). with a memory size K (K = 10 is the default parameter in Mitchell et al. (2022b)). The classifier and counterfactual model in SERAC are pre-trained on the monolingual dataset in the editing language. The parameters of the LLM for both methods mentioned above are frozen. To compare the effect of parameter-updating KE, We evaluate the ROME (Meng et al., 2022) method, which locates the knowledge in the editing language first and then edits it. After updating the parameters, we evaluate the performance with a query in the testing language. For all baselines, we use their original proposed default parameters and LLaMA2-7b as the backbone in the experiments. All KE methods are tested on the multilingual knowledge editing.
2https://huggingface.co/meta-LLaMA/LLaMA-2-7b-hf 3https://huggingface.co/bigscience/bloomz-7b1-mt 4https://huggingface.co/xlm-roberta-base

heading: 5 Experimental Results
text: We define a standard notation (“LANGUAGE 1 (edit) → LANGUAGE 2 (test)”) to refer to the multilingual KE experiments. For example, “ES (edit) → EN (test)” refers to an experiment in which Spanish is the editing language and English is the language we tested the knowledge, as shown in Figure 2. The knowledge base of the experiments consists of all the above-mentioned multilingual knowledge instances in the MzsRE test set.

heading: 5.1 English-based Multilingual KE
text: In this subsection, we conduct experiments in which English is used as either the editing or testing language, investigating the effects corresponding to all other languages. The evaluation results of LLMs on the LLaMA backbone in 12 languages after editing in English (aka “EN (edit) → ALL (test)”) are shown in Table 2 (based on Exact Match (EM)) and Table 9 (based on F1). Experimental results on the LLaMA backbone obtained from “ALL (edit) → EN (test)” are shown in Table 3 and Table 10. We examine ReMaKE with LLaMA backbone in various scenarios, including zero-shot (“ReMaKE-zero”), monolingual few-shot setting (“ReMaKE-few-mono”), and bilingual few-shot settings (“ReMaKE-few-bi”). We compare results of these settings with those from three baseline methods and the pre-editing setup (“LLaMA”).
As shown in Table 2, current KE approaches perform reasonably well in the monolingual case (see Reliability for SERAC, IKE, and ROME for “EN (edit) → EN (test)”) but do not work in the multilingual setting. “LLaMA” fails with poor pre-editing results (less than 2%) as the knowledge editing test examples are counterfactual. SERAC scores all zeros in the multilingual case except for the Locality metric in response to irrelevant queries. ROME performs similarly poorly in the multilingual setting. The top performer, IKE, shows 100% accuracy for monolingual KE, which is considerably better than other baselines. ReMaKE reveals a significant improvement over IKE in multilingual language conditions. ReMaKE, although fundamentally similar to IKE, provides bilingual few-shot examples and an additional means to filter out irrelevant queries (by returning null knowledge), leading to significant improvements in all four metrics. Furthermore, the scalability and the precision of the editing are also improved by an accurate multilingual retriever.
For Reliability (average accuracy), ReMaKE
averagely outperforms baseline IKE by up to +31.19%. Take “EN (edit) → ES (test)” as an example, SERAC has the worst reliability score (0.00%) as the counterfactual model (used to generate predictions about new knowledge) in SERAC is monolingual, and IKE and ROME have reliability scores of 36.45% and 12.25%, respectively. ReMaKEzero achieves a reliability score of 52.62% instead. When scaled up to a few-shot setting, ReMaKEfew-mono drops to 43.47% due to the negative influence of the monolingual context, but adding bilingual examples to the context makes ReMaKEfew-bi the most capable KE with a reliability score of 68.10%.
With regard to the results of “ALL (edit) → EN (test)”, ReMaKE-few-bi achieves the highest scores, outperforming baselines by significant margins (up to +40.53% and +40.40%) in the average reliability and generality scores5. It records a relia-
5A counterfactual model is required for each language for SERAC, leading to significant computation overhead. It is not included in this experiment to this end.
bility score 86.41% in “ES (edit) → EN (test)”. The discrepancies in ReMaKE scores across languages reflect the disparate capabilities of multilingual LLMs. After editing knowledge expressed in English, ReMaKE-few-bi attains the highest crosslingual reliability score of 81.70% (“EN (edit) → DE (test)”) when testing the LLM in German. The lowest reliability score 32.44% is obtained when the test language is Thai (TH), indicating the effect of KE on LLMs is sensitive to language settings. A similar phenomenon can be observed for the same KE method (ReMaKE-few-bi) on a different backbone LLM (i.e., BLOOMZ in Appendix A.3). The sensitivity may be caused by unbalanced distributions of training data across languages. It can be observed that a high-resourced language and a powerful LLM are preferable choices. Even though ReMaKE appears sensitive to language settings and backbone LLMs, it consistently enhances the multilingual KE performances of baselines.
After knowledge editing, the locality of an LLM
can be significantly influenced, as shown in Tables 2-3. The locality is calculated by comparing preedit and post-edit predictions to show the degree an LLM is affected by irrelevant input after editing. IKE performs poorly in locality, with most of the EM scores below 1%. It can be observed that ReMaKE can achieve consistently high locality scores across language settings and backbone LLMs due to its robustness against contextual interference. The English classifier implemented in SERAC treats all non-English multilingual queries as irrelevant information, resulting in abnormally high scores in the locality metric while producing a minimal score (0%) in the reliability assessment (Table 2).
It is expected that all KE methods record notably low portability, given their limited capability in impacting LLMs’ reasoning capability. Understanding the mechanism responsible for the reasoning capability of an LLM remains a challenge. We notice that ReMaKE-zero outperforms all other KE methods and its few-shot counterpart (ReMaKEfew), possibly due to a reduced level of context interference imposed on an LLM. We leave this hypothesis to a future study.

heading: 5.2 Multilingual KE between All Languages
text: The results of the above-mentioned metrics based on EM are illustrated as heat maps in Figures 3 and 7 (Appendix A.5) for multilingual KE between all twelve languages (“ALL (edit) → ALL (test)”). The discrepancies in the reliability and generality scores between a certain language group (i.e., ZH, TH, and TR) and the rest of the language groups are significant. It appears natural segregation exists between these languages and the rest of Indo-European languages, probably due to their linguistic characteristics and the language distribution in the training dataset.
Moreover, the portability scores captured in Figure 7(c) (Appendix A.5) are below 10%, which are much lower than those shown in English-based multilingual KE (Tables 2-3). It is more difficult for multilingual KE methods to impact the reasoning capability of an LLM when English is not in the loop. Whether we should employ English as a pivot for multilingual KE to enhance the portability score, particularly when the editing and testing languages are non-English, requires a future study.

heading: 6 Analysis and Discussion

heading: 6.1 Ablation Study on Few-shot Editing
text: We have shown that ReMaKE-few-bi outperforms ReMaKE-zero significantly on the reliability and the generality scores. We conduct ablation studies in this subsection to delve deeper into the impact of two key factors on the performance of multilingual KE in few-shot editing: the quantity of bilingual examples and the size of backbone model.
The results of ReMaKE in response to the quantitative change in bilingual examples are illustrated in Figure 4(a) for “EN (edit) → ALL (test)”. The increase in the number of in-context examples is associated with an observable rise in the reliability score. The generality, locality, and portability scores are shown in Figure 8 in Appendix A.6.
We also analyze the effect of the backbone (i.e., BLOOMZ series) with different model sizes on 16-shot editing performance. Figures 4(b) and 12 demonstrate a convincing win of BLOOMZ-7b1 over its siblings with fewer numbers of parameters in all four metrics. Even though it is hard to differentiate the performance of BLOOMZ-1b1 and BLOOMZ-3b in some specific languages, they outperform BLOOMZ-560m in all four metrics. A scaling pattern is observed between the backbone model size and multilingual KE performance.

heading: 6.2 Ablation Study on Size of Knowledge Base
text: We conduct an ablation study to investigate the effects of the size of the knowledge base on a range of benchmark metrics, including the above-mentioned four metrics, retrieval accuracy and retrieval time consumed. A few-shot KE scenario (“EN (edit) → ZH (test)”) is performed on the LLaMA backbone.
It can be observed in Figure 5 that all benchmark metrics remain stable except for a minor decrease (-0.81% Reliability) with the increase in the size of the knowledge base. This decrease may be attributed to a minor degradation of the retriever’s accuracy (-1.61%) as the knowledge scale increases.

heading: 6.3 Computing Cost
text: We gather the time consumed in KE on “EN (edit) → ZH (test)” in Table 4 to show the computation cost associated with various editors. It is noted that the proposed ReMaKE-0shot achieves the second-best computation efficiency measured in time. SERAC, the top performer in this test, is not suitable for multilingual KE, making ReMaKE the state-of-the-art editor in terms of computational cost efficiency. ROME requires a substantial time to train and update the parameters through gradient calculation and back-propagation, resulting in the least efficient editor in this category.

heading: 7 Conclusion
text: In this paper, we propose ReMaKE, a retrievalaugmented multilingual knowledge editor, to update multilingual knowledge in LLMs by leveraging prompts composed of retrieved new knowledge and user inputs. To achieve multilingual
knowledge editing, we automatically construct the MzsRE dataset to cover 12 languages. ReMaKE is a model and language-agnostic knowledge editor not restricted to a specific LLM and language setting. Our experimental results show that ReMaKE achieves SOTA multilingual knowledge editing performance.
Limitations
As we extend the initial zsRE test set to implement a multilingual knowledge base of the proposed ReMaKE, the volume of the knowledge base is limited to 743 entries. Although ReMaKE can be easily scaled up to cope with real-world applications, the implication of implementing a large-capacity knowledge base on the proposed key metrics warrants a future study. A predefined question-andanswering template is used to define multilingual knowledge contained in the knowledge base. Future work will focus on developing a formal template to accommodate a more comprehensive scope of tasks. We found KE methods, in general, have low portability scores, indicating their limitations to impact knowledge that requires an LLM’s reasoning capability. How to perform multilingual knowledge editing to secondary associated knowledge needs further investigation.

heading: Acknowledgement
text: This project has received funding from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant numbers 10039436].
The computations described in this research were performed using the Baskerville Tier 2 HPC service (https://www.baskerville.ac.uk/). Baskerville was funded by the EPSRC and UKRI through the World Class Labs scheme (EP/T022221/1) and the Digital Research Infrastructure programme (EP/W032244/1) and is operated by Advanced Research Computing at the University of Birmingham.

heading: A Appendix
text: A.1 Statistics of MzsRE We list the statistics of MzsRE in 12 languages in Table 5.
A.2 Retriever Accuracy We further investigate the accuracy of the multilingual retriever of ReMaKE using sampled sentence pairs in the MzsRE dataset. The results are captured in Figure 6. The retriever achieves an accuracy of over 94% for all languages. The sub-optimal retrieval accuracy for some languages (i.e., Chinese, Russian) may contribute to the suboptimal performance of multilingual KE results in these languages.
Furthermore, we evaluate the retriever for the different test subsets (reliability, generality, locality, and portability) as shown in Table 6. The results demonstrate that the retrieval accuracy of portability subset is lower than other subsets, which means that the retriever lacks reasoning ability.
A.3 Exact Match Scores of ReMaKE with BLOOMZ Backbone
In order to compare the results of ReMaKE with different base LLMs, we implement ReMaKE on the
BLOOMZ backbone, and the exact match scores are show in Table 7 and Table 8.
A.4 Results of English-based multilingual KE with F1
We demonstrate the F1 scores of editing in English and testing in other languages and vise versa on the LLaMA and BLOOMZ backbones in Table 9 and Table 10.
A.5 Supplemental Results Multilingual KE We supplement the experimental results (Generality, Locality, Portability) of multilingual knowledge editing on ReMaKE-16shot-bi on the LLaMA backbone based on “ALL (edit) → ALL (test)” editing in Figure 7.
A.6 Effects of the Quantity of In-context Examples
We supplement the experimental results (Generality, Locality, Portability) of few-shot learning on ReMaKE-few-bi on the LLaMA backbone in Figure 8. We reach a similar conclusion with the finding obtained in subsection 5.1, in which ReMaKEzero takes the lead instead in the portability score as few-shot examples tend to introduce contextual interference to the KE process.
We supplement the comparison results of ReMaKE-few-bi on the BLOOMZ backbone under the 0-shot, 2-shot, 4-shot, 8-shot, 16-shot settings with editing in English and testing in other languages in Figure 9. Also we conduct the same setting with editing in other languages and testing in English in Figure 10 and Figure 11. From the results, it proves that few-shot learning could greatly improve the performance compared to zero-shot for the reliability and generality.
A.7 Supplemental Results of Model Size We supplement the experimental results (Generality, Locality, Portability) of different model sizes on ReMaKE-16shot-bi on the BLOOMZ backbone in Figure 12 with editing in English and testing in other languages.
A.8 In-context Examples Selection Method Liu et al. (2022) has demonstrated that the searchbased examples selection approach consistently outperforms the random selection baseline. All the above few-shot experimental results are conducted with the unsupervised prompt searching method. We compare the results of random selection and
search-based strategy for examples in Table 11. It follows the conclusion of Liu et al. (2022) that search-based selection could increase the performance (reliability score), such as from 41.45% to 67.97% on “EN(edit) → ES (test)”.

heading: Future Work
text: Although ReMaKE can be easily scaled up to cope with real-world applications, the implication of implementing a large-capacity knowledge base on the proposed key metrics warrants a future study. A predefined question-andanswering template is used to define multilingual knowledge contained in the knowledge base. Future work will focus on developing a formal template to accommodate a more comprehensive scope of tasks. We found KE methods, in general, have low portability scores, indicating their limitations to impact knowledge that requires an LLM’s reasoning capability. How to perform multilingual knowledge editing to secondary associated knowledge needs further investigation