Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?
Authors: Shaoyang Xu
Junzhuo Li
Deyi Xiong
Venue: EMNLP 2023 Main (2023)
Keywords: multilingual language models
factual knowledge
cross-lingual knowledge transfer

Abstract: Multilingual pretrained language models serve as repositories of multilingual factual knowledge. Nevertheless, a substantial performance gap of factual knowledge probing exists between high-resource languages and low-resource languages, suggesting limited implicit factual knowledge transfer across languages in multilingual pretrained language models. This paper investigates the feasibility of explicitly transferring relatively rich factual knowledge from English to non-English languages. To accomplish this, we propose two parameter-free $\textbf{L}$anguage $\textbf{R}$epresentation $\textbf{P}$rojection modules (LRP2). The first module converts non-English representations into English-like equivalents, while the second module reverts English-like representations back into representations of the corresponding non-English language. Experimental results on the mLAMA dataset demonstrate that LRP2 significantly improves factual knowledge retrieval accuracy and facilitates knowledge transferability across diverse non-English languages. We further investigate the working mechanism of LRP2 from the perspectives of representation space and cross-lingual knowledge neuron.

Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?
Shaoyang Xu', Junzhuo Li and Deyi Xiong SScooo of New Media and Communication, Tianjin University, Tianjin, China *College of Intelligence and Computing, Tianjin University, Tianjin, China [syxu, jzli, dyxiong) ) @ @tju.edu.cn
Abstract
between English and other languages in terms of the amount of factual knowledge captured for each language (Kassner et al., 2021; Jiang et al., 2020a).
Multilingual pretrained language models serve repositories of multilingual factual knowl- edge. Nevertheless, substantial perfor- mance gap of factual knowledge probing ex- ists between high-resource languages and low- resource languages, suggesting limited implicit factual knowledge transfer across languages in multilingual pretrained language models. This paper investigates the feasibility of explicitly transferring relatively rich factual knowledge from English to non non-English languages. To ac- complish this, we propose two parameter-fri free Language Representation Projection modules (LRP2). The first module converts non- English representations into English-lik, equivalents, while the second module reverts English- like representations back into representations of the corresponding non non-English language. Experi- mental results the mL.MMA dataset demon- strate that LRP2 significantly improves factual knowledge retrieval accuracy and facilitates knowledge transfcrability across diverse non- English languages. We further investigate the working mechanism of LRP2 from the perspec- tives of representation space and cross-lingual knowledge neuron.
Many works on cross-lingual transfer (Conneau et al., 2020; Chi al., 2021; Wu al., 2022; Yang et al.. 2022) validate the effectiveness of cross-lingual alignment of representation spaces in facilitating cross-linges looss-lingaal knowledge transfer. These studies primarily evaluate their methods on specific downstream tasks, including natural lan- guage inference (Conneau et al., 2018), sentence retrieval (Artetxe and Schwenk, 2019), question answering (Lewis et al., 2020) and text genera- tion (Wu al., 2022), etc.
Different from such studies, we focus on the task of factual knowledge probing in multilingual pre- trained language models and attempt to answer question in this paper: Can cross- linsull alignment of representation spaces enable factual knowledge transfer across languages? In particular, we ex- plore the feasibility of transferring factual knowl- edge from English to non-English languages.
To answer this question, propose LRP2, which incorporates two parameter-free Language Representation Projection modules into multilin- gual pretrained models: a language-independent representation projection module that projects rep- resentations of non-English languages into English- like representations and a tonguage-speciffe repre- sentation projection module that maps the English- like representations back to representations of indi- vidual non-English languages. These two modules, as depicted in Figure 1, locate different layers of Transformer.
1 Introduction
Previous studies demonstrate that language model is a knowledge base that can recall factual knowl- edge without additional fine-tuning (Petroni et al., 2019; Jiang et al., 2020b). This task of factual knowledge probing, aiming to examine what fac- tual knowledge language models capture during the pre-training phase, can be extended to multiple lan- guages in multilingual pretrained language models, e.g., mBERT (Devlin et al., 2019), XLM (Con- neau and Lample, 2019), mT5 (Xue et al., 2021), XGLM (Lin et al., 2022) and BLOOM (Scao et al., 2022). Although multilingual pretrained models serve as repositories of multilingual fac- tual knowledge, a factual knowledge gap exists
Experiments on mLAMA (Kassner et al., 2021) suggest that LRP2 improves factual knowledge trieval accuracy and facilitates knowledge transfer across diverse languages. We further conduct in- depth analysis to investigate the varying degrees of representation alignment required by different non- -on-English languages, as well as the transferabil-
Corresponding author.

---

ity of different types of factual knowledge. Delving into the working mechanism of LRP2, identify cross-lingual knowledge neurons multilingual pretrained language models.
Emboodiing
[MASK]
Feed Forward
[MASK]
x(L-j)
A
Multi-heud Atention
Our contributions are summarized as follows.
Baac England
ISPP
We propose a parameter-free framework LRP2 that enhances factual knowledge re- trieval accuracy and cross-lingual factual knowledge transfer.
Neuool English
Feed Ferward
X(-)
Memory Access
Memory Access
Multi-head Attention
cagioi
LIRP
Englani ot (MaS
We reveal that LRP2 poses impact the alignment of representation spaces and enhances the overlap of knowledge neurons across languages.
Feed Forward
x*
Multi-bead Attemtoo
di (MASK]
SE11 4 [MASK]
Embedding
LRP2
Basclinc
We discover that cross- looss-lingual knowledge neu- rons exist multilingual language models.
The diagram of the proposed LRP2 that inserts two language representation projection modules ad- ditional layers into the multilingual pretrained language model. The input question " [ á i # E [MASK]". u. U byy and vūn represent language vectors obtained for Chinese and English from the i-th and j-th layer of multilingual pretrained language model in advance. We Chinese to showcase our framework and our method is applicable to other languages in the same way. For simplicity, we ignore other sublayers in Transformer in the diagram. Note that our method is based on the core assumption that the representation spaces of two languages can be transferred through a Euclidean dis- tance mapping. This form of straightforward mapping relatively coarse, incapable of achieving the level of precise semantic transfer depicted in the figure, which is presented for the sake of illustration but may appear somewhat overly idealized.
Figure I:
2 Multilingual Factual Knowledge Probing
the multilingual factual knowledge probing task, multilingual pretrained language models take language-specifie fill-n-thhe-blank queries as input, such as "The capital of England is [MASK]" in En- glish, or the corresponding Chinese question " A # म क [MASK]". As knowledge base, the probed pretrained language model initially encodes the input query, then retrieves its parameterized memory and ultimately predicts an answer with a probability distribution over the vocabulary.
The success of factual knowledge transfer across languages relies on gegguage-indeendeenn repre- sentation space for different languages to trigger similar memories within the probed multilingual pretrained model and language-speciff represen- tations to allow the model to predict tokens in the corresponding language.
pretrained language model to probed. From the i-th layer of the model, we gather sentence-level vectors through mean-pooling over the representa- tions of tokens in the corresponding sentence. We then further average these sentence vectors, ob- taining of E R where is the hidden dimension of the model. In this way, we collect set of vectors Vi- [v] 2/ ..., of where denotes the number of layers of the model. These language vectors serve as the basis for language representation pro- jection within the proposed LRP2 framework.
3 LRP2
The primary objective of LRP2 to bridge the gap factual knowledge probing between English and non-English languages by aligning their represen- tation spaces.
Libovický et al. (2020) demonstrate that it is possible to induce language- representations for given language, by subtracting its correspond- ing language vector. The proposed LRP2 draws inspiration from this work and initiates its pro- cess by computing a set of language vectors V) for each language Specifically, for language I, we feed a set its sentences into the multilingual
As illustrated in Figure 1, LRP2 incorporates two language representation projection modules into the probed multilingual pretrained language model, which are referred as the Language-Independen Representation Projection (LIRP) module and the Language- Representation Projection (LSRP) module, respectively. These two modules

---

English (Source)
Language Family
Language Resource Medium
Model
Avg
Indo-Furopean
European Retrieval Accuracy
High
Low
MBERT
35.2
20.9
18.4
23.4
22.2
17.4
20.0
mBERT (LRP2)
35.2
21.2
19.4
24.1
23.0
17.7
20.6
BLOOM
35.1
17.8
18.4
21.7
17.2
16.1
18.0
BLOOM (LRP2)
35.1
21.3
22.4
25.8
21.2
19.3
21.7
English-centric Cross-lingual Transferability
BBERT
37.0
31.8
41.6
37.7
30.5
35.2
mBERT (LRP2)
37.9
33.1
43.1
31.5
BLOOM
20.4
20.3
25.7
19.3
17.6
20.4
BLOOM (LRP2)
24.5
24.7
30.3
24.0
21.4
24.6
Table 1: Evaluation results on mLAMA. We report factual knowledge retrieval accuracy and English-centric cross-lingual transferability. We list average results for Indo-European, non-Indo- ua--Ind-EEurpean, high-resource, medium- resource, low-resource and all non-English languages. We measure the amount of language resource based on the number of Wikipedia articles for each language.
are inserted into the model as two additional lay- ers. Representations of a non-English language with limited information are projected to the En- glish representation space by LIRP, which enables the non-English language access relatively rich memory encoded in the parameters of the model, in the form of English-like representations. The accessed memory is then projected back the non-English language by LSRP so that answers in the corresponding non-English language can be yielded.
where hf represent the j-th layer hidden states the input query in language I. h/ English-like representations because of the first projection. They are transformed back into the language I's repre- sentation space, resulting in h². These language- specific representations are further fed to the suc- ceeding layers of the model.
4 Experiments
Specifically, given input query in English language the LIRP first projects the con- textual representations from the i-th layer of the model into English-like representations, which can formulated as follows:
We conducted extensive experiments to examine the effectiveness of the proposed LRP2 framework in factual knowledge transfer across languages.
4.1 Settings
M = ni of vin (15i<L)
(1)
We utilized the TREx portion of mLAMA (Kass- et al., 2021) for experiments. Further in- formation regarding mLAMA and the dataset em- ployed to acquire language vectors can found in Appendix A.1. We calculated factual knowl- edge retrieval accuracy as well as English-centric cross-lingual transferability for each language. The details on these evaluation metrics can be found in Appendix A.2. The experiments were based two multilingual pretrained language models, mBERT! and BLOOM (the version with 559 million param- eters). The details of probing them can be found in Appendix A.3. Note that the i-layer for inserting LIRP and the j-layer for inserting LSRP are two hyperparameters, the details on the setting of them can be found in Appendix A.4.
where hi represent the i-th layer hidden states of the input query in language l. of and v denote the language vectors of the i-th layer for non-English language 1 and English respectively. By performing this projection, the representations of non-English language are mapped into the English space and subsequently fed the succeeding layers.
As mentioned in Section 2, in the multilingual factual knowledge probing task, it is essential for the multilingual pretrained language model to yield answers in the corresponding language. To recover the language-specinn information of the input lan- guage, we insert the LSRP into the j-th layer of the model. The pack-proeection the input language is formulated as:
hf = h/ 2 + o/ (i<j<n)
(2)

---

5.1 LRP2 Affects the Alignment of Representation Spaces across Languages
Lshonet proeqee
pryedes
We utilized Chinese-English parallel queries in the mLAMA dataset collect sentence representa- tions and further calculated the layer-wise cosine similarity of these two languages' sentence repre- sentations, as the distance between the representa- tion spaces of these two languages. We conducted comparative analysis of the distance with and without the utilization of LRP2.
L.011
(a) Representation Spaces
(b) Knowledge Neurons
Figure 2: Distance between representation spaces and overlap rate of knowledge neurons in different layers. The languages considered here are Chinese and English. 'mBERT (LRP2)' represents the results with the inser- tion of the LIRP module into the 4-th layer and the LSRP module into the 10-th layer of mBERT, which yields the best transferability result of Chinese.
Figure 2a presents the distance between the rep- resentation spaces of Chinese and English. It clearly shows the distinct functions of LIRP and LSRP. Specifically, the LIRP module first brings Chinese sentences closer to the representation space of English, thereby facilitating cross-lingual knowledge transfer, while the LSRP module in- creases the distance between Chinese sentences and the representation space of English, inducing language-specific outputs Chinese.
Model
Same
Different
Avg 11.7%
mBERT
17.9%
11.5%
MBERT (LRP2)
18.5%
11.9%
12.1%
Table 2: Overlap rate of knowledge neurons for factual relations Chinese and English.
5.2 LRP2 Enhances the Overlap of Knowledge Neurons across Languages
4.2 Results
Table presents the experimental results on mLAMA, it shows that LRP2 achieves significant improvements in terms of both factual knowledge retrieval accuracy and cross-lingual transferability across various non-English languages over the base- line. The results indicate that cross- loos-linguul align- ment of representation spaces indeed facilitates the transfer of rich factual knowledge from English to non-English. More specifically, for both mBERT and BLOOM, LRP2 demonstrates better perfor- mance in certain non-Indo- Europenn languages as well as medium- and high-resouree languages. Additional experimental results on X- FACTR (Jiang et al., 2020a) are provided in Appendix B.1. To provide further insights, present the performance changes for different languages as the number of layers between LIRP and LSRP varies in Appendix B.2. The specific effects of LRP2 on different non-English languages are provided in Appendix B.3. In addition, we observe that the transferability of knowledge shows variations across different types of factual relations, evidenced in Appendix B.4.
Dai al. (2022) discover that knowledge neurons expressing specific factual knowledge exist in pre- trained Transformers. Building upon their work, we identify knowledge neurons in multilingual pre- trained Transformers and employ them to elucidate the working mechanism of LRP2. The details on how we identify knowledge neurons in multilin- gual pretrained language models are provided in Appendix C.
Table showcases the overlap rate of knowledge neurons for factual relations in Chinese and En- glish. Notably, we have two interesting findings. First, the overlap rate of knowledge neurons as- sociated with the same relations is considerably higher compared to that with different relations, suggesting the existence of language-inddepedeen knowledge neurons within mBERT. Second, LRP2 increases the overlap rate of knowledge neurons between Chinese and English. This improvement indicates that LRP2 facilitates the alignment of En- glish and non non-English representation spaces and enhances the activation of knowledge neurons in non-English languages, making them more simi- lar those in English. In this way, non-English languages acquire factual knowledge transferred from English. Additionally, Figure 2b visualizes the overlap rate knowledge neurons across differ- ent layers. Notably, the layers between LIRP and
5 Working Mechanism of LRP2
this section, we study the working mechanism of LRP2 from the perspectives of representation space and knowledge neuron.

---

LSRP exhibit a prominent increase in the overlap rate of knowledge neurons between Chinese and English.
ment of English and non-Einglish representation spaces brought by LRP2 can help non- -on-Englihh lan- guages to stimulate knowledge neurons similar to English, thereby acquiring knowledge transferred from English.
6 Related Work
Factual Knowledge Probing Previous works (Petroni et al., 2019; Jiang et al., 2020b) have shown that a language model a knowledge base. Subsequent works (Kassner et al., 2021; Jiang et al., 2020a) extend monolingual factual knowledge probing to multiple languages. Notably, Jiang et al. (2020a) improve multilingual factual knowledge probing in code-switching style. Significantly different from this, suggest that it is essential allow multilingual pretrained language models to yield language-specific answers.
Limitations
While LRP2 significantly improves factual knowl- edge retrieval accuracy and facilitates knowl- edge transfcrability across diverse non-English lan- guages, is noteworthy that the LIRP and LSRP modules in LRP2 are inserted into muitilingual pre- trained language models as two additional layers. Thus, the effectiveness of LRP2 heavily relies on the inherent capabilities of multilingual pretrained language models.
Through extensive experiments conducted on the proposed LRP2 framework, we have demon- strated that cross-lingual alignment of represen- tation spaces enables factual knowledge transfer across different languages. Although this finding is applicable to multilingual pretrained language models of varying architectures, experiments are limited two relatively small models due the limited compute resource available us. We plan to investigate LRP2 on larger language models when more compute resource available.
Model Editing A variety of approaches have been proposed to edit knowledge in monolingual language models (Sinitsin al., 2020; Cao al., 2021; Mitchell et al., 2022; Meng al., 2022; Dai et al., 2022). Recently, Xu et al. (2022) define a cross-lingual model editing task, where knowledge updates in one language need to occur in other lan- guages as well. In this paper, we focus on factual knowledge that already exists in multilingual lan- guage models and enhance the transfcrability of them, rather than trying update a model with new knowledge.
A Acknowledgements
The present research supported by Zhejiang Lab (No. 2022KHOAB01), We would like to thank the anonymous reviewers for their insightful com- ments.
Cross-lingual Knowledge Transfer Cross- lingual transfer learning approaches are usually categorized into instance transfer (Zheng et al., 2021; Yang al., 2022), parameter transfer (Chen al., 2019; Zhou et al., 2019), and feature transfer (Libovicky
References
al., 2020; Zhao al., 2021). Most of these works explore cross- cross-lingual knowledge transfer on specific downstream tasks, while we focus on factual knowledge captured by language models and explore the possibility of cross-lingual factual knowledge transfer.
Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguistics, 7:597-610.
Nicola De Cao, Wilker Aziz, and Ivan Titov, 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7- 11 November, 2021, pages 6491-6506. Association for Computational Linguistics.
7 Conclusion
We have presented a simple yet effective method to transfer factual knowledge from English to non- English languages in multilingual pretrained lan- guage models. We empirically confirm that cross- lingual alignment of representation spaces enables factual knowledge transfer across languages in mul- tilingual pretrained language models. Further anal- ysis on knowledge neurons shows that the align-
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-souree cross-lingual model transfer: Learning what to share. In Proceedings of the 57th Conference of the Associ- ation for Computational Linguistics, ACL 2019, Flo- rence, Italy, July 28- August 2, 2019, Volume I: Long

---

Papers, pages 3098-3112, Association for Computa- tional Linguistics.
16-20, 2020. pages 9943-5959, Association for Com- putational Linguistics.
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song. Xian-Ling Mao, Heyan Huang. and Ming Zhou. 2021. Infoxlm: An coormatiio-theorette
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig, 2020b. How can we know what language models know. Trans. Assoc. Comput. Linguisties, 8:423-438.
framework for cross-ling minssllingual language model pre- vre-trainng. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, NAACL-HLT 2021, On- line, June 6-11, 2021. pages 3576-3588, Association for Computational Linguistics.
Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: investigating knowledge in multilingual pretrained language models. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 3250-3258, Association for Com- putational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2020, On- line, July 5-10, 2020, pages 8440 8451. Association for Computational Linguistics.
Patrick S. H. Lewis, Barlas Oguz, Ruty Rinott, Sebas- tian Riedel, and Holger Schwenk. 2020. MLQA: evaluating cross-lingual extractive question answer- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020. Online, July 5-10, 2020, pages 7315-7330. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems 32: An- nual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057-7067.
Jindrich Libovicky, Rudolf Rosa, and Alexander Fraser. 2020. On the language neutrality of pre-trained multilingual representations. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACI., pages 1663-1674. Association for Computational Linguistics.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: evaluating cross- lingual sentence representations. In Proceedings of
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' OoHoro, Jeff Wang, Luke Zettle- moyer, Zornisaa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages Linguistics.
the 2018 Conference on Empirical Methods in Natu- ral Language Processing, Brussels, Belgium, Octo- ber November 4. 2018, pages ciation for Computational Linguistics.
Asso-
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493- 8502. Association for Computational Linguistics.
Association for Computational
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HIIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual knowl- edge in GPT. CoRR, abs/2202.05262.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022. Fast model editing at scale. The Tenth International Con- ference on Learning Representations, ICLR 2022, Virtual Event, April 25.29, 2022. OpenReview.net.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language mod- els as knowledge bases? In Proceedings of the 2019 Conference rul Language Processing and the 9th International Joint Conference on Natural Language Processing,
Zhengbao Jiang, Antonios Anastasopoulos. Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: multilingual factual knowledge retrieval from pre- trained language models. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2020, Online, November
Empirical Methods in Natu-

---

EMNLP-IJCNLP" 2019, Hong Kong, China, Novem- ber 3-7, 2019. pages 2463-2473, Association for Computational Linguistics.
Biao Zhang, Philip Williams, Ivan Titow, and Rico Sen- nrich. 2020. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1628-1639. Associa- tion for Computational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili., Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang. Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier. Samson Tan, Pedro Ortiz Suarez, Vic- tor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji. Amit Alfassy. Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/221 1. 1.05100.
Wei Zhao. Steffen Eger, Johannes Bjerva, and Isabelle Augenstein. 2021. Inducing language-agnoss mul- tilingual representations. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, *SEM 2021, Online, Au- gust 5-6, 2021, pages 229-240. Association for Com- putational Linguistics.
Zheng. Li Dong. Shaohan Huang. Wenhui Wang- Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Consistency reg- ularization for cross- gooss-lnngul fine-tume tine-tuning ings of the 59th Anmual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing, ACIAJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 3403-3417. Associa- tion for Computational Linguistics.
Proceed-
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 36-30, 2020. OpenRe- view.net.
Joey Tianyi Zhou, Hao Zhang, Jin, Hongyuan Zhu, Meng Fang, Rick Siow Mong Goh, and Kenneth Kwok. 2019. Dual adversarial neural transfer for low- resource named entity recognition. Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers, pages 3461-3471, Association for Computational Linguis- tics.
Xianze Wu, Zaixiang Zheng, Hao Zhou, and Yong Yu. 2022. Laft: Cross-lingual transfer for text generation by language-agnostic finetuning. In Proceedings of the 15th International Conference on Natural Lan- guage Generation, pages 260-266.
Yang Xu, Yutai Hou, and Wanxiang Che. 2022. Lan- guage anisotropic cross- lonsull model editing. CoRR, abs/2205.12677
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLI 2021, Online, June 6-11, 2021, pages 483-498 498. Association for Computational Linguistics.
Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. 2022. Enhancing cross-lingual transfer by manifold mixup. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event. April 25-29, 2022. OpenReview.net.
Da Yin, Hritik Bansal, Masoud Monajatipoor, Liu- nian Harold Li, and Kai- Wei Chang. 2022. Geom- lama: Geo-diverse commonsense probing on multi- lingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2039-2055. Association for Computational Linguis- tics.

---

A Experiment Details
English (Source)
xh
ko
ml
ceb
A.1 Datasets
Accwrecy
MBHKT
22.6
14.4
12.2
18.3
22.8
14.3
10.6
MARRT ( (RP)))
22.6
15.4
13.3
18.8
23.3
15.6
12.8
mLAMA (Kassner et al., 2021) is a multilingual factual knowledge probing dataset containing 53 languages and 44 factual relations, and the TREx part contains 41 of them. To obtain language vec- tors, used OPUS-100 (Zhang al., 2020) to collect 10,000 filtered sentences for most of the 53 languages, and for languages not included in OPUS- such as ceb, we obtained data from the OPUS.
Fuglisb-centric
Choss-lingull Phonsrrablliy
mARRT
30.0
24.9
48.5
46.4
25.4
23.4
mBERTT (LRP2)
32.6
28.6
48.7
47.0
25.3
30.1
Table 3: Evaluation results of mBERT on X-FACTR.
Probing BLOOM We notice that the objects to be predicted can appear in the middle of the corresponding query templates in the mLAMA dataset. However, due the pre-training task of causal language modeling, autoregressive models like BLOOM are more adept at answering factual knowledge questions by predicting the next token in a given query. To address the mismatch be- tween the form of query templates in mLAMA and the generative nature of BLOOM, we employed compromise approach inspired by Yin al. (2022). Specifically, when probing the autoregres- sive BLOOM, we filled each query with objects from its candidate pool construct complete sen- tences. We then calculated the model's generation probabilities for these sentences, which serve as the prediction probabilities for different objects. Due to limitation of compute resource, we restricted the size of the candidate pools to 10 when probing BLOOM.
A.2 Evaluation Metries
We calculated factual knowledge retrieval accuracy for each language l as Ace; 2D:1
[R, 100, where R, represents the set of correctly predicted knowl- edge for language and D2 represents the entire probing data for language Additionally, we cal- culated English-centric cross-lingual transferability R
100. Here, the denom- Trans; RRUURen
inator R2 U Real corresponds to the amount of knowledge stored in the probed model, whether in non-English language I or in English form, while the numerator [R4 n Renl represents the amount of the stored knowledge both in the form of language and English, indicating the amount of transferable knowledge.
A.3 Probing mBERT and BLOOM
A.4 Hyperparameters
Following mLAMA (Kassner et al., 2021), adopted a typed querying approach for probing. This entails considering all candidate objects of relation as the candidate pool. For each query asso- ciated with a specific relation, we determined the ranking of the correct answer within its candidate pool. The prediction is considered correct if the сorrect answer is ranked at the top position.
The i-layer for inserting LIRP and the j-layer for inserting LSRP are two hyperparameters. We sys- tematically evaluate different combinations of them for each language and report the best results. This exploration allows to investigate the potential for cross-lingual factual knowledge transfer facilitated by the alignment of representation spaces.
Probing mBERT When probing mBERT, the input query follows the format like "The capital of England is [MASKJ". the model's 's probability predictions for the [MASK] tokens are used com- pute the ranking. The number of [MASK] tokens depends on the length of the tokenized object to be predicted. cases of multiple [MASK] tokens, we calculated the average log probability of these tokens. We utilized the complete candidate pools for probing mBERT (with average number of candidates per relation of approximately 90).
B Additional Results
B.1 Experiments on X-FACTR
Yet another dataset used to probe multilingual fac- tual knowledge is X-FACTR (Jiang et al., 2020a). In contrast to mLAMA, this dataset contains fewer languages and slightly more factual relations (23 and 46, respectively). We supplemented experi- ments on 6 languages of X-FACCR, using mBERT as the baseline model. The results are listed in Table 3, which shows that LRP2 can also achieve improvements on the X-FACTR dataset.
hhtts:///pus.nlpp.cu

---

worthy that LRP2 is effective for languages that are not covered by the training data of BLOOM. This can be attributed to BLOOM's s utilization of a byte-level BPE algorithm for subword tokeniza- tion (Scao et al., 2022), ensuring that unknown tokens are never yielded. In this way. unknown lan- guages can be effectively represented to certain extent, enabling the transfer of factual knowledge between them and other languages.
(8) MBRRT
(b) BLOOM
Figure 3: Cross- transferability change with the number of layers between LIRP and LSRP for ru, ja, eu, sk and be. Note that one-to-many relationship exists between the number of the layers and the configu- ration LIRP and LSRP (i.e., the same difference of numbers corresponds multiple pairs of numbers). We display the best results for each specific number of layers here.
B.4 The Transferability Varies across Factual Relations
We assess the transferability change of each fac- tual relation in every language and consider a fac- tual relation to be transferable from English to non-English language if its transferability im- proves under any configurations of the LIRP and LSRP modules. Figure 6 illustrates the transfer- able percentages across all factual relations for mBERT. We observe that 37 out of 41 relations exhibit transferability from English over 80% non- MonnEEnlishh languages. Notably, the relations P17, P1412, and P138, representing Place (e.g., Ger- many, Ireland) and Language (e.g., Italian, Span- ish) demonstrate consistent transferability across all languages. However, some factual relations dis- play lower transferability, e.g., P413, P264, P140, and P108, which represent Athlete Position (e.g.. midfielder, pitcher), Orgamization (e.g., Decca, Mo- town), Religion (e.g.. Buddhism, Islam) and Orga- nization (e.g., Apple, Microsoft), respectively. In addition, Figure 7 reveals a similar trend in the transfcrability of factual relations between BLOOM and mBERT. Specifically, the factual lations P264, P413 and P449 exhibit lower transfer- ability, while relations representing Place or Lan- guage, such as P937, P530, P407, P37, and so on, demonstrate higher transferability in BLOOM.
B.2 Different Languages Necessitate Varying Optimal Layer Settings
Figure 3 presents the change of cross-lingual trans- ferability for five languages the number of lay- ers between LIRP and LSRP varies. Notably, observe that different languages exhibit distinct requirements for representation space alignment to achieve optimal transferability. addition, we notice that the performance of certain languages is very sensitive to the choice of model layers for the insertion of LIRP and LSRP modules. For cer- tain numbers of layers between LIRP and LSRP for some languages, such as 9 for language in Figure 3a, none of the particular insertion settings (the layers where LIRP and LSRP are inserted into are 1/10, 2/11, 3/12, respectively) lead to efficient knowledge transfer. We hypothesize that such sen- sitivity may stem from the relatively fragile nature of the representation space learned by mBERT for these languages. Consequently, the representations of these languages could easily lose semantic in- formation and become meaningless after language representation projections, leading complete failure of knowledge transfer.
C Identifying Knowledge Neurons in Multilingual Pretrained Models
In addition, Table 4 and Table 5 show mBERT's and BLOOM'ss optimal layer configurations for all languages respectively, further underscoring the substantial disparity in the optimal layer settings among various languages.
We identify knowledge neurons in multilingual pretrained models using Knowledge Attribution proposed by Dai et al. (2022). We first identify the knowledge neurons of all prompts in re- lation. Specifically, for each prompt, we calcu- late the knowledge attribution scores of neurons and take top-20 neurons as its knowledge neu- rons. Further, for each factual relation, we take the top-20 neurons with the highest number of oc- currences in its all prompts as knowledge neurons
B.3
The Impact of LRP2 Differs across Non-English Languages
Figure 4 and Figure illustrate the specific ef- fects of LRP2 on different non- non-Engiin non-English languages for mBERT and BLOOM respectively. It is note-

---

66
15
kr
14
Table 4: mBERT's optimal layer configurations for all languages, 'LIRP' indicates which layer of mBERT the LIRP module is inserted into, 'LSRP" follows the same pattern.
H mb
bx
SSP
3
20
34 18
18
32
12
31
31
2
Table 5: BLOOM's optimal layer configurations for all languages. 'IJRP" indicates which layer BL.OMM the LIRP module is inserted into, 'LSRP" follows the same pattern.
it. For a language, we identify knowledge neu- rons of all its factual relations in mLAMA, such as P101. P.7. etc. Unlike Dai et al. (2022), we perform score ranking at each layer of the model, i..., for factual relation, we obtain its knowledge neurons in all layers, c.g., KWV (KNN KCN Pio ..., KN P.or.. where L is the number of layers of pretrained language models. Specifically, we identified knowledge neurons for both Chinese and English in mBERT. For Chinese, we additionally detected knowledge neurons under the configuration that yields the best transfcrability result of Chinese.

---

Cooseirgual Carbaaiityy
4%
0%
Dravidion
Korcanic
Indo-European
Austro-Asittic
Austronesian
Afro-Asiatic
Uralic
Figure 4: The effect of LRP2 on English-centric cross-lingual transferability of different non-English languages. Results are based on mBERT.
Chos-lingnaln change
10%
88%
6%
25
07%
Kartvelion
Sino- Tihetan
Kra-Dai
Turkic
Japonic
Dravidian
Koreanic
Ludo-European
Austr-Assitic
Austronesian
Afre-Asiatic
Uralic
Figure 5: The effect of LRP2 on English-centric cross lingull transferability of different non non-Eng Enn--Enllshhh languages. Results are based BLOOM. Note that the training data for BLOOM cover only 14 of all languages in the mLAAM dataset, which are marked with an asterisk (*),
Transienble parentiges
80%
40%
20%
014
Langunge
Occupation
Religion
Music Genze
Athlete position
Place
Organization
Spocialization
Musical Instrument
Undefined
Figure 6: Transferable percentages of all factual relations in mLAMA. Results are based on mBERT.
100%
Trunsteablee perennags
30%
40%
20%
Language
Occupation
Religion
Music Geure
Athlete position
Place
Organization
Spccialization
Musical Instrument
Underined
Figure 7: Transferable percentages of all factual relations in mL.AM. Results are based on BL.OM.


Reviews:
[{"review_id": "4HJ7fEbIyu", "reviewer": "Reviewer_iytb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable feedback and your interest in our methodology and findings. We present our responses to your comments below.\n\n1. The proposed approach should be evaluated through more zero-shot cross-language transfer experiments, extending beyond just mLAMA.\n\n- Another dataset used to probe multilingual factual knowledge is X-FACTR[1]. In contrast to mLAMA, this dataset contains fewer languages and slightly more factual relations - 23 and 46, respectively. Due to time limitations, we have supplemented experiments on 6 languages of X-FACTR, using mBERT as the baseline model.  The results are listed below：\n\n  - Retrieval Accuracy：\n\n  |             | English(source) | zh       | ko       | nl       | vi   | ceb      | ja       |\n  | ----------- | --------------- | -------- | -------- | -------- | ---- | -------- | -------- |\n  | mBERT       | 22.6            | 14.4     | 12.2     | 18.3     | 22.8 | 14.3     | 10.6     |\n  | mBERT(LRP2) | 22.6            | **15.4** | **13.3** | **18.8** | 23.3 | **15.6** | **12.8** |\n\n  - English-centric Cross-lingual Transferability：\n\n  |             | English(source) | zh       | ko       | nl       | vi       | ceb  | ja       |\n  | ----------- | --------------- | -------- | -------- | -------- | -------- | ---- | -------- |\n  | mBERT       | 1               | 30.0     | 24.9     | 48.5     | 46.4     | 25.4 | 23.4     |\n  | mBERT(LRP2) | 1               | **32.6** | **28.6** | **48.7** | **47.0** | 25.3 | **30.1** |\n\n- It shows that LRP2 can also achieve improvements on X-FACTR dataset. The results of the  X-FACTR experiment will be included in Appendix B.4 of the updated version.\n\n[1] X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models\n\n2. Figure 2 lacks detailed analysis, such as the explanations for the languages (e.g., \"eu\" and \"sk\") with significant performance fluctuations across layers.\n\n- Previous works[1,2] find that different layers within multilingual pretrained models have very different effects on cross-lingual transfer. Through Figure 2, we aim to provide further insight into the phenomenon that, even within the same multilingual pretrained model, different languages necessitate varying numbers of model layers to achieve optimal cross-lingual knowledge transfer.\n- The phenomenon of \"significant performance fluctuations across layers\", notably witnessed in certain languages (e.g. \"eu\" and \"sk\"), illustrates that the performance of these languages is very sensitive to the choice of model layers to insert LIRP and LSRP modules. For certain numbers of layers between LIRP and LSRP for some languages, such as 9 for language \"eu\" in Figure 2(a), none of the particular insertion settings (the layers where LIRP and LSRP are inserted into are 1/10, 2/11, 3/12,  respectively) lead to efficient knowledge transfer.\n- This sensitivity may stem from the fact that the representation space learned by the multilingual pretrained model for these languages is relatively vulnerable, and only representation projections at specific locations can preserve the meaningfulness of the representations, thus enabling successful knowledge transfer. Conversely, in other settings, the representations of these languages easily lose semantic information and become meaningless, leading to a complete failure of knowledge transfer.\n- We will make the following changes to the content of the paper：\n  - To enhance the readability of the main body, we will relocate Figure 2 and a detailed analysis of it into Appendix B.5.\n  - We will delve deeper into our findings regarding the central question explored in this paper in section 4.2, and briefly mention the phenomenon of \"significant performance fluctuations across layers\" introduced by LRP2 there.\n\n[1] INFOXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training\n\n[2] Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology\n\n\n\n3. The paper does not provide a recommendation for selecting the number of layers between LIRP and LSRP, which is a critical hyperparameter affecting performance across different languages and tasks. This lack of guidance limits the application and further extension of the proposed method.\n\n- As described in Appendix A.4, we conducted a grid search to determine the optimal layer configuration for each language. Certain configuration details are presented below, taking into account the 12 layers of mBERT, numbered from 1 to 12, the term 'LIRP' in the table below indicates which layer of mBERT the LIRP module is inserted into, 'LSRP' follows the same pattern, and 'layer_num' indicates the number of layers between these two modules.\n\n  - mBERT's optimal layer configuration on mLAMA\n\n  |           | zh   | ko   | nl   | vi   | ceb  | ja   |\n  | --------- | ---- | ---- | ---- | ---- | ---- | ---- |\n  | LIRP      | 4    | 1    | 8    | 1    | 1    | 3    |\n  | LSRP      | 10   | 3    | 9    | 2    | 2    | 11   |\n  | layer_num | 6    | 2    | 1    | 1    | 1    | 8    |\n\n  - mBERT's optimal layer configuration on X-FACTR\n\n  |           | zh   | ko   | nl   | vi   | ceb  | ja   |\n  | --------- | ---- | ---- | ---- | ---- | ---- | ---- |\n  | LIRP      | 8    | 8    | 4    | 1    | 3    | 3    |\n  | LSRP      | 11   | 12   | 5    | 2    | 4    | 11   |\n  | layer_num | 3    | 4    | 1    | 1    | 1    | 8    |\n\n- First, both tables above underscore the substantial disparity in the optimal layer settings among various languages. Unfortunately, we haven't been able to establish a clear correlation between the specific optimal layer settings (or the number of layers between the two modules) of different languages and their inherent language characteristics, such as language family and language resources, which hinders us from providing precise recommendations for hyperparameter selection.\n\n- Second, we observe that for some languages, like vi and ja, mBERT's optimal layer settings remain consistent across different datasets. This indicates that for certain languages, the identified hyperparameter are dataset-independent and could potentially be applied across diverse tasks to facilitate cross-lingual knowledge transfer.\n\n- We will include the optimal layer settings and our findings in Appendix A.4 of the updated version.\n\n\n\n4. The method relies on strong assumptions, assuming similar distributions of different language spaces. & The Euclidean distance mapping used in LSRP and LIRP fails to adequately represent the semantic transfer of a sentence between languages.\n\n- The core assumption behind our method is that through a Euclidean distance mapping, the representation space between language A and language B can be transferred. This kind of simple mapping is certainly relatively coarse and cannot achieve precise semantic transfer. Perhaps the over-idealization of the method's working mechanism in Figure 1 has led to misunderstandings. We will provide a precise clarification of our core assumption in the description section of Figure 1.\n\n- To explore the underlying mechanism of LRP2, we conducted an analysis of multilingual knowledge neurons in section 5. The analysis results demonstrate that LRP2 can enhance the overlap of knowledge neurons across languages, revealing the actual working mechanism of LRP2.\n\n- Furthermore, we have expanded our analysis by utilizing mLAMA to calculate the layer-wise cosine similarity of sentence representations of parallel Chinese and English queries. The results are detailed below, where \"mBERT\" denotes the direct calculation on mBERT, and \"mBERT(LRP2)\" represents the statistical outcome with the insertion of the LIRP module into the 4-th layer and the LSRP module the 10-th layer of mBERT.\n\n  |             | 1    | 2    | 3    | 4        | 5    | 6    | 7    | 8    | 9    | 10       | 11   | 12   |\n  | ----------- | ---- | ---- | ---- | -------- | ---- | ---- | ---- | ---- | ---- | -------- | ---- | ---- |\n  | mBERT       | 0.70 | 0.75 | 0.86 | 0.88     | 0.91 | 0.92 | 0.88 | 0.87 | 0.90 | 0.91     | 0.90 | 0.55 |\n  | mBERT(LRP2) | 0.70 | 0.75 | 0.86 | **0.90** | 0.93 | 0.93 | 0.89 | 0.88 | 0.92 | **0.89** | 0.89 | 0.51 |\n\n- The above results illustrate that LRP2 first diminishes the disparity between semantic representations of different languages and then increases it, providing another explanation for the mechanism behind LRP2. We will include the analysis of representation space of different languages in section 5.\n\nWe present a summary of our responses below.\n\n- We have supplemented experiments on another multilingual factual knowledge probing dataset to further demonstrate the effectiveness of the method.\n- We will move Figure 2 to Appendix B.5 and analyze it in more detail, such as explaining the phenomenon of \"significant performance fluctuations across layers\"\n- We will include the optimal layer settings and our findings in Appendix A.4 of the updated version.\n- We will add an analysis of the similarities between representations of different languages to further explore the underlying mechanism of LRP2 in section 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents interesting findings, such as the existence of cross-lingual knowledge neurons in multilingual pre-trained models.\n\n2. The proposed method is simple and demonstrates certain performance improvements.\n\n3. The paper is generally well-written.", "reasons_to_reject": "1.\tThe proposed approach should be evaluated through more zero-shot cross-language transfer experiments, extending beyond just mLAMA.\n2.\tFigure 2 lacks detailed analysis, such as the explanations for the languages (e.g., \"eu\" and \"sk\") with significant performance fluctuations across layers.\n3.\tThe paper does not provide a recommendation for selecting the number of layers between LIRP and LSRP, which is a critical hyperparameter affecting performance across different languages and tasks. This lack of guidance limits the application and further extension of the proposed method.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the feasibility of transferring factual knowledge from English to non-English languages in multilingual language models. The authors propose a parameter-free framework called Language Representation Projection (LRP2) that aims to improve the transferability of factual knowledge across diverse non-English languages. The results show some performance improvements, but the method relies on strong assumptions, assuming similar distributions of different language spaces. Additionally, the Euclidean distance mapping used in LSRP and LIRP fails to adequately represent the semantic transfer of a sentence between languages. Furthermore, the experiment only utilizes mLAMA, and the exploration of tasks is limited, which raises concerns about the persuasiveness of the findings.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6853hiAA4v", "reviewer": "Reviewer_fiJ8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your time to review our paper and that you find value in our work. We agree that there are some problems that need to be solved in this work. We respond to your comments below.\n\n1. Some of the core claims of the paper cannot be assessed without reading the appendices (see especially sect. 4.2 and 5). This should really be a long paper.\n\n- The core content of this paper is straightforward and can be distilled into two key points: 1. we propose a simple method based on language representation projection to explore the feasibility of cross-lingual factual knowledge transfer. 2. we investigate the mechanism behind cross-lingual knowledge transfer from different perspectives. Such clarity makes it easy for us to adjust the structure of the paper.\n- First, we will streamline the paper's content without losing readability as follows:\n  1. Remove details about the proposed method in section 1, and focus the introduction on the background and contributions of this work.\n  2. Integrate section 2, \"Multilingual Factual Knowledge Probing,\" into section 1 as a part of the background context.\n  3. Relocate Figure 2 in the main body and the description of it in section 4.2 to Appendix B.5.\n- Second, we will harness any available space, including an additional fifth page (assuming acceptance), to make the following changes to the main body:\n  1. Present more precise descriptions of method performance and our core findings in Section 4.2.\n  2. Move our insights on knowledge neurons from Appendix C.2 to section 5.\n  3. Supplement the analysis of representation space in section 5, including calculations of cosine distance variations between representations of different languages with and without LPR2, as well as 2-D visualizations of sentence representations.\n- Considering the core content of this paper is clear and simple, we believe it is more suitable for publication as a short paper. To ensure greater clarity and readability, we will make content adjustments that effectively present the paper's central concepts.\n\n2. The core assumption underlying the proposed method (see above) should be verified or at least motivated on the basis of previous work on cross-lingual representation learning. For instance cf. the strong claim \"By performing this projection, the representations of non-English language l are mapped into the English space\" in line 164-165.\n\n- First, there are previous studies[1,2] supporting this assumption, and the work we follow to obtain language vectors[1] is one of them, which demonstrates that language-neutral representations can be induced for language X, by subtracting its corresponding language vector. Thereby, there is a certain degree of confidence that another addition of the language Y's language vector to these neutralized representations can futher map language X into the space of language Y. We will explain the motivation behind our method in section 3.\n\n- Second, as suggested by you, we have expanded our analysis by utilizing mLAMA to calculate the layer-wise cosine similarity of sentence representations of parallel Chinese and English queries. The results are detailed in the table below, where \"mBERT\" denotes the direct calculation on mBERT, and \"mBERT(LRP2)\" represents the statistical results with the insertion of the LIRP module into the 4-th layer and the LSRP module the 10-th layer of mBERT, which yields the best transferability result of Chinese.\n\n  |             | 1    | 2    | 3    | 4        | 5    | 6    | 7    | 8    | 9    | 10       | 11   | 12   |\n  | ----------- | ---- | ---- | ---- | -------- | ---- | ---- | ---- | ---- | ---- | -------- | ---- | ---- |\n  | mBERT       | 0.70 | 0.75 | 0.86 | 0.88     | 0.91 | 0.92 | 0.88 | 0.87 | 0.90 | 0.91     | 0.90 | 0.55 |\n  | mBERT(LRP2) | 0.70 | 0.75 | 0.86 | **0.90** | 0.93 | 0.93 | 0.89 | 0.88 | 0.92 | **0.89** | 0.89 | 0.51 |\n\n- It illustrates that LRP2 indeed maps the representations of language X to the space of language Y. Furthermore, 2-D visualizations of sentence representations can also confirm this.\n\n- We will augment section 5 with the verification of the assumption that LPR2 can map representations of language X into the space of language Y.\n\n[1] On the Language Neutrality of Pre-trained Multilingual Representations\n\n[2] A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations\n\n3. The reported improvements are very modest for mBERT. They are better for BLOOM but the baseline scores for this model are very low. & The paper states in different places that the proposed method \"significantly\" improves factual knowledge accuracy, however said improvements are either very small (for mBERT) or mild on top of a weak baseline (for BLOOM) and I see no mention of statistical significance.\n\n- First, for mBERT, the average English-centric Cross-lingual Transferability without and with LRP2 is 35.2 and 36.3 respectively, indicating that the improvement brought by LRP2 is indeed modest. However, as depicted in Figure 4, LRP2 demonstrates its capacity to yield enhancements across the majority of languages (41 out of 52), showing the noteworthy breadth of its effectiveness. We will provide a more precise description of LRP2's performance.\n\n- Second, we apologize for the oversight during our experimentation on BLOOM. Specifically, in the calculation of sentence generation probabilities for candidate sentences, we mistakenly implemented the calculation as follows: $l_c={1 \\over K} \\sum_{i=1}^{K}log(p(t_{i-1}|t_{<i}))$, instead of $l_c={1 \\over K} \\sum_{i=1}^{K}log(p(t_{i}|t_{<i}))$. After rectifying this error, the results became more reasonable (BLOOM's baseline retrieval accuracy on English increased from 12.4 to 35.1). Due to time limitations, we have supplemented experiments on 12 languages of mLAMA. We provide the strong baseline results of BLOOM as follows, where languages marked with an asterisk (*) are covered by BLOOM's pre-training data.\n\n  - Retrieval Accuracy:\n\n  |             | English(source) | zh*      | ja       | eu*      | ga       | vi*      | cy       | es*      | pt*      | ta*      | ur*      | hi*      | ar*      |\n  | ----------- | --------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n  | BLOOM       | 35.1            | 18.7     | 22.7     | 13.3     | 16.4     | 28.9     | 13.7     | 25.9     | 25.8     | 13.2     | 14.4     | 13.5     | 15.1     |\n  | BLOOM(LRP2) | 35.1            | **27.9** | **24.3** | **15.4** | **18.2** | **34.0** | **17.7** | **28.5** | **28.9** | **14.2** | **16.6** | **17.6** | **19.2** |\n\n  - English-centric Cross-lingual Transferability:\n\n  |             | English(source) | zh*      | ja       | eu*      | ga       | vi*      | cy       | es*      | pt*      | ta*      | ur*      | hi*      | ar*      |\n  | ----------- | --------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n  | BLOOM       | 1               | 18.0     | 23.0     | 13.6     | 20.0     | 34.9     | 14.5     | 34.5     | 33.9     | 13.6     | 11.9     | 14.8     | 13.7     |\n  | BLOOM(LRP2) | 1               | **24.7** | **23.9** | **16.4** | **21.2** | **39.2** | **18.8** | **37.9** | **35.9** | **16.1** | **14.2** | **19.3** | **19.2** |\n\n- In addition, we have conducted experiments on another multilingual factual knowledge probing dataset, X-FACTR[1], and also included supplementary experiments on the XLM-RoBERTa-base model[2]. The results of these experiments are presented in the tables below.\n\n  1) mBERT's results on X-FACTR\n\n  - Retrieval Accuracy：\n\n  |             | English(source) | zh       | ko       | nl       | vi       | ceb      | ja       |\n  | ----------- | --------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n  | mBERT       | 22.6            | 14.4     | 12.2     | 18.3     | 22.8     | 14.3     | 10.6     |\n  | mBERT(LRP2) | 22.6            | **15.4** | **13.3** | **18.8** | **23.3** | **15.6** | **12.8** |\n\n  - English-centric Cross-lingual Transferability:\n\n  |             | English(source) | zh       | ko       | nl       | vi       | ceb  | ja       |\n  | ----------- | --------------- | -------- | -------- | -------- | -------- | ---- | -------- |\n  | mBERT       | 1               | 30.0     | 24.9     | 48.5     | 46.4     | 25.4 | 23.4     |\n  | mBERT(LRP2) | 1               | **32.6** | **28.6** | **48.7** | **47.0** | 25.3 | **30.1** |\n\n  2) XLM's results on mLAMA\n\n  - Retrieval Accuracy:\n\n  |           | English(source) | zh       | ga       | cy       | af       | eu   | la      | gl       |\n  | --------- | --------------- | -------- | -------- | -------- | -------- | ---- | ------- | -------- |\n  | XLM       | 24.7            | 15.7     | 9.4      | 10.8     | 14.0     | 9.4  | 6.8     | 18.4     |\n  | XLM(LRP2) | 24.7            | **17.3** | **11.5** | **11.3** | **14.3** | 9.4  | **8.0** | **20.0** |\n\n  - English-centric Cross-lingual Transferability:\n\n  |           | English(source) | zh       | ga       | cy       | af       | eu       | la       | gl       |\n  | --------- | --------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n  | XLM       | 1               | 28.5     | 17.8     | 20.1     | 33.4     | 15.1     | 12.8     | 36.4     |\n  | XLM(LRP2) | 1               | **31.0** | **22.3** | **21.7** | **36.1** | **15.5** | **17.4** | **37.3** |\n\n- We will revise the new results into the paper to substantiate the method's effectiveness.\n\n[1] X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models\n\n[2] Unsupervised Cross-lingual Representation Learning at Scale\n\n4. Core equations (1 and 2): can these simple vector additions be called 'projections'?\n\n- Thank you for pointing this out. As described in Wikipedia, \"In Euclidean geometry, a translation is a geometric transformation that moves every point of a figure, shape, or space by the same distance in a given direction.\" Since the vector addition used in our method falls within the scope of the \"translation\" definition, we might adjust the term \"projection\" involved in our method to \"translation\" in the next version.\n\n5. The proposed method implies choosing two Transformer layers to which the language-specific vectors are applied. Even though this choice has a crucial effect on the results (Fig 2), it is only vaguely explained in one of the appendices (A.4). It is not entirely clear how much hyper-parameter tuning happened behind the main table of results (Tab 1). This motivates my lower score for reproducibility.\n\n- As described in Appendix A.4, we conducted a grid search to determine the optimal layer configuration for each language. Certain configuration details are presented below, taking into account the 12 layers of mBERT, numbered from 1 to 12, the term 'LIRP' in the table below indicates which layer of mBERT the LIRP module is inserted into, 'LSRP' follows the same pattern, and 'layer_num' indicates the number of layers between these two modules.\n\n|           | ceb  | cs   | cy   | fa   | gl   | id   | ko   | lt   | pl   | pt   | ro   | sk   | ur   | vi   | af   | ar   | de   | he   | hi   | ja   | zh   | es   | th   | az   | bg   | bn   | da   | el   | fr   | sv   | tr   | ga   | ru   | sr   | be   | ca   | eu   | hu   | hy   | it   | ka   | la   | lv   | nl   | ta   | uk   | sq   | et   | fi   | ms   | hr   | sl   |\n| --------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| LIRP      | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 3    | 3    | 3    | 3    | 3    | 3    | 4    | 5    | 5    | 6    | 6    | 6    | 6    | 6    | 6    | 6    | 6    | 7    | 7    | 7    | 8    | 8    | 8    | 8    | 8    | 8    | 8    | 8    | 8    | 8    | 8    | 8    | 9    | 10   | 10   | 10   | 11   | 11   |\n| LSRP      | 2    | 5    | 2    | 2    | 2    | 2    | 3    | 3    | 2    | 2    | 2    | 9    | 2    | 2    | 4    | 4    | 4    | 6    | 7    | 11   | 10   | 6    | 11   | 7    | 7    | 7    | 7    | 11   | 7    | 7    | 7    | 10   | 12   | 12   | 10   | 9    | 11   | 9    | 9    | 9    | 12   | 9    | 10   | 9    | 9    | 9    | 10   | 11   | 11   | 11   | 12   | 12   |\n| layer_num | 1    | 4    | 1    | 1    | 1    | 1    | 2    | 2    | 1    | 1    | 1    | 8    | 1    | 1    | 1    | 1    | 1    | 3    | 4    | 8    | 6    | 1    | 6    | 1    | 1    | 1    | 1    | 5    | 1    | 1    | 1    | 3    | 5    | 5    | 2    | 1    | 3    | 1    | 1    | 1    | 4    | 1    | 2    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    | 1    |\n\n- This table underscores the substantial disparity in optimal layer settings among various languages. Unfortunately, we haven't been able to establish a clear correlation between the specific optimal layer settings (or the number of layers between the two modules) of different languages and their inherent language characteristics, such as language family and language resources, which hinders us from providing precise recommendations for hyperparameter selection.\n- We will include the complete optimal layer settings in Appendix A.4 of the updated version.\n\n6. Fig 2: the 'one-to-many relationship' in this plot makes it hard to understand the presented trends. These plots seem to contain a considerable amount of noise.\n\n- While the trend in Figure 2 might appear perplexing, it clearly illustrates the phenomenon where the performance of certain languages, notably \"eu\" and \"sk\", is very sensitive to the choice of model layers for the insertion of LIRP and LSRP modules. For certain numbers of layers between LIRP and LSRP for some languages, such as 9 for language \"eu\", none of the particular insertion settings (the layers where LIRP and LSRP are inserted into are 1/10, 2/11, 3/12, respectively) lead to efficient knowledge transfer.\n\n- We believe that removing the \"one-to-many relationships\" in Figure 2 will make understanding even more difficult. To enhance the readability of the main body, we will move Figure 2 to Appendix B.5 and explain it in more detail, such as explaining the significant performance fluctuations across different numbers of layers shown in the figure.\n\nWe present a summary of our responses below.\n\n- We will make content adjustments to ensure greater clarity and readability. For instance, we will relocate Figure 2 from main body to Appendix B.5 and provide a detailed analysis there. Moreover, we will transfer our insights on knowledge neurons from Appendix C.2 to section 5 and supplement the analysis of representation space there.\n- We will explain the motivation behind our method in section 3 and augment section 5 with the verification of the assumption that LPR2 can map representations of language X into the space of language Y.\n- We have corrected the mistake we made in implementing the probing of BLOOM and provided strong baseline results of BLOOM. Furthermore, we have conducted experiments on another multilingual factual knowledge probing dataset, X-FACTR, and included supplementary experiments on the XLM-RoBERTa-base model, to further substantiate the method's effectiveness.\n- We will adjust the term \"projection\" involved in our method to \"translation\".\n- We will include the complete optimal layer settings in Appendix A.4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a lightweight, parameter-free method to improve the accuracy of factual knowledge returned by a multilingual LM in language X, by leveraging the knowledge already existing within the model for the English language.\n- the addressed problem is important and timely\n- the technique is simple and not computationally costly", "reasons_to_reject": "Main reasons:\n- Some of the core claims of the paper cannot be assessed without reading the appendices (see especially sect. 4.2 and 5). This should really be a long paper.\n- The core assumption underlying the proposed method (see above) should be verified or at least motivated on the basis of previous work on cross-lingual representation learning. For instance cf. the strong claim \"By performing this projection, the representations of non-English language l are mapped into the English space\" in line 164-165.\n- The reported improvements are very modest for mBERT. They are better for BLOOM but the baseline scores for this model are very low.\n\nOther issues:\n- core equations (1 and 2): can these simple vector additions be called 'projections'? \n- The proposed method implies choosing two Transformer layers to which the language-specific vectors are applied. Even though this choice has a crucial effect on the results (Fig 2), it is only vaguely explained in one of the appendices (A.4). It is not entirely clear how much hyper-parameter tuning happened behind the main table of results (Tab 1). This motivates my lower score for reproducibility.\n- Fig 2: the 'one-to-many relationship' in this plot makes it hard to understand the presented trends. These plots seem to contain a considerable amount of noise.\n- the paper states in different places that the proposed method \"significantly\" improves factual knowledge accuracy, however said improvements are either very small (for mBERT) or mild on top of a weak baseline (for BLOOM) and I see no mention of statistical significance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a lightweight method to transfer factual knowledge across languages in multilingually pre-trained LMs. The core assumption of the work is that sentence representations in language X can be projected to the space of language Y by simply adding a fixed language-specific vector v to those representations. The language-specific vectors v_X and v_Y are obtained by averaging over the max-pooled sentence representations of N sentences in language X and Y, respectively.\nGiven a query in (low-resource) language X, the said vectors are used to push the query representation closer to the representation space of English (based on the assumption that English is the language with most factual knowledge within the model).\nMore specifically, the addition operation is performed once after layer i to move X representations closer to language space Y, and then after layer j (with j>i) to move them back close to space X.\nThe method thus does not require a learning phase and is parameter-free.\n\n\nWhile this work is promising and addresses a timely and important problem, it has two serious issues:\n1) It has been submitted as a short paper, but is really a long paper with a lot of appendices. Much of the appendices' content is essential to understand the paper and support its claims.\nThis particularly applies to A.2 (evaluation metrics) and C (analysis of knowledge neurons), without which most claims in the conclusions cannot be assessed.\nRearranging the paper to solve this issue can only result in a long submission, in my opinion.\n\n2) The whole method builds on a rather strong assumption that a simple addition of a fixed term to sentence representations in language X can moved them  closer to equivalent representations in language Y. Unfortunately the assumption is not motivated based on previous work, nor is it properly verified empirically (by this, I mean that the paper does not check whether the said addition actually results in representations that are significantly closer to the representations of an equivalent query in the target language).\nInstead, the authors present results in terms of accuracy in the knowledge probing task (mLAMA benchmark) and a so-called \"English-centric cross-lingual transferability\" score, which if I understand correctly *from the appendices* corresponds to the ratio of correct answer overlap between any given language X and English.\nThe effect of the method on said scores is not convincing: specifically, it is very small for mBERT, and more visible on BLOOM but the latter model has a very low baseline score, which is not compared to the scores reported for similar models in the original mLAMA paper. \nGiven all this, I find it very hard to judge if the proposed method has a significant effect at all on the problem it set out to solve.\n\nAs said above, I do believe this work has potential but it requires more work to be publishable. My recommendations for a resubmission can be summarized as follows:\n- make it a long paper with all due explanations in the main body\n- probe the direct effect of the vector addition on the query representations (do they  actually get closer, and if so how much, to an equivalent query representation in the target language)\n- ensure a proper baseline English score for the BLOOM model (perhaps by improving the probing method currently used on BLOOM, cf. appendix A.3).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pIY0Arc09z", "reviewer": "Reviewer_LiBy", "summary": "", "questions": "N.A", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time to review and your valuable feedback. We respond to your comments below.\n\n1. The presentation could be better in this paper. For example, the Figure 2 is not legible.\n\n- While the trend depicted in Figure 2 might seem puzzling at first glance, it illustrates the phenomenon where the performance of certain languages, notably \"eu\" and \"sk\", is very sensitive to the choice of model layers for the insertion of LIRP and LSRP modules. For certain numbers of layers between LIRP and LSRP for some languages, such as 9 for language \"eu\" in Figure 2(a), none of the particular insertion settings (the layers where LIRP and LSRP into are inserted are 1/10, 2/11, 3/12, respectively) lead to efficient knowledge transfer. Such sensitivity may stem from the relatively fragile nature of the representation space learned by multilingual pretrained model for these languages. Consequently, the representations of these languages could easily lose semantic information and become meaningless after language representation projections.\n- As this observation is of lesser prominence than the core finding of this paper, we will relocate Figure 2 to Appendix B.5. Additionally, to facilitate reader comprehension of Figure 2, we will provide a more comprehensive analysis of it in Appendix B.5, such as explaining the significant performance fluctuations across different numbers of layers shown in the figure.\n- Furthermore, we will make the following adjustments to enhance the paper's presentation:\n  - Remove details about the proposed method in section 1, and focus the introduction on the background and contributions of this work.\n  - Integrate section 2, \"Multilingual Factual Knowledge Probing,\" into section 1 as a part of the background context.\n  - Present more precise descriptions of method performance and core findings in Section 4.2.\n  - Transfer our insights on knowledge neurons from Appendix C.2 to section 5.\n  - Supplement the analysis of representation space in section 5, including calculations of cosine distance variations between representations of different languages with and without LPR2, as well as 2-D visualizations of sentence representations.\n\n2. The impacts of the LIRP and LSRP are not given. For example, The attention score/map of some example instances could be given to show the difference between the original attention and projected attention for the understanding of why projecting can help transfer.\n\n- First, the analysis of knowledge neurons in section 5 shows that LRP2 can help non-English languages to stimulate knowledge neurons similar to English, thereby acquiring knowledge transferred from English. This analysis elucidates the underlying mechanism of LRP2 and clarifies the rationale behind the observed knowledge transfer.\n\n- Second, to further investigate the impacts of LIRP and LSRP modules, we have expanded our analysis by utilizing mLAMA to calculate the layer-wise cosine similarity of sentence representations of parallel Chinese and English queries. The results are detailed below, where \"mBERT\" denotes the direct calculation on mBERT, and \"mBERT(LRP2)\" represents the statistical outcome with the insertion of the LIRP module into the 4-th layer and the LSRP module the 10-th layer of mBERT, which yields the best transferability result of Chinese.\n\n  |             | 1    | 2    | 3    | 4        | 5    | 6    | 7    | 8    | 9    | 10       | 11   | 12   |\n  | ----------- | ---- | ---- | ---- | -------- | ---- | ---- | ---- | ---- | ---- | -------- | ---- | ---- |\n  | mBERT       | 0.70 | 0.75 | 0.86 | 0.88     | 0.91 | 0.92 | 0.88 | 0.87 | 0.90 | 0.91     | 0.90 | 0.55 |\n  | mBERT(LRP2) | 0.70 | 0.75 | 0.86 | **0.90** | 0.93 | 0.93 | 0.89 | 0.88 | 0.92 | **0.89** | 0.89 | 0.51 |\n\n- This table clearly shows the distinct functions of LIRP and LSRP. Specifically, the LIRP module reduces the distance between representations of different languages, thereby facilitating cross-lingual knowledge transfer, while the LSRP module increases the distance between these representations, inducing language-specific outputs. Furthermore, 2-D visualizations of sentence representations can also help illustrate the role of the two modules.\n\n- We will augment section 5 with the explanation of the impacts of the LIRP and LSRP.\n\nWe present a summary of our responses below.\n\n- We will make content adjustments to enhance the paper's presentation. For example, we will relocate Figure 2 to Appendix B.5 and provide a detailed analysis of it there.\n- We will augment section 5 with the explanation of the impacts of the LIRP and LSRP, through the analysis of representation space.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This proposes a parameter-free framework LRP2 that enhances the factual knowledge in low resources language in mLAMA dataset.\n2. This paper gives promising results for transferring high-resource data to low-resource data by directly projecting the representations in the multilingual pre-trained language models.\n3. The parameter-free framework is easy to understand and intuitive.", "reasons_to_reject": "1. The presentation could be better in this paper. For example, the Figure 2 is not legible.\n2. The impacts of the LIRP and LSRP are not given. For example, The attention score/map of some example instances could be given to show the difference between the original attention and projected attention for the understanding of why projecting can help transfer.", "questions_to_authors": "N.A", "paper_topic_and_main_contributions": "- What is this paper about? This paper aims to transfer factual knowledge from high-resource language to low-resource language\n-  what contributions does it make? This paper proposes two parameter-free modules LRP2 to convert non-English representations into English style and then to revert them back. With these proposed modules, this paper demonstrates better results on mLAMA dataset. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N.A", "Typos_Grammar_Style_and_Presentation_Improvements": "N.A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes and explores a lightweight approach to transfer factual knowledge across languages in multilingually pre-trained language models by using Language Representation Projection modules (LRP2). Overall, the paper presents a consistent takeaway that usage of LRP2 on multiple multilingual models shows improvement in the knowledge probing tasks using the mLAMA dataset. However, as pointed out by multiple reviewers, many important details are packed into the appendix and also into the discussion posts. The AC strongly recommends including it in the appendix and on the fifth page (in case of acceptance) given that the paper is pretty dense.
Decision: Accept-Main