Qualifying Knowledge and Knowledge Sharing in Multilingual Models

Authors: Nicolas Guerin, Emmanuel Chemla
Primary area: interpretability and explainable AI
Keywords: Knowledge Retrieval, Pretrained Language Model, LLM Interpretability, Multilingual Models, Cross-linguistic Dataset

Abstract:
Pre-trained language models (PLMs) have demonstrated a remarkable ability to encode factual knowledge. However, the mechanisms underlying how this knowledge is stored and retrieved remain poorly understood, with important implications for AI interpretability and safety. In this paper, we disentangle the multifaceted nature of knowledge: successfully completing a knowledge retrieval task (e.g., “The capital of France is __”) involves mastering underlying concepts (e.g. France, Paris), relationships between these concepts (e.g. capital of), the structure of prompts, including the language of the query. We propose to disentangle these distinct aspects of knowledge and apply this typology to offer a critical view of neuron-level knowledge attribution techniques. For concreteness, we focus on Dai et al.'s (2022) Knowledge Neurons (KNs) across multiple PLMs, testing 10 natural languages and unnatural languages (e.g. Autoprompt).
Our key contributions are twofold: (i) we show that KNs come in different flavors, some indeed encoding entity level concepts, some having a much less transparent, more polysemantic role , and (ii) we uncover an unprecedented overlap in KNs across up to all of the 10 languages we tested, pointing to the existence of a partially unified, language-agnostic retrieval system. To do so, we introduce and release the mParaRel dataset, an extension of ParaRel, featuring prompts and paraphrases for cloze-style knowledge retrieval tasks in parallel over 10 languages.

Decision: reject
Reviewer scores: 8, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730120040569

thread_type:
official_review

<thread_object>
creator: reviewer_blns

<content>
summary:
In this paper, the authors provide a new perspective to refine the concept of "knowledge" by introducing the Concept Neurons and Relation Neurons. The authors also construct a multilingual version of the ParaRel dataset called mParaRel to analyze the language-agnostic knowledge neurons.

strengths:
The analysis of knowledge storage and retrieval mechanisms for large language models is very important. This paper introduces concept neurons and relation neurons to study the knowledge mechanism of large language models, which helps to further explain large models. Besides, the authors create a dataset called mParaRel covering 10 languages, which helps to further analyze the internal mechanisms of multilingual large language models.

weaknesses:
1. The authors should conduct more comprehensive experiments to demonstrate the accuracy of the localized concept neurons and relationship neurons. The stability or reliability of the experimental results.

2. Compared to their previous work, the authors extend the discovery of language-agnostic knowledge neurons from 2 languages to 10 languages, while this contribution is somewhat limited.

questions:
1. Why not use the change rate of output probability as an evaluation metric, when manipulating concept neurons or relationship neurons? The change rate of output probability has been adopted by previous work (Dai et al. 2022).

2. Does the findings of this article apply to larger scale models? It would be better if a larger model could be analyzed, but it may be limited by computational resources.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650950343

thread_type:
official_review

<thread_object>
creator: reviewer_zsm3

<content>
summary:
This paper investigated the knowledge representation mechanism of language models based on knowledge neuron. Specifically, by investigating knowledge neurons across different instantiations, prompts and languages, they find that 1) knowledge neurons exhibit different flavors, some encoding entity-level concepts, others with a more polysemantic role. 2) part of the knowledge neurons are shared by certain languages. 3) some knowledge neurons of certain LMs are shared between natural and unnatural prompts (specifically autoprompt).

strengths:
- The findings and conclusions of this paper contribute to researchers' understanding of the knowledge storage mechanisms in LMs, thereby facilitating further in-depth research in the future.
- The proposed multilingual dataset can benefit future relevant studies.
- This paper offers several interesting perspectives for analyzing the knowledge mechanisms of LMs.

weaknesses:
- My primary concern regarding this paper is the **reliability and generalizability of its conclusions**.
  - All analyses are based on the knowledge neuron (Dai et al. 2022), a gradient-based algorithm initially proposed for encoder-only models. However, the authors did not verify whether their conclusions hold with other knowledge attribution methods (e.g., ROME, SAE). This significantly limits the applicability of their corresponding conclusions.
  - In their analysis of Relation Neuron and Concept Neuron, the authors subjectively define the selection of relevant knowledge neurons through threshold values. Consequently, the identified knowledge neurons are significantly affected by the chosen thresholds, making it uncertain whether these neurons genuinely encode the information as claimed by the authors. To address this, **the authors should incorporate more intuitive and precise analytical methods to ascertain the reliability of the identified neurons**. For instance, they could investigate whether perturbing these neurons affects the model's expression of a specific concept without impacting other concepts.
  - For a probing study, the authors should incorporate more recent and larger models, considering that LLaMA-2 has already been released for over a year. Additionally, it would be beneficial to conduct a more in-depth analysis of the consistency and differences in conclusions across different models.
  - Regarding the authors' analysis of different prompt types, they considered only a single prompt search algorithm (AutoPrompt). However, since both AutoPrompt and KN are gradient-based methods, the reliability of the corresponding conclusions is questionable.

questions:
See Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722519757

thread_type:
official_review

<thread_object>
creator: reviewer_kxuv

<content>
summary:
This paper explores how multilingual language models encode and share knowledge across languages. Specifically, the authors categorize Knowledge Neurons (KNs) into two subtypes: relation neurons and conceptual neurons. They investigate to what extent these neurons exist, their contributions to knowledge retrieval, and their impact on downstream tasks. The paper introduces mParaRel, a multilingual variant of ParaRel, to facilitate experiments across ten languages. The authors find that while most KNs do not adhere to a clearly defined role, some KNs do serve specific functions. Their findings suggest that many KNs are shared across languages, indicating a partially language-agnostic retrieval mechanism in multilingual models.

strengths:
1. Differentiating conceptual neurons and relation neurons is a meaningful theoretical advancement in studying knowledge representation.

2. The mParaRel dataset is a valuable contribution to multilingual probing research.

3. The experiments provide sufficient support for the paper's central claim regarding the individuality and shareability of knowledge neurons across languages.

weaknesses:
1. The contribution of the paper is not novel or different from the lines of work in multilingual knowledge probing, where they all indicate that there exist language-agnostic and language-specific neurons such as [1][2][3]. None of the works were cited or discussed here. 

2. The authors rely on threshold-based methods to classify neurons as either conceptual or relational, but variations in thresholds produce different neuron classifications. This makes the claim unrobust: line 208 "When the thresholds become more restrictive, the number of neurons with well-defined roles decreases". There needs to be more effort in verifying that the relation neurons are indeed representing perform identical relational functions across languages.

[1] Kojima, Takeshi, et al. "On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons." NAACL (2024). 

[2] Tang, Tianyi, et al. "Language-specific neurons: The key to multilingual capabilities in large language models." ACL (2024).

[3] Wang, Weixuan, et al. "Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs." arXiv preprint arXiv:2406.09265 (2024).

questions:
What are your insights regarding the differences in knowledge-sharing levels among various language pairs? Figure 4 indicates that Spanish shares more knowledge with Danish and Dutch than with French or Italian, which belong to the same Latin family. Do you observe any cross-model similarities related to this finding?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730877812478

thread_type:
official_review

<thread_object>
creator: reviewer_6nqi

<content>
summary:
This paper presents a valuable investigation into knowledge representation within pre-trained language models (PLMs), specifically exploring how these models encode and retrieve factual knowledge. The authors introduce a fine-grained typology of knowledge that separates various aspects, such as entity concepts, relationships, and prompt structures. By examining the Knowledge Neuron (KN) framework across multiple PLMs, they reveal that not all neurons are monosemantic; rather, a significant portion exhibit polysemantic behavior, with only certain neurons specialized for specific concepts or relationships. 

The study’s empirical findings, derived from a range of models (e.g., BERT, Llama 2, and mBERT) and 10 languages, uncover substantial overlap in knowledge representation, suggesting a cross-linguistic, language-agnostic retrieval system. The authors also introduce the mParaRel dataset, an extension of ParaRel in 10 languages, enhancing the scope of knowledge retrieval evaluation. These contributions imply that language models may benefit from shared multilingual training, offering insights into efficient expansion to new languages. 

Overall, the paper provides a thoughtful critique of neuron-level attribution techniques, indicating that future PLM training could optimize for unique linguistic characteristics rather than relearning factual knowledge for each language.

strengths:
Originality: This paper is highly novel, introducing a new typology for analyzing knowledge in PLMs that challenges the Knowledge Neuron hypothesis of monosemanticity. By revealing polysemantic behaviors in neurons, the study provides a nuanced view of knowledge representation, paving the way for a broader understanding of how PLMs encode concepts and relationships.

Quality: The study is well-executed, with a solid experimental design across a variety of PLMs and languages. The release of the multilingual mParaRel dataset further enhances the quality and replicability of this work.

Clarity: The paper is clearly organized, especially in detailing the methods and experimental results, making the findings accessible to readers.

Significance:
This work has considerable impact, as it suggests that knowledge in multilingual models may be stored in a language-agnostic manner, meaning retraining for each language might be unnecessary. This insight holds potential for more efficient multilingual model training, especially for low-resource languages, and offers a promising direction for leveraging neuron-level analysis in PLMs to improve model interpretability and efficiency.

weaknesses:
P9 L468: The formula $(\text{number of languages})^{-\alpha}$ is introduced, but the value and explanation of $\alpha$ are not provided. It would help to describe this parameter to make the formula more rigorous.

P9 L468-469: Assuming $\alpha$ is between 0 to 1, both $(\text{number of languages})^{-\alpha}$ and $p^{\text{number of languages}}$ decay as the number of languages increases. How can we be confident that the curve shown in Fig. 4c follows the former form and not the latter? A brief explanation of how the curve was fitted to these two functions would be helpful, as it directly impacts one of the major conclusions that KNs are multilingual.

questions:
P4 L163-164: "we apply our analysis to the earliest of these methods by way of illustration." Could you please specify which is "the earliest" of these methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732027209841

thread_type:
public_comment

<thread_object>
creator: ~constanza_fierro1

<content>
title:
Nitpick: name of the dataset

comment:
This is a cool work! Just a small suggestion, maybe it would be a good idea to change the name of the dataset, as we also developed a translation of ParaRel a few years ago and named it mParaRel (https://aclanthology.org/2022.findings-acl.240/) so it could get confusing...
</content>
</thread_object>

</thread>
<thread>
creation date:
1732792891530

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Submission of Rebuttal Revision

comment:
Thank you for the valuable feedback provided during the review process. We have uploaded a revised version of our paper addressing the key concerns raised. Below is a summary of the major updates:

- Clarified Lines 163-164 for improved readability and precision.
- Fitted the curve in Section 6.2 and added the fitted curve along with the baseline to the corresponding graph.
- Included the proposed references and discussed them in the Related Work section.
- Expanded the discussion on the use of thresholds in Section 4, incorporating feedback from reviewers.
- Provided clearer explanations for the motivations behind our metric choices in Section 5.3.
- Added a summary table comparing model behaviors against expectations for the Boosting experiment (Appendix).
- Discussed the potential bias of using gradient-based AutoPrompt and Knowledge Attribution methods in Section 6.3.
- Included a discussion emphasizing the importance of studying more than two languages in the Related Work.
- Renamed our multilingual dataset as required.
- Added the discussed control experiment for the Boosting experiment (Appendix).

We hope these revisions address the concerns raised and improve the clarity and quality of the paper. As always, we welcome any further questions or comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

This paper aims to build on the notion o knowledge neurons in order to find specific neurons where interlingual knowledge is stored. Two kinds of knowledge neurons are introduced: conceptual neurons adn relation neurons. Thee existence of these neurons is experimentally searched and some specific neurons are spotted, albeith the majority of neurons do not have a specific role. 

**Strengths**

- Introducing a classification of knowledge neurons
- Proving that this classification fits existing neurons

**Weaknesses**

- This paper is an observation paper that, besides the possibility to retreive these neurons, it does not give any possible way to modify these neurons in order to change the global behavior of the network. 
- A comparison with alternative methods like ROME and MEMIT is missing

**Final remarks**

Knowledge neurons are interesting abstractions of a very complex phenomena. The idea that only a bunch of neurons is responsible for a specific behavior is very interesting. However, attribution is not sufficient to demostrate that these knowledge neurons exist.