IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neurons

Authors: dan  shi
renren  jin
tianhao  shen
weilong  dong
xinwei  wu
deyi  xiong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:50:21

Abstract: It is widely acknowledged that large language models (LLMs) encode a vast reservoir of knowledge after being trained on mass data. Recent studies disclose knowledge conflicts in LLM generation, wherein outdated or incorrect parametric knowledge (i.e., encoded knowledge) contradicts new knowledge provided in the context. To mitigate such knowledge conflicts, we propose a novel framework, IRCAN (Identifying and Reweighting Context-Aware Neurons) to capitalize on neurons that are crucial in processing contextual cues. Specifically, IRCAN first identifies neurons that significantly contribute to context processing, utilizing a context-aware attribution score derived from integrated gradients. Subsequently, the identified context-aware neurons are strengthened via reweighting. In doing so, we steer LLMs to generate context-sensitive outputs with respect to the new knowledge provided in the context. Extensive experiments conducted across a variety of models and tasks demonstrate that IRCAN not only achieves remarkable improvements in handling knowledge conflicts but also offers a scalable, plug-and-play solution that can be integrated seamlessly with existing models. Our codes are released at https://github.com/danshi777/IRCAN.

Main paper: 
 introduction: 
 Large language models (LLMs), trained on extensive data, are known for encapsulating a broad spectrum of knowledge [13, 40, 26, 35] . However, due to the rapid evolution of information/knowledge as well as noise in training data, LLMs may possess incorrect or outdated knowledge. To mitigate this issue, in real-world applications, methods like retrieval-augmented generation (RAG) are usually used to integrate latest event knowledge or knowledge from external databases into the context of prompts fed into LLMs. This enables online updates to knowledge and the incorporation of domain-specific information, enhancing the accuracy and reliability of the outputs of LLMs.Such generation formalism equips LLMs with two sources of knowledge: (1) parametric knowledge, which is acquired during pre-training and encoded within model parameters; and (2) contextual knowledge, which is supplied as the prefix context within the input [3] . However, previous studies have shown that when LLMs encounter contradictions between these two types of knowledge (known as knowledge conflicts [43, 44] ), they may overly adhere to their inherent parametric knowledge and fail to pay sufficient attention to new knowledge introduced in the context [28, 4, 43] , leading to hallucinations [48, 19, 18] . For example, although we present the latest information "As of 2023, India has surpassed China as the most populous country." in the context to LLaMA-2-7B, when it is faced with the question "Which country is the most populous in the world?\nAnswer:", it still provides the outdated answer "China".We hypothesize that within LLMs, there exist neurons that specifically focus on processing context, akin to knowledge neurons [8] . With this assumption, to alleviate the aforementioned issues, we propose a framework IRCAN for Identifying and Reweighting Context-Aware Neurons to encourage the model to pay more attention to contextual knowledge during generation. Specifically, we first measure the contribution of each neuron to the context processing by calculating their attribution scores. Subsequently, we increase the weights of the detected context-aware neurons, which allows the model to effectively up-weight the contextual knowledge during generation.We conduct extensive experiments on a diverse array of models from multiple families, including LLaMA [39] , Gemma [31] and Amber [27] , spanning various parameter scales (2B, 7B, 8B, 13B) and encompassing both pre-trained and instruction-tuned models. To conduct a comprehensive evaluation, we carry out experiments on two types of tasks: completion and multiple-choice. Experiment results demonstrate that our method can effectively identify neurons responsible for processing the context within LLMs. Moreover, by enhancing these neurons, LLMs can be guided to remain more faithful to the information provided in the context when facing knowledge conflicts, rather than sticking to its intrinsic knowledge. Additionally, our method can serve as a plug-and-play module, easily integrated with existing approaches. In completion tasks, IRCAN has achieved state-of-the-art performance, with substantial improvements of 129% and 136% in terms of accuracy for LLaMA-2-7B and LLaMA-3-8B respectively. Remarkably, when our method is integrated with CAD [36] , a previous strong method for dealing with knowledge conflicts, the performance of LLMs can be further improved. In multiple-choice tasks, IRCAN achieves comparable results to the baseline, and when combined with CAD, our method sets new state-of-the-art results.The main contributions of our work are summarized as follows:• We pioneer the exploration of attribution methods to knowledge conflicts for LLMs, offering a novel approach to resolving knowledge conflicts. • We propose an attribution method to identify neurons within LLMs that are responsible for processing context based on integrated gradient. Furthermore, by enhancing these context-aware neurons, the LLMs' fidelity to contextual knowledge is effectively improved. • We conduct extensive experiments and experiment results demonstrate that the proposed approach can significantly boost the performance of LLMs on tasks involving knowledge conflicts. 
related work: 
 To correct outdated or incorrect knowledge in language models, previous studies have explored three main strategies: fine-tuning, model editing and contrastive decoding.Fine-tuning Fine-tuning aims to update the internal knowledge of an existing LLM through further training on additional data, including datasets with the latest information or domain-specific datasets [50, 20, 11, 45, 17] . However, this process requires substantial computational resources and a large amount of training data, as well as significant training time, which can be unaffordable in many cases. More seriously, it may lead to catastrophic forgetting issues.Model Editing Model Editing seeks to edit incorrect or undesirable knowledge encoded in pretrained models. Some studies initially identify knowledge-related parameters of the existing pretrained models and then directly edit particular knowledge into these parameters [12, 29, 30, 41, 42] .Other efforts have been explored to store new or correct knowledge in an extra memory, replacing the original predictions with this knowledge during generation [33, 9] . Additionally, meta-learning based methods learn to edit models through meta-learning [2, 32] . However, these approaches are only applicable to modifying specific knowledge. In contrast, our method is independent of specific knowledge: regardless of the type of knowledge contained in the context, it enhances the LLM's utilization of this knowledge.Contrastive Decoding Contrastive decoding strategies are adopted during generation, which amplify the differences in output probabilities between various model scales [23] or different layers of an LLM [5] , thereby reducing hallucinations. Among these, context-aware decoding (CAD) [36] amplifies the difference between output probabilities with and without context, encouraging the LLM to attend to its context during generation. Since its task and goal are the same as ours, we utilize it as a baseline for comparison in our experiments. Significantly different from CAD, our approach IRCAN operates at a finer granularity of neurons, thereby providing a degree of interpretability for analyzing and resolving knowledge conflict issues. 
methodology: 
 We focus on tasks involving context-specific knowledge conflicts. The input of these tasks is formulated as (c, q), where c is the context, and q represents the question in completion tasks or the question combined with a suffix consisting of choices in multiple-choice tasks. We propose a novel method that is dedicated to improving the faithfulness of LLMs to the context to address these tasks. The proposed IRCAN methodology is structured into three phases: Initially, we compute the attribution scores of each neuron to assess its influence on context processing. Subsequently, neurons that are responsive to context, termed "context-aware neurons", are identified. In the final step, we enhance the influence of these detected neurons through a reweighting process, thereby augmenting their impact on the model's generation. The framework of IRCAN is illustrated in Figure 1 .Previous work has found the existence of knowledge neurons that store and express factual knowledge in FFNs [8] . We speculate that certain neurons responsible for processing contextual knowledge also exist in FFNs. Inspired by Hao et al. [14] , who introduce an attribution method to interpret the information interactions inside Transformer, we propose a context-aware attribution method based on integrated gradients [37] to identify these neurons. Our method calculates the contribution scores of FFN neurons in perceiving the context towards predicting answers. This evaluation helps determine which neurons play a critical role in context processing.The attribution score of each neuron to be evaluated is denoted as Attr(n l i ), where n l i represents the intermediate neuron at the i-th position in the l-th FFN layer of the language model. Initially, we take only the question as input, record the activation value of each neuron and denote it as v q l i . Subsequently, we input both the context and the question into the language model and record the new activation value, denoted as v (c,q) l i . To calculate the attribution score Attr(n l i ), we gradually change the activation value of a neuron n l i from v q l i to v (c,q) l i when the input consists of both context and question. At the same time, the output probability of the model changes accordingly. We calculate the probability of the correct answer predicted by the language model, denoted as:EQUATIONwhere y * denotes the correct answer; v l i is a given value assigned to the neuron activation A(n l i ). We integrate the gradient of the probability during this process as the neuron's context-aware attribution score, as follows:EQUATIONwhere∂P [vq l i +α(v (c,q) l i -vq l i )]∂v (c,q) l i calculates the gradient of the model probability with regard to v (c,q) l i , α controls the integration from v q l i to v (c,q) l i . Theoretically, the integrated gradients technique adheres to two fundamental axioms of attribution methods: Sensitivity and Implementation Invariance [37] . The Sensitivity axiom stipulates that if modifying a neuron alters the prediction, that neuron should be assigned a non-zero attribution score. The Implementation Invariance axiom dictates that the attributions should remain identical for two networks with equivalent functionality. Adherence to these axioms ensures that the attribution scores accurately reflect the importance of neurons and are invariant to implementation details. The attribution scores facilitate the identification of neurons essential for context processing.Intuitively, by integrating over the gradient as α changes from 0 to 1, Attr(n l i ) accumulates the output probability changes caused by the activation value changes from the absence to the presence of context. If the neuron has a strong perception and processing capability regarding the context, the gradient will be significant, resulting in a large integration value. Therefore, the attribution score can measure the neuron's sensitivity to the context and its contribution to processing the context. Directly calculating continuous integrals is intractable. We instead use the Riemann approximation of the integration to efficiently compute the attribution score. Specifically, we sum the gradients at points occurring at sufficiently small intervals from v q l i to v (c,q) l i :EQUATIONwhere m is the number of approximation steps. Following previous work [8] , we set m to 20, which performs well in our experiments.Based on the calculated neuron attribution scores Attr(n l i ), we first retain neurons with scores above the threshold t, creating a coarse set of context-aware neurons. Then, for each example, we select the top z neurons with the highest attribution scores as the candidate set. In our experiments, t and z are set to 10% and 20 respectively. Ultimately, we count the number of co-occurrences of neurons in all candidate sets, and we select the top h neurons with the highest number of co-occurrences as context-aware neurons. These identified context-aware neurons are shared across all data instances.In order to make LLMs generate outputs that are more faithful to the context, we enhance the identified context-aware neurons. We adopt a simple yet effective enhancement measure:EQUATIONi.e., amplifying the weights of these neurons to β (i.e., enhancement strength) times their original weights. This adjustment amplifies the role these neurons play as information flows through them, thus enhancing their influence on the model's output. 
experiments: 
 We conducted experiments on two different types of knowledge conflict tasks (i.e., completion and multiple-choice) to verify the effectiveness of IRCAN. These tasks demand the model to reason about information in the context and generate context-faithful responses.Completion Task We conducted completion task experiments on the MemoTrap [25] dataset. It evaluates the models' ability to adhere to the given context in order to generate an unfamiliar phrase, rather than defaulting to a well-known phrase that has been encoded in its parameters during training. Specifically, it consists of instructions that challenge the language model to conclude a well-known proverb with a terminal word that diverges from its traditional ending (e.g., "Write a quote that ends in the word 'returned': Long absent, soon ", where the commonly used ending is "forgotten"). It is designed to explore the potential for language models to fall into memorization traps.Multiple-choice Task For the multiple-choice task, we utilized the COSE_KRE and ECARE_KRE datasets [46] , which were derived from ECQA [1] and e-CARE [10] , respectively. The derivation process involved selecting one of the incorrect answer choices and prompting ChatGPT to generate explanations supporting this incorrect answer. Specifically, the selected incorrect answer was treated as the correct answer, and the explanations generated by ChatGPT were used as the context for the multiple-choice question.Illustrative examples from all datasets are shown in Table 4 in Appendix A. We expect LLMs to pay more attention to the knowledge in the context.For the completion task, we conducted experiments on four LLMs with diverse parameter scales: Gemma-2B [31] , Amber [27] , LLaMA-2-7B and LLaMA-2-13B [39] . We employed accuracy as the evaluation metric, which quantifies the proportion of correctly generated words. The prompt was formed by combining the context and question, allowing the LLMs to generate a continuation. Regular expressions were used to extract the generated ending word.For the multiple-choice task, we evaluated a diverse list of LLMs of the chat version: LLaMA-2-7B-Chat, LLaMA-2-13B-Chat, LLaMA-3-8B-Instruct2 and Gemma-2B-it [31] . We evaluated the performance of these LLMs by measuring the accuracy of selecting the correct answer. We perform prompt engineering to design the appropriate prompt for each model. Please refer to Appendix B for more details. We selected the answer based solely on the highest probability among options, which is the official implementation of MMLU [15] and widely used in other benchmarks [16, 21] .Although we used accuracy as the primary metric to evaluate our method more comprehensively, we also designed a supplementary metric: stubbornness rate (SR), which measures whether the LLM persistently adheres to its internal memorized knowledge. This metric is defined as the accuracy with which the model's generation matches the original golden label (for the MemoTrap dataset, i.e., the common ending word of a well-known proverb; for the COSE_KRE and ECARE_KRE dataset, i.e., the original golden option). A lower stubbornness rate indicates that the LLM exhibits a decreased reliance on the knowledge encapsulated within its internal parameters during the generation process, suggesting a greater propensity to incorporate contextual information to a certain extent.To demonstrate the effectiveness of IRCAN, we compared it with the following baselines: Original: which refers to the LLMs without any modification. Prompt engineering based methods: we curated three types of prompts to explicitly instruct LLMs to pay more attention to the knowledge in context on the multiple-choice task. According to the content added in the prompt, we denote these three prompt engineering based methods as Based_on, Based_on_Formatted, and Utilizing_Formatted, respectively. Please refer to Appendix C for the details of these prompts. Inference-Time Intervention (ITI) [22] : Datasets Models this method identifies a direction in the activation space associated with factually correct statements and shifts activations along this direction during inference, thereby enhancing the truthfulness of LLMs. Analogous to its experimental setup, for each sample in each dataset, we concatenated the question/answer together and extracted head activations at the last token to collect a probing dataset.Models Gemma-2B LLaMA-2-7B Amber (7B) LLaMA-3-8B LLaMA-2-13B ACC ↑ SR ↓ ACC ↑ SR ↓ ACC ↑ SR ↓ ACC ↑ SR ↓ ACC ↑ SR ↓ Original 23.Gemma-2B-it LLaMA-2-7B-Chat LLaMA-3-8B-Instruct LLaMA-2-13B-Chat ACC ↑ SR ↓ ACC ↑ SR ↓ ACC ↑ SR ↓ ACC ↑ SR ↓ COSE_KRE OriginalWe then used the ITI method to identify the direction and intervened activations along this direction. We implemented two intervention directions in our experiments, i.e., Probe Weight Direction and Mass Mean Shift. Context-aware decoding (CAD) [36] : it adjusts the output probabilities of LLMs to emphasize differences when the context is utilized versus when it is absent. Specifically, this approach reduces the weighting of prior knowledge in favor of more pertinent contextual information. Since this method and our approach work in completely different ways: CAD manipulates the model's final output probabilities, while IRCAN focuses on identifying and enhancing neurons responsible for processing the context, these two methods can be seamlessly combined without any obstacles.In our experiments, we also report the performance of the combined system, which is referred to as IRCAN-CAD in our experiments. Opin [49] : this method argues that transforming factual questions into questions seeking opinions allows people to pay more attention to the context. Therefore, it uses opinion-based prompts to enable the model to generate context-faithful responses. Since the prompts designed by Opin do not apply to the completion task, we used this method as a baseline only on the multiple-choice datasets. To ensure a fair comparison, we adopted the same method for selecting the generated answer as described in §4.2 for Opin.We treat the number of identified context-aware neurons h ( §3.2), ranging from 1 to 16, and the enhancement strength β for these neurons ( §3.3), ranging from 2 to 20, as hyperparameters. All datasets were randomly divided into validation and test sets in a 1:1 ratio. We employed grid search to identify the optimal hyperparameter configuration that maximized performance on the validation set. Subsequently, we report the test set results obtained using the identified optimal hyperparameter combination in Table 1 and Table 2 .Completion Task The main results on the MemoTrap dataset are shown in Table 1 . IRCAN significantly outperforms all the baselines. Intervening along the Mass Mean Shift direction brings different degrees of performance degradation to the vast majority of LLMs. Even more, the interventions cause Gemma-2B and Amber to completely fail to respond normally. Improvements along the Probe Weight Direction are also limited. This suggests that finding a direction relevant to the context and shifting activations along this direction during inference might not be a good way to enhance the attention of an LLM to contextual knowledge. The CAD method shows a slightly better improvement, but there is still a significant gap compared to our IRCAN. Notably, IRCAN achieves remarkable improvements of 129% for LLaMA-2-7B and 136% for LLaMA-3-8B in terms of accuracy when compared to the Original baseline. Such substantial performance improvements, achieved through strengthening merely a few or a dozen neurons, indicate that the neurons identified by our method play a pivotal role in processing context.Additionally, the significant decrease in the SR metric observed for IRCAN suggests that the augmentation of context-aware neuron weights facilitates the utilization of knowledge derived from the provided context for the LLMs. Concurrently, this adjustment allows the model to substantially disregard the intrinsic knowledge embedded within its parameters. This indicates a shift in the model's reliance from pre-stored information to dynamically acquired context, improving its adaptability and accuracy in real-time processing.Furthermore, when our method is integrated with the CAD approach, it leads to additional performance improvements over CAD across all models tested. This substantiates the complementary characteristics of our proposed methodology and the CAD approach. It implies that these two distinct strategies can collectively amplify the models' capacity to harness the information embedded within the context in unique and beneficial ways.It is also notable that, in the Original setting, LLaMA-3-8B achieves the lowest ACC, along with the highest SR. This observation may seem somewhat counterintuitive. We believe it is due to the fact that LLaMA-3-8B, trained on extensive, high-quality multi-source data, has acquired more extensive world knowledge and tends to rely more on its pre-stored intrinsic knowledge when generating responses.Multiple-choice Task As presented in Table 2 , ITI still performs poorly on the multiple-choice task. Moreover, only instructing LLMs to pay more attention to the knowledge in the context is not sufficient to enhance the model's utilization of contextual knowledge. Furthermore, the prompts with the best performance differ for different datasets and different models. Therefore, the requirement of meticulous prompt engineering undermines their generalizability. In contrast, IRCAN significantly enhances the performance of LLMs in resolving knowledge conflicts in the multiple-choice task. This improvement is observed across the COSE_KRE and ECARE_KRE datasets for the majority of the evaluated LLMs. The multiple-choice task poses a greater challenge compared to the MemoTrap dataset, where the context directly provides the required knowledge for generation. In the multiple-choice task, LLMs are required to interpret and reason based on the implicit knowledge in the context to facilitate generation. Consequently, the improvements in performance observed for the two baseline models and our proposed IRCAN are less pronounced on this dataset than those noted on MemoTrap. Nonetheless, IRCAN still achieves a gain in accuracy and a drop in SR for the majority of LLMs engaged in this task, setting new state-of-the-art results.We also observed that IRCAN does not perform as effectively for more capable models, such as LLaMA-3-8B-Instruct and LLaMA-2-13B-Chat. This may be attributed to the fact that the context in the examples of the COSE_KRE and ECARE_KRE datasets is generated by ChatGPT, which could potentially contain inaccurate information. Consequently, IRCAN struggles to precisely identify context-processing neurons in these datasets. Enhancing these neurons fails to help LLMs accurately follow context, instead relying more on their internal knowledge.To investigate the importance of context-aware neurons, we conducted a series of ablation experiments. Initially, we examined the impact on model accuracy by erasing the detected context-aware neurons. Specifically, we set the weights of these neurons to 0 to deactivate them during the forward pass. We also performed a comparative analysis by randomly enhancing or erasing the same number of neurons as implemented in IRCAN. To ensure the reliability and robustness of our experimental results, we replicated the experiments 10 times independently and reported the average of these results as the foundation for our final analysis. This was designed to minimize selection bias and to reinforce the statistical significance of our findings.The outcomes of these ablation studies illustrated in Figure 2 indicate a substantial drop in accuracy when context-aware neurons are deactivated, compared to the results of IRCAN. However, no matter whether erasing or enhancing random neurons, the performance remains similar to that of the original model. This suggests the critical role of our detected context-aware neurons in resolving knowledge conflicts, thereby validating their importance in the functionality of the model. 
effect of enhancement strength: 
 The relationship between the accuracy of the LLaMA-2-7B model on the MemoTrap dataset and the enhancement strength β is depicted in Figure 3 , where the number of enhanced neurons is fixed at 14. For further results on various enhancement strengths β, please refer to Figure 11 in Appendix D. It can be observed that as the enhancement strength for the context-aware neurons increases, model performance gradually improves, highlighting the pivotal role of the neurons identified by our method. Then, consistent with our intuition, performance begins to decline beyond a certain enhancement strength (7 in this scenario). This decline could be due to excessively high enhancements of certain neurons, leading to uncontrollable outputs or a reduction in model capabilities. 6 : The distribution of context-aware neurons across layers with Amber's 5 checkpoints.The impact of the number of enhanced neurons h on the performance of LLaMA-2-7B on the MemoTrap dataset is shown in Figure 4 . We present results where the enhancement multiplier is fixed at 7, with comprehensive results available in Appendix D (see Figure 12 ). Similarly, observations reveal that as the number of enhanced neurons increases, the model's accuracy initially improves but subsequently begins to decline, resonating with the results observed with the enhancement strength.We illustrate the distribution of the candidate set ( §3.2) of context-aware neurons identified by IRCAN across layers of Gemma-2B, LLaMA-2-7B, LLaMA-2-13B, and LLaMA-3-8B in Figure 5 . Additionally, Figure 6 depicts the changes in the distribution of these identified context-aware neurons across layers of Amber during different training stages. 3 Overall, the context-aware neurons are primarily located in the top layers, with a relatively small portion in the intermediate layers. This aligns with prior findings that language models predominantly encode "semantic" information in the top layers [38, 5] . Notably, when comparing the distributions of LLaMA-2-7B and LLaMA-2-13B with LLaMA-3-8B, we observe that the context-aware neurons of LLaMA-3-8B exhibit a more prominent aggregation in the top layers, as LLaMA-3-8B was trained on significantly more data than LLaMA-2-7B and LLaMA-2-13B. This observation is further substantiated by the distribution of context-aware neurons across five distinct checkpoints of Amber.We also analyzed the overlap of identified context-aware neurons across different prompts. Results (Appendix E) show a high degree of overlap, demonstrating IRCAN consistently identifies these neurons regardless of the prompt.To investigate whether up-weighting context-aware neurons impairs the model's general abilities, we conducted evaluations of IRCAN on six widely-used benchmarks. These benchmarks are used in the widely-recognized Open LLM Leaderboard, 4 including ARC [6] , HellaSwag [47] , MMLU [15] , Winogrande [34] , GSM8K [7] , and TruthfulQA [24] . We describe the experimental details in Appendix F. The experimental results, as shown in Table 3 , reveal that IRCAN rarely impacts the general ability of the LLMs. Surprisingly, in some cases, it even leads to a slight performance improvement. These results suggest that IRCAN can reliably improve the capability of the LLMs in addressing knowledge conflict tasks while maintaining their excellent general capabilities. 
conclusion and future work: 
 In this paper, we have presented IRCAN, a framework designed to mitigate knowledge conflicts in LLMs by identifying and reweighting context-aware neurons. Our extensive experiments across various models and tasks demonstrate that IRCAN significantly improves the fidelity of models to contextual knowledge. By enhancing context-aware neurons, IRCAN not only boosts model performance but also integrates seamlessly with existing methods, achieving state-of-the-art results in both completion and multiple-choice tasks. This work marks a significant step towards more reliable and nuanced AI systems capable of accurate context-sensitive information processing.Limitations Our current study has only experimented on a few synthetic datasets. However, exploring the effectiveness of IRCAN in scenarios such as long-context tasks and RAG is also interesting and valuable. For instance, by enhancing the model's sensitivity and fidelity to retrieved documents in context, IRCAN is expected to improve the performance of generation models in RAG systems, enabling more accurate and contextually relevant text generation. We leave this for our future work. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: When our methods are used as intended and function properly, there is no negative social impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited all the papers associated with the datasets and codes used in our work. And we have obtained the license for their use. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We have released the code and the detailed documentation to GitHub.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Examples of the datasets used in our experiments are shown in Table 4 .Write a quote that ends in the word "returned": q Long absentThe input prompt to LLMs for this task consists of the necessary instruction (e.g., "Choose the correct option to answer the following question."), context, question, and guiding suffix (e.g., "The correct answer is"). Through observation of the responses from various LLMs and multiple trials, we customized different suffixes aligned with their respective generative styles of each model, aiming to prompt them to immediately output the correct option in the continuation of the prompts. The prompts employed in our experiments across various models are illustrated in Figures 7 to 10 The original prompt and three deliberately designed prompts that instruct LLMs to pay more attention to the knowledge in the context are shown in Table 5 . Modifications relative to the original prompt are highlighted in bold.Choose the correct option to answer the following question: We conducted experiments to explore the effect of hyperparameters on model performance. Figure 11 shows variations due to changes in enhancement strength β and Figure 12 details changes associated with the number of enhanced neurons h. We intercept the results for β from 3 to 10 and h from 10 to 15 to show the results. We conducted experiments to validate the robustness of identified context-aware neurons to different prompts. Specifically, we conducted experiments on the COSE dataset to identify context-aware neurons using prompts different from those used in IRCAN for each model. We displayed the intersection of the top 300 neurons identified across different prompts, as shown in the Table 6 . The results, illustrated in Figure 13 , reveal that over 50% of the neurons identified by IRCAN coincide with those detected using alternative prompts. This significant overlap substantiates the efficacy and robustness of IRCAN in consistently identifying neurons that process contextual information across diverse prompts.Gemma-2B-it LLaMA-3-8B-Instruct 0 We use the Eleuther AI LM Evaluation Harness5 to conduct our general ability evaluation experiments. For the ARC, HellaSwag, MMLU, Winogrande, and GSM8K benchmarks, a 5-shot setting was employed, while a zero-shot setting was utilized for the TruthfulQA assessment. Metrics for evaluation varied with each benchmark: acc_norm for ARC and HellaSwag; acc for Winogrande, MMLU, and TruthfulQA (truthfulqa-mc2); and strict exact_match for GSM8K, which is consistent with the Open LLM Leaderboard. Moreover, an average of the results across these six benchmarks was calculated.Our experiments for identifying context-aware neurons can be run on a single A100 GPU with 80 GB of memory. The duration required for these experiments depends on the scale of model parameters, the size of the dataset, and the length of individual examples within the dataset. Overall, the time consumption is entirely acceptable. Taking the COSE_KRE dataset as an example, the neuron identification experiment on the smallest model (2B parameters) completes in less than 2 hours, and the experiment on the largest model (13B parameters) takes less than 18 hours.Importantly, the IRCAN does not introduce any additional inference time costs. The identification and reweighting of context-aware neurons are performed offline, allowing the modified model to be directly utilized during online testing. As a result, the inference process remains unaffected in terms of computational time. Justification: We provide sufficient information on the computer resources needed to reproduce the experiments, including GPU type, GPU memory and time of execution, in Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research conducted in our paper conforms with the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://llama.meta.com/llama3/Amber offers 360 checkpoints (0 to 359) from various training stages. Checkpoint 359 is the final one. Five evenly distributed checkpoints were selected for experiments.https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboardhttps://github.com/EleutherAI/lm-evaluation-harness 


Decision: This paper aims to address the issues where LLMs’ parametric knowledge contradicts that in the context. A technique is proposed to mechanically identify neurons responsible for processing the context, after amplifying their influence on the model, the models are better at respecting the information in the context.

The reviews are unanimously positive. The response addressed most of the concerns, and authors already have a plan to incorporate the reviewers’ feedback into the revision. Accept.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel framework, IRCAN, aimed at addressing knowledge conflicts in Large Language Models (LLMs). By identifying and enhancing neurons that are crucial for processing contextual cues using an attribution score derived from integrated gradients, the framework significantly improves the generation of context-sensitive outputs. Tested across various models and tasks, IRCAN not only enhances model performance notably but also integrates seamlessly as a plug-and-play solution with existing models, establishing new performance benchmarks in handling knowledge conflicts.
strength_weakness: strength_weakness: 1. Lack comparisons between other steering methods, like steering probed directions or steering SAE's features.
2. The propose methods will cost more inference time.
3. I think knowledge conflict also has its benefits. It may help LLMs defend some jailbreaking or harmful uses. So increase the model's faithfulness to its context may introduce some safety problems.
questions: questions: Who you choose neurons? How do neurons compare with probed directions (https://arxiv.org/abs/2306.03341) or features from SAEs (https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html#assessing-interp/) ?
limitations: limitations: As mentioned in weaknesses.
paper_summary: paper_summary: The paper proposed a new framework, IRCAN, to enable LLMs to pay more attention to new knowledge in context and generate context-sensitive outputs. The framework first identifies neurons that significantly contribute to context processing by utilizing a context-aware attribution score derived from integrated gradients and then reweighting these neurons. Experiments show the framework can effectively mitigate knowledge conflicts while not harming the general abilities of LLMs.
strength_weakness: strength_weakness: 1. The authors state they have discussed the limitations in the checklist, but no limitation section is found.
questions: questions: 1. Why the paper does not compare the proposed framework with fine-tuning? How will instruction-tuning the model to be more sensitive to new knowledge in context help? While the proposed framework and fine-tuning both involve updating parameters and the proposed framework costs many hours that might not be much more efficient than fine-tuning, it seems natural to consider fine-tuning as a baseline.
2. What is the performance of instructing the LLM to pay more attention to the knowledge in context in the prompt? For the multi-choice task, it seems that simply instructing the model to adopt the knowledge in the context will be effective enough, which will cost much less compared to the framework.
limitations: limitations: The authors state they have discussed the limitations in the checklist, but no limitation section is found.
paper_summary: paper_summary: The paper addresses the valuable problem of mitigating parametric and contextual knowledge conflicts in LLM generation with a novel and reasonable method. It is well-written, with a comprehensive experimental design showing significant improvements in completion and multi-choice tasks. However, the evaluation is limited to short contexts, raising concerns about scalability to longer contexts. The assumption that contexts are contradictory to parametric knowledge may not always hold, and the method's performance on RAG tasks and its impact on inference speed needs further exploration. Additionally, unexpected performance results between llama3-8b and llama2-7b require explanation.
strength_weakness: strength_weakness: - W1: The evaluation is limited to datasets with relatively short contexts, it is questionable whether the proposed method can scale to long contexts.
- W2: The paper assumes that the contexts are contradictory to the parametric knowledge. However, in many cases, only a tiny fraction of the context is inconsistent with the parametric knowledge, while the others are consistent or unknown. Is the proposed method still valid under these situations?
- W3: The (partly) contradiction may frequently occur in RAG, it would be great to know how this method performs on RAG tasks.
- W4: It seems the proposed method has to select neurons first before answering and this involves forward passes once with and once without contexts, will this process make the inference significantly slower? Some results and discussions on time complexity would be highly appreciated.
- W5: As shown in Table 1, llama3-8b performs worse than llama2-7b, which is not as expected. Is there any explanation for this?
- W6: Are the salient neurons selected specifically for each question (with or without context), or are shared across data points like this: https://arxiv.org/abs/2311.15983, why or why not?
questions: questions: Please refer to W1-W6.
limitations: limitations: Yes.
paper_summary: paper_summary: This work presented a framework IRCAN to locate key neurons for processing contextual cues, thereby mitigating conflicts between knowledge obtained from pre-training and knowledge within the context. Experiments on completion and multi-choice tasks showed that the IRCAN benefits the base model in knowledge conflict tasks.
strength_weakness: strength_weakness: 1. While the paper has demonstrated the effectiveness of the proposed IRCAN on 3 datasets, its performance on additional datasets remains unexplored. It would be beneficial for the authors to include more knowledge-related datasets in future versions of the paper to further validate the generalizability of the model.
2. In the completion task, the experiments compared the accuracy of the proposed IRCAN and baselines, and the IRCAN improved the performance by a large margin. However, the experiments are conducted on merely one dataset, weakening the generalization of the IRCAN, and there is no computing comparison, for example, GPU time-consuming.
questions: questions: 1. This paper assumed that contextual knowledge was more reliable than parametric knowledge, and what if the context itself was fake or misleading?
2. How did the authors construct knowledge datasets along with the context?
3. The paper mentioned: “To calculate the attribution score Attr(n^{l}_{i}), we gradually change the activation value of a neuron n^{l}_{i} from v^{l}_{q_{i}} to v^{l}_{(c,q)_{i}}…”, what are the details of the activation value’s change process?
4. In section 3.3, what is the object W(n^{l}_{i}) for the reweighting? Activation value or the attention matrix?
5. For the evaluation of general abilities, results in Table 3 show that IRCAN can cause other knowledge-related tasks like MMLU and Winograd to fluctuate. What could be the cause of this fluctuation? Does that demonstrate that emphasizing the context may bring knowledge degradation?
limitations: limitations: Various LLMs were employed in this paper, however, most of the models were 7B/8B. The authors could validate the proposed IRCAN in larger LLMs to complete the work in the future.

Comments: title: Thanks for the rebuttal!
Thanks for your reply
comment: Thank you for your supportive comments and positive evaluations. We sincerely appreciate the time and effort you have dedicated to reviewing our responses.
We sincerely appreciate the time and effort you have dedicated to reviewing our paper and our rebuttal. We are grateful for your constructive comments and valuable insights.
We sincerely thank you for your constructive feedback and for the revised score. Your insightful comments and suggestions have been instrumental in refining the quality of our paper.
Thank you for the detailed responses, my concerns were addressed, and I will change my previous score.
Thanks for the rebuttal and I will keep my evaluation.
Thanks for the responses, which addressed my questions. I will retain my positive scores.
Thank you very much for your positive feedback and for improving the score of our paper. We sincerely appreciate your valuable comments and suggestions, which have greatly contributed to enhancing the quality of our work.
Thanks for your response. I will improve the score from 5 to 6. Please make sure these new experiments are appropriately included in the revised version. I would recommend the authors discuss the application scenarios of the proposed method in the limitation or discussion section.
Thank you for your prompt and insightful response. We appreciate the time and effort you've taken to review our rebuttal. 

The RAG technology supplements models by fetching external data in response to queries, thus ensuring more accurate and current outputs. In fact, in the field of RAG, researchers have recognized that noise in retrieved external data can adversely affect the quality of generated content. To address this, they often employ post-retrieval processing techniques to remove noise from the retrieved documents.

For instance, some researchers incorporate a **re-ranking** stage subsequent to the initial retrieval process, where the retrieved documents are reassessed, scored, and reorganized to more effectively emphasize those most relevant to the query while diminishing the influence of less relevant ones. Methods such as sequence pair classification and re-scoring are introduced to re-rank documents [1-5], thereby improving the relevance between the retrieved content and the query.

Additionally, some approaches involve **filtering** phase to remove documents that fail to meet specified quality or relevance standards. For example, FiD-TF [6] and RECOMP [7] focus on removing irrelevant or redundant tokens and information from retrieved documents. Self-RAG [8] introduces a self-reflection mechanism to efficiently filter out irrelevant content.

Therefore, we believe that addressing the noise problem in retrieved text is a critical issue that the RAG field is actively tackling and one that the RAG community should indeed resolve. Our IRCAN method can complement these efforts by enhancing the model’s sensitivity and fidelity to re-ranked or filtered text, thereby further improving the effectiveness of RAG and its ability to deliver more accurate and current outputs.

 

[1] Glass et al. Re2g: Retrieve, Rerank, Generate. NAACL 2022.

[2] Dai et al. Promptagator: Few-shot Dense Retrieval From 8 Examples. ICLR 2023.

[3] Ram et al. In-Context Retrieval-Augmented Language Models. TACL 2023.

[4] Shao et al. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy. EMNLP 2023.

[5] Hofstätter et al. Fid-light: Efficient and effective retrieval-augmented text generation. SIGIR 2023.

[6] Berchansky et al. Optimizing Retrieval-augmented Reader Models via Token Elimination. EMNLP 2023.

[7] Xu et al. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation. arXiv, abs/2310.04408.

[8] Asai et al. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv, abs/2310.11511.
Thanks for your reply. The new experiments address my concerns well. I have one more question about your limitation section.
> For instance, by enhancing the model’s sensitivity and fidelity to retrieved documents in context, IRCAN is expected to significantly improve the performance of generation models in RAG systems

In RAG systems, the retrieved text can sometimes be noisy. Would enhancing the model’s sensitivity and fidelity to retrieved documents help in these scenarios?