Venue: NAACL 2025

# Steering Knowledge Selection Behaviours in LLMs

via SAE-Based Representation Engineering

 Yu Zhao\({}^{1}\) Alessio Devoto\({}^{3}\) Giwon Hong\({}^{1}\) Xiaotang Du\({}^{1}\) Aryo Pradipta Gema\({}^{1}\)

Hongruu Wang\({}^{2}\) Xuanli He\({}^{4}\) Kam-Fai Wong\({}^{2}\) Pasquale Minervini\({}^{1,5}\)

###### Abstract

Large language models (LLMs) can store a significant amount of factual knowledge in their parameters. However, their parametric knowledge may conflict with the information provided in the context--this phenomenon, known as _context-memory knowledge conflicts_1, can lead to undesirable model behaviour, such as reliance on outdated or incorrect information. Analysing the internal activations of LLMs, we find that they can internally register the signals of knowledge conflict at mid-layers. Such signals allow us to detect whether a knowledge conflict occurs and use _inference-time_ intervention strategies to resolve it. In this work, we propose SPARE, a _training-free_ representation engineering method that uses pre-trained sparse auto-encoders (SAEs) to control the knowledge selection behaviour of LLMs. SpARE identifies the functional features that control the knowledge selection behaviours and applies them to edit the internal activations of LLMs at inference time. Our experimental results show that SpARE can effectively control the usage of either knowledge source to resolve knowledge conflict in open-domain question-answering tasks, surpassing existing representation engineering methods (\(+10\%\)) as well as contrastive decoding methods (\(+15\%\)).

Footnote 1: We will refer to these as _knowledge conflicts_ for brevity.

\begin{tabular}{c c}  & **Context:** Geoffrey Hinton is a computer scientist and cognitive scientist. In 2022, he was awarded the Nobel Prize in Physics for his contributions to deep learning. \\ \end{tabular}

## 1 Introduction

Large language models (LLMs) have shown remarkable capability to memorise factual knowledge and solve knowledge-intensive tasks Petroni et al. (2019); Brown (2020); Touvron et al. (2023); Jiang et al. (2023); Anil et al. (2023). Nevertheless, the knowledge stored in their parameters (_parametric knowledge_) can be inaccurate or outdated Xu et al. (2024). To alleviate this issue, retrieval and tool-augmented approaches have been widely adopted to provide LLMs with external knowledge (_contextal knowledge_) Karpukhin et al. (2020); Lewis et al. (2020); Wu et al. (2022); Schick et al. (2024). However, contextual knowledge may sometimes conflict with the parametric knowledge of the model, leading to what we refer to as _knowledge conflicts_. Such conflicts can cause undesired behaviour, where the model may rely on inaccurate information sources, resulting in incorrect outputs Mallen et al. (2023); Xie et al. (2024); Su et al. (2024); Wang et al. (2023); Zhao et al. (2024).

Prior research found that LLMs tend to prefer contextual knowledge (e.g., retrieved passages) over their parametric knowledge when conflicts occur Su et al. (2024); Xie et al. (2024); Hong et al. (2024). For instance, Su et al. (2024) show that most LLMs choose parametric knowledge in less than \(10\%\) examples. However, in more general applications, LLMs should retain the ability to use their parametric knowledge when presented with misinformation Chen and Shu (2023); Ren et al. (2024); Mullen et al. (2023); Zhong et al. (2023). Existing works investigate fine-tuning and prompting-based strategies to detect and resolve knowledge

Figure 1: In the event of a knowledge conflict, the model can rely on the context or on the parametric knowledge. The figure presents the predictions of Llama2·7B steered by SpARE.



conflicts (Wang et al., 2023); however, they need additional interactions with the model, e.g., by asking the LLMs to examine the conflicts sentence by sentence, resulting in high latency times and preventing practical applications.

In this work, we investigate _representation engineering_ methods to efficiently steer the usage of parametric and contextual knowledge of LLMs at _inference time_. Although representation engineering has provided an efficient and transparent framework for controlling the behaviour of LLMs, we find that existing methods fail to effectively steer knowledge usage. This may be because these methods directly modify the internal activations of LLMs, such as hidden states Turner et al. (2023); Zou et al. (2023) or MLP activations Qiu et al. (2024); Meng et al. (2022). These activations are polysemantic dense vectors that overlap with many independent semantic features Olah (2023). Thus, minor edits in one dimension can influence multiple semantic features, making it difficult to adjust activations accurately without affecting other features in practice.

Recently, sparse auto-encoders (SAEs) have been proposed to address the difficulty of interpreting polysemantic activations by decomposing them into a large-scale monosemantic feature dictionary Huben et al. (2024); Gao et al. (2024); Templeton et al. (2024). Therefore, we introduce SAEs as a tool for precise activation editing to guide the knowledge selection of LLMs. Specifically, we propose SpARE, a **S**parse **A**uto-Encoder-based **R**epresentation **E**ingneering method to steer the knowledge selection behavior of LLMs. SpARE first identifies the SAE activations that are related to specific knowledge selection behaviours Section4.2); then, it extracts functional features that control the usage of contextual and parametric knowledge, and finally applies them to steer the behaviour of the model (Section4.3).

Our experimental results on open-domain question-answering tasks show that SpARE effectively controls the knowledge selection behaviours by utilising a small set of SAE features, e.g., less than \(0.05\%\) SAE activations for Gemma2-9B in the \(6\) layers2. SpARE yields more accurate results than state-of-the-art representation engineering methods (\(+10\%\)), contrastive decoding (\(+15\%\)), and incorrect learning (\(+7\%\)), achieving the best performance on steering knowledge selection behaviours of LLMs under knowledge conflicts.

Footnote 2: For Gemma2-9B, we use the pre-trained SAEs from GemmaScope [https://huggingface.co/google/gamma-sco](https://huggingface.co/google/gamma-sco) pe, and the selected activations is presented in AppendixD.

## 2 Background

Problem SetupFollowing Longpre et al. (2021); Hong et al. (2024); Xie et al. (2024), we use open-domain question-answering (ODQA) tasks to investigate the behaviour of LLMs when there is a conflict between the parametric knowledge of the model and contextual knowledge. In ODQA datasets with knowledge conflicts, each instance is presented as \((Q,E_{M},M,E_{C},C)\), where \(Q\) is the question, \(E_{M}\) is the evidence that supports the memorised knowledge stored in the model parameters, \(E_{C}\) is the evidence that conflicts with the language model's memorised knowledge, \(M\) is the answer based on the \(E_{M}\), and \(C\) is the answer based on the \(E_{C}\).

Sparse Auto-EncodersRecent works have proposed using sparse auto-encoders (SAEs) to interpret the complex representations of LLMs by decomposing them into a large set of monosemantic features Huben et al. (2024); Gao et al. (2024); Templeton et al. (2024). Given an activation \(\mathbf{h}\in\mathbb{R}^{d}\) from the residual stream of LLMs, a SAE with \(n\) latent dimensions encodes it into a sparse vector \(\mathbf{z}\in\mathbb{R}^{n}\) and decodes it to recover \(\mathbf{h}\):

\[f_{\theta}(\mathbf{h})=\sigma\left(\mathbf{W}_{\theta}\left( \mathbf{h}-\mathbf{b}\right)+\mathbf{b}_{\theta}\right)=\mathbf{z}, \tag{1}\] \[g_{\phi}(\mathbf{z})=\mathbf{W}_{\phi}\,\mathbf{z}=\sum_{i=1}^{ n}z_{i}\mathbf{f}_{i}+\mathbf{b}=\hat{\mathbf{h}}\]

where \(\sigma\) is an activation function that outputs a non-negative value such as ReLU, \(\mathbf{W}_{\theta}\in\mathbb{R}^{n\times d}\), \(\mathbf{b}\in\mathbb{R}^{d}\), \(\mathbf{b}_{\theta}\in\mathbb{R}^{n}\), \(\mathbf{W}_{\phi}\in\mathbb{R}^{d\times n}\), \(z_{i}\) is the \(i\)-th element of the SAE activation \(\mathbf{z}\), and \(\mathbf{f}_{i}\in\mathbb{R}^{d}\) is the \(i\)-th column of \(\mathbf{W}_{\phi}\). The \(\left\{\mathbf{f}_{i}\right\}_{i=1}^{n}\) learned through the SAE are considered highly monosemantic, and the SAE activation \(\mathbf{z}\) indicates the activated values of \(\left\{\mathbf{f}_{i}\right\}_{i=1}^{n}\).

## 3 Detection of Knowledge Conflicts

In this section, we investigate whether we can detect the occurrence of conflicts during the generation process, since identifying such conflicts is a prerequisite for exploring inference-time strategies to control the LLM.

We focus on the residual stream Elhage et al. (2021) of the model and look for a signal of knowledge conflict. To this end, we create two groups of input instances, \(\mathcal{D}_{E_{M}}=\left\{(Q,E_{M})\right\}\) and 

\(=\{(Q,E_{C})\}\). In \(\mathcal{D}_{E_{M}}\), the model is provided with a context that is coherent with the model internal memorized knowledge, whereas in \(\mathcal{D}_{E_{C}}\) the model is provided with a context that does not agree with model parametric knowledge, thus causing a knowledge conflict. To determine whether a signal of conflict arises in the residual stream, we focus on the last position of the sequence during generation, which is supposed to encode the information to predict the first token of the answer.

We apply a linear probing method Conneau et al. (2018); Zhu and Li (2023); Allen-Zhu and Li (2023) to investigate whether the residual stream contains a signal of knowledge conflict. Specifically, we train logistic regression models to classify whether a given activation (the hidden state, MLP or Self-Attention activations) is from the \(\mathcal{D}_{E_{C}}\) or \(\mathcal{D}_{E_{M}}\), i.e. whether it contains a knowledge conflict or not. We use activations from each layer as input and formulate this as a binary classification task. The evaluation is conducted on a held-out test set. We present probing results on Llama2-7B Touvron et al. (2023) and Gemma2-9B Riviere et al. (2024) using AUROC as metric in Fig. 2. We observe that the probing accuracy increases from the first layer to the middle layers, and this trend is the same across different types of activations. This indicates that we can detect the signal of knowledge conflict in the residual stream of the mid-layers. The probing accuracy decreases in the later layers, especially for MLP and Self-Attention activations, which indicates that MLP and Self-Attention modules do not further add the signal of conflicting knowledge to the residual stream. We provide more details and analysis about knowledge conflict detection in Appendix A and Zhao et al. (2024b).

The above analysis shows that knowledge conflicts can be identified in the internal states of LLMs. Moreover, it provides insight into which layers can be more influential in the knowledge selection (Section 6.1).

## 4 Resolving Knowledge Conflicts by Representation Engineering

In this section, we introduce SpARE, our SAEB-based representation engineering method, to steer the usage of parametric and contextual knowledge to generate the answers. SpARE consists of the three following steps: _1)_ collecting activations that lead to different knowledge selection behaviours (Section 4.1); _2)_ identifying SAEA activations that are related to each knowledge selection behaviour (Section 4.2); _3)_ steering the usage of either knowledge source by editing the hidden states of LLMs at inference time (Section 4.3).

### Collecting Activations with Different Knowledge Selection Behaviours

We showed in Section 3 that we can detect the knowledge conflict by probing the residual stream. We now want to characterise the activations that lead to different knowledge selection behaviours. To this end, given a set of instances \(\mathcal{D}_{E_{C}}\) that cause a knowledge conflict, we separate it into two groups based on the model's predictions: \(\mathcal{D}_{C}\), where the model generates an answer that aligns with the context, and \(\mathcal{D}_{M}\), where the model ignores the context and generates an answer relying on the parametric knowledge. These two subsets characterise two knowledge selection behaviours of the model. In the following, we omit the notation to specify the layer of **h** and **z** for simplicity, as the method can be applied to arbitrary layers. We collect the hidden state at the last position of the input that is used to generate the first token of the answer.

We collect the hidden states from \(\mathcal{D}_{C}\) and \(\mathcal{D}_{M}\) for \(N\) samples, denoting them as \(\{\mathbf{h}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{h}^{j}_{M}\}_{j=1}^{N}\), respectively. We then obtain the SAE activation for each sample by \(\mathbf{z}^{j}_{C}=f_{\theta}(\mathbf{h}^{j}_{C})\) and \(\mathbf{z}^{j}_{M}=f_{\theta}(\mathbf{h}^{j}_{M})\). Finally, we compute the average of the sets \(\{\mathbf{z}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{z}^{j}_{M}\}_{j=1}^{N}\) to obtain the mean vectors \(\overline{\mathbf{z}_{C}}\) and \(\overline{\mathbf{z}_{M}}\), respectively. More details are presented in Appendix C.1 and Appendix C.3.

At this stage, \(\overline{\mathbf{z}_{C}}\) and \(\overline{\mathbf{z}_{M}}\) contain the information to steer the generation towards \(C\) or \(M\). However, there might still be instance-specific activations with non-zero values that are not responsible for the knowledge selection behaviour. In the next section, we identify functional activations related to

Figure 2: The knowledge conflict probing results of Llama2-7B and Gemma2-9B on NQSwap Longpre et al. (2021). The probing results on hidden states, MLP and Self-Attention activations are coloured differently.

 

knowledge selection behaviours and then construct two orthogonal SAE activations, \({\bf z}_{C}\) and \({\bf z}_{M}\) for steering the knowledge selection behaviours.

### Identifying Functional SAE Activations

As shown by previous works Gao et al. (2024); Templeton et al. (2024a), a single SAE activation can capture one monosemantic feature. In this work, we hypothesise a combination of a small set of SAE activations can be responsible for a functional feature, such as knowledge selection in case of conflict. Our hypothesis is motivated by Task Vector Hendel et al. (2023); Todd et al. (2024), which shows that hidden states contain the functional information that drives a task.

We now show how we find the SAE activations that are responsible for driving the knowledge selection. First, we calculate mutual information between each SAE activation and the knowledge selection behaviours, which measures to which extent the behaviour depends on each activation. Let the random variable \(Z_{i}\) be the \(i\)th activation of SAE, and \(Y=\left\{C,M\right\}\) be the generated answers; we calculate the mutual information \(I(Z_{i};Y)\) between them. A higher \(I(Z_{i};Y)\) indicates a higher dependency between \(Z_{i}\) and the knowledge selection behaviour. We then select the top-\(k\) activations with the highest \(I(Z_{i};Y)\), denoted as \(\mathcal{Z}\). More details are available in Appendix C.2

In the following, we determine which knowledge selection behaviour each \(Z_{i}\in\mathcal{Z}\) positively correlates with. Given the sets of activations \(\{{\bf z}^{j}_{C}\}_{j=1}^{N}\) and \(\{{\bf z}^{j}_{M}\}_{j=1}^{N}\), we estimate the expected value of each activation feature \(Z_{i}\in\mathcal{Z}\) in both sets, denoted as \(\mathbb{E}_{C}[Z_{i}]\) and \(\mathbb{E}_{M}[Z_{i}]\). We then have that \(Z_{i}\) is positively correlated with the behaviour of selecting contextual knowledge if \(\mathbb{E}_{C}[Z_{i}]-\mathbb{E}_{M}[Z_{i}]>0\). Conversely, if this condition is not met, \(Z_{i}\) is positively correlated with the behaviour of selecting parametric knowledge. Finally, we construct two functional SAE activations \({\bf z}_{C}\) and \({\bf z}_{M}\in\mathbb{R}^{n}\), that steer the usage of contextual and parametric knowledge, respectively. For each element, \(z_{Ci}\) and \(z_{Mi}\) are set to \(0\) if \(Z_{i}\notin\mathcal{Z}\), and the remaining values are taken from \(\overline{{\bf z}_{C}}\) and \(\overline{{\bf z}_{M}}\) based on their expectations:

\[z_{Ci}=\begin{cases}\overline{z_{Ci}},&\text{if $\mathbb{E}_{C}[ Z_{i}]-\mathbb{E}_{M}[Z_{i}]>0$}\\ 0,&\text{otherwise}\end{cases}\] \[z_{Mi}=\begin{cases}\overline{z_{Mi}},&\text{if $\mathbb{ E}_{C}[Z_{i}]-\mathbb{E}_{M}[Z_{i}]<0$}\\ 0,&\text{otherwise}\end{cases}\]

### Editing Activations to Steer Behaviours

In the following, we introduce how we utilise the functional activation \({\bf z}_{C}\) and \({\bf z}_{M}\) to control the usage of knowledge sources at inference time. Suppose we want to control the LLM to use its parametric knowledge and ignore the conflict contextual knowledge that might be misinformation. In this case, we aim to _remove_ the features that steer the contextual knowledge usage and _add_ the features that steer the parametric knowledge usage. To avoid removing or adding unnecessary features, we restrict the values to edit by the following two constraints. Let an activation be \({\bf h}\) and corresponding SAE activations be \({\bf z}=f_{\theta}({\bf h})\). First, in Eq. (2), we determine the value we need to remove from \(z_{i}\) to avoid the undesired behaviour, i.e., generating contextual knowledge in this case. At this step, we ensure that the resulting activation remains non-negative after the removal, i.e., subtract at most \(z_{i}\) when \(z_{i}<z_{Ci}\):

\[z_{i}^{-}=\min\left\{z_{i},z_{Ci}\right\}. \tag{2}\]

Then, in Eq. (3), we determine the value we need to add to \(z_{i}\) to encourage the desired behaviour,

Figure 3: The workflow of SPARE steers the knowledge selection behaviour. The figure presents an example of steering the model to use parametric knowledge. First, the SAE encoder \(f_{\theta}\) encodes hidden state \({\bf h}\) into the SAE activation \({\bf z}\). Then, it determines the values of SAE activations \({\bf z}^{-}\) and \({\bf z}^{+}\) for editing (Eq. (2) and Eq. (3)). Finally, we edit the hidden state using the features extracted from the SAE decoder \(g_{\phi}\) (Eq. (4)).

 

i.e., generating parametric knowledge in this case. Here, we ensure that no excess value is added once the activation reaches \(z_{Mi}\):

\[z^{+}_{i}=\max\left\{z_{Mi}-z_{i},0\right\}. \tag{3}\]

Finally, we obtain the edited hidden states \(\mathbf{h}^{\prime}\) by:

\[\mathbf{h}^{\prime}=\mathbf{h}+\alpha\left(-g_{\phi}\left(\mathbf{ z}^{-}\right)+g_{\phi}\left(\mathbf{z}^{+}\right)\right), \tag{4}\]

where \(\alpha\in\mathbb{R}^{+}\) is a user-defined hyperparameter that controls the degree of editing. Note that we do not directly edit the activation \(\mathbf{z}\) of hidden state \(\mathbf{h}\) to obtain the modified hidden states by \(\mathbf{h}^{\prime}=g_{\phi}(\mathbf{z}^{\prime})\) with \(\mathbf{z}^{\prime}=\mathbf{z}-\mathbf{z}^{-}+\mathbf{z}^{+}\) for two reasons: _1)_ it can result in unexpected information loss of original \(\mathbf{h}\) due to the reconstruction loss of the SAE, leading to a nearly zero accuracy in our experiments; _2)_ the definition of the SAE activation shown in Eq. (1) requires each element of \(\mathbf{z}\) be a non-negative value, which prevents us from using \(\alpha\) to flexibly control the strengthen of editing like Eq. (4).

Similarly, if we want to control the model to be faithful to the context in the case of the contextual knowledge is more likely correct, we can swap \(z_{Ci}\) and \(z_{Mi}\) in Eq. (2) and Eq. (3). In this work, we only edit hidden states at the last position of the input for ODQA tasks.

## 5 Experimental Results

### Settings

DatasetsWe use two widely adopted open-domain question-answering datasets with knowledge conflicts NQSwap Longpre et al. (2021) and Macnoise Hong et al. (2024) to investigate the controlling capability of several methods.

ModelsWe evaluate our method using Llama3-8B Dubey et al. (2024) and Gemma2-9B Riviere et al. (2024), which have corresponding public pre-trained SAEs. Moreover, we also evaluate our method using Llamaa2-7B Touvron et al. (2023) with our pre-trained SAEs to examine the feasibility of adopting SpARE to an LLM without public SAEs. More details are presented in Appendix B.

EvaluationWe use the greedy decoding method to evaluate the LLMs in an open-ended generation setting. We use \(3\) in-context demonstrations to align the answer format. The demonstrations use non-conflict evidence \(E_{M}\) and memorised answer \(M\), so they do not point out which knowledge source to select. More details are presented in Appendix C.4. The test examples use \(E_{C}\), leading to a knowledge conflict for LLMs, and a behaviour-controlling method needs to steer the usage of either parametric or contextual knowledge to generate the answer. We compare the evaluation results under control with the results without any control to show each method's controlling capability.

BaselinesWe compare SpARE against the following inference-time _representation engineering_ methods: 1) TaskVec Hendel et al. (2023); 2) ActAdd Turner et al. (2023); 3) SEA Qiu et al. (2024) with linear and non-linear versions, noted by subscript "linear" and "SqExp". We compare with the following _contrastive decoding_ methods: 1) DoLa Chuang et al. (2024); 2) CAD Shi et al. (2024). Moreover, we also compare using in-context learning (ICL) Brown (2020) to steer the knowledge selection. We use \(E_{C}\) and \(C\) in the demonstrations to guide the model to ignore its parametric knowledge and use the contextual knowledge, and use \(E_{C}\) and \(M\) to guide the model to ignore the contextual knowledge and use its parametric knowledge. ICL is not an inference-time strategy because it requires changing the original input of the model to achieve a desired behaviour. More details of baseline implementation and hyperparameters searching are presented in Appendix C.5 and Appendix C.6.

HyperparametersWe select the proper hyperparameters for SpARE in the developments set that is also used to select the hyperparameters of baselines, and the details are presented in Appendix C.6. In the following, we apply SpARE from the 12th to the 15th and 13th to the 16th layer for Llama2-7B and Llama3-8B and from the 23rd to 25th and the 29th to 31st layers for Gemma2-9B; we analyse the performance of editing individual layers in Section 6.1.

### Overall Performance Comparison

MetricsWe use Exact Match (EM) to evaluate the performance. Specifically, we evaluate the control capability of generating contextual or parametric answers using the following metrics:

* [noitemsep,topsep=0pt]
* accuracy of steering the usage of contextual knowledge to generate answers \(C\).
* accuracy of steering the usage of parametric knowledge to generate answer \(M\).

Experimental ResultsWe present the main results in Table 1. First, we find SpARE _outperforms existing representation engineering methods 

TaskVec, ActAdd and SEA on steering the usage of both contextual and parametric knowledge. This indicates that SpARE can more accurately extract features related to knowledge selection behaviours through the SAE and use them to steer the generation more effectively.

Second, we find SpARE _outperforms contrastive decoding methods_ DoLa and CAD, especially in steering the usage of parametric knowledge. Though contrastive decoding strategies can effectively improve the use of contextual knowledge, they struggle to steer the use of parametric knowledge. In contrast, SpARE can more effectively steer the usage of both knowledge by adding and removing the desired and undesired functional features, which we will further analyse in the later ablation study.

Moreover, SpARE _surpasses the non-inference-time controlling method_ ICL. It suggests the SpARE can both effectively and efficiently control the knowledge selection behaviours of LLMs. It also suggests a promising capability of representation engineering to control the behaviours of LLMs at inference time in practical applications.

### Multi-Perspective Controlling Analysis

In the following, we analyse the controlling capability of SpARE from different perspectives: _1)_ the capability of changing the behaviour (Fig. 3(a)), _2)_ the potential negative impact of intervention (Fig. 3(b)), and _3)_ the ablation study (Fig. 3(c)).

Capability of Changing the BehavioursUnlike merely comparing overall performance across the entire dataset, we further examine their capability of changing the original knowledge selection behaviour of LLMs by the following two metrics:

\(\textbf{EM}_{C\to M}\) accuracy of changing the behaviour from generating contextual answers \(C\) to parametric answers \(M\) in the subset of instances where the model generates \(C\) without controlling.

\(\textbf{EM}_{M\to C}\) accuracy of changing the behaviour from generating parametric answers \(M\) to contextual answers \(C\) in the subset of instances where the model generates \(M\) without controlling.

As shown in Fig. 3(a), SpARE outperforms contrastive decoding methods and is located in the upper-right area of the figure. SpARE also outperforms representation engineering methods. It suggests that the SAE enables accurately extracting features related to knowledge behaviour and thus more effectively changes both the original behaviours of using contextual and parametric knowledge. We also observe that all methods are less effective in steering toward the use of parametric knowledge than contextual knowledge. This finding matches the previous works Su et al. (2024); Xie et al. (2024); Ortu et al. (2024), which shows

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline \multirow{2}{*}{**Metric**} & \multicolumn{5}{c}{**NQSwap** Longpre et al. (2021)} & \multicolumn{5}{c}{**Macnoise** Hong et al. (2024)} \\ \cline{2-7}  & & **Llama3-8B** & **Llama2-7B** & **Gamma-2-9B** & **Llama3-8B** & **Llama2-7B** & **Gamma-2-9B** \\ \hline \multicolumn{7}{|l|}{**Steer to Use Parametric Knowledge**} \\ \hline  & _Without Controlling_ & \(26.03_{k+0.02}\) & \(22.23_{k+4.75}\) & \(26.32_{k+1.80}\) & \(18.96_{k+2.65}\) & \(22.37_{k+1.89}\) & \(17.06_{k+3.75}\) \\ \hline  & TaskVec Hendel et al. (2023) & \(24.16_{k+5.65}\) & \(24.88_{k+0.85}\) & \(29.85_{k+0.83}\) & \(21.23_{k+1.89}\) & \(22.93_{k+2.31}\) & \(28.92_{k+1.19}\) \\  & ActualJ Turner et al. (2023a) & \(37.87_{k+9.96}\) & \(31.43_{k+3.68}\) & \(27.67_{k+0.82}\) & \(26.17_{k+0.22}\) & \(27.52_{k+0.77}\) & \(29.75_{k+1.68}\) \\  & SEAnzuci Qiu et al. (2024) & \(20.03_{k+1.83}\) & \(23.73_{k+0.86}\) & \(24.43_{k+0.91}\) & \(12.84_{k+1.15}\) & \(15.64_{k+0.24}\) & \(28.10_{k+2.78}\) \\ EM\({}_{M}\) & SEA\({}_{\text{edge}}\) Qiu et al. (2024) & \(13.64_{k+1.62}\) & \(16.60_{k+0.55}\) & \(23.79_{k+1.38}\) & \(14.24_{k+1.45}\) & \(16.24_{k+1.06}\) & \(28.07_{k+1.30}\) \\  & DLo Chang et al. (2024) & \(25.53_{k+1.9}\) & \(16.50_{k+3.91}\) & \(20.58_{k+1.06}\) & \(16.52_{k+2.65}\) & \(15.66_{k+0.88}\) & \(19.81_{k+2.58}\) \\   \({}^{\dagger}\)CAD Shi et al. (2024) & \(33.72_{k+0.84}\) & \(31.23_{k+1.45}\) & \(41.17_{k+0.59}\) & \(25.85_{k+0.75}\) & \(30.81_{k+0.94}\) & \(33.15_{k+2.87}\) \\  & ICI Brown (2020) & \(43.73_{k+1.55}\) & \(31.67_{k+0.49}\) & \(43.13_{k+3.63}\) & \(29.52_{k+1.16}\) & \(31.23_{k+0.34}\) & \(29.19_{k+2.35}\) \\  & SPARE Ours & **47.51**\({}_{1.30}\) & **43.76**\({}_{k+1.44}\) & **44.11**\({}_{1.30}\) & **30.72**\({}_{1.42}\) & **35.43**\({}_{1.10}\) & **35.53**\({}_{2.07}\) \\ \hline \multicolumn{7}{|l|}{**Steer to Use Contextual Knowledge**} \\ \hline  & _Without Controlling_ & \(42.69_{k+0.40}\) & \(41.67_{k+0.66}\) & \(45.96_{k+2.48}\) & \(69.36_{k+3.57}\) & \(62.38_{k+3.06}\) & \(59.25_{k+2.68}\) \\ \hline  & TaskVec Hendel et al. (2023) & \(41.88_{k+9.45}\) & \(38.25_{k+1.23}\) & \(45.52_{k+1.06}\) & \(88.47_{k+1.93}\) & \(86.91_{k+0.44}\) & \(59.25_{k+1.49}\) \\ \hline  & AccAdd Turner et al. (2023a) & \(51.91_{k+0.83}\) & \(47.48_{k+3.93}\) & \(46.90_{k+1.89}\) & \(73.01_{k+1.55}\) & \(69.64_{k+2.09}\) & \(59.69_{k+2.89}\) \\ \multicolumn{7}{|l|}{SEAmzuci Qiu et al. (2024)} & \(43.61_{k+1.03}\) & \(47.48_{k+3.93}\) & \(52.95_{k+1.90}\) & \(69.78_{k+0.97}\) & \(67.32_{k+0.28}\) & \(60.31_{k+2.25}\) \\ \multicolumn{7}{|l|}{EM\({}_{C}\)} & SEA\({}_{\text{edge}}\) Qiu et al. (2024) & \(57.08_{k+2.92}\) & \(48.04_{k+0.45}\) & \(61.45_{k+0.54}\) & \(72.04_{k+1.60}\) & \(68.20_{k+1.10}\) & \(61.45_{k+0.30}\) \\  & DOLa Chuang et al. (2024) & \(44.29_{k+3.46}\) & \(33.54_{k+3.38}\) & \(15.90_{k+0.10}\) & \(68.45_{k+3.83}\) & \(50.59_{k+1.15}\) & \(23.34_{k+10.5}\) \\  & CAD Shi et al. (2024) & \(65.65_{k+5.50}\) & \(54.69_{k+3.25}\) & \(63.10_{k+2.32}\) & \(78.69_{k+3.85}\) & \(70.07_{k+3.77}\) & \(65.61_{k+2.44}\) \\  & ICI Brown (2020) & \(73.35_{k+5.82}\) & \(63.33_{k+1.35}\) & \(70.19_{k+2.51}\) & \(51.75_{k+5.60}\) & \(47.51_{k+1.86}\) & \(47.24_{k+3.81}\) \\  & SPARE Ours & **77.69**\({}_{k+1.24}\) & **69.32**\({}_{k+1.26}\) & **73.78**\({}_{k+0.74}\) & **92.24**\({}_{k+0.49}\) & **87.30**\({}_{k+1.96}

 

LLMs prefer contextual knowledge, and thus more difficult to steer the behaviour of using parametric knowledge.

Impacts of InterventionA sufficient but unnecessary intervention can change the behaviour of LLMs, but it also can introduce noise and decrease accuracy. Here, we investigate the potential negative impact of methods by steering LLMs using the same knowledge they will use without control. We expect LLMs to maintain their original behaviours, measured by the following two metrics:

\(\textbf{EM}_{M\to M}\) accuracy of maintaining the behaviour of generating \(M\) when steering the use of parametric knowledge in the subset of instances where the model generates \(M\) without controlling.

\(\textbf{EM}_{C\to C}\) accuracy of maintaining the behaviour of generating \(C\) when steering the use contextual knowledge in the subset of instances where the model generates \(C\) without controlling.

As shown in Fig. 3(b), as it minimally alters the original behaviour when guiding the model to produce similar outcomes. SpARE has a close performance to ICL, indicating it can steer the behaviour effectively while introducing a little unnecessary editing. Though CAD maintains the most accuracy in contextual knowledge, its performance decreases substantially in maintaining the behaviour of generating parametric knowledge. Finally, while other representation engineering methods may alter the entire model behaviour due to editing of polysemantic features, SpARE provides a more precise approach to editing through the SAE activations and thus delivers better performance in maintaining the model behaviours.

Ablation StudyWe present the ablation study in the following settings: _1)_SpARE input-independent: it uses \(\mathbf{z}_{C}\) and \(\mathbf{z}_{M}\) to steer the generation without calculating \(\mathbf{z}^{-}\) and \(\mathbf{z}^{+}\) based on the input activation; 2) SpARE remove only: it edits the hidden states by only removing the functional features of the undesired behaviour; _3)_SpARE add only: it edits the hidden states by only adding the functional features of the desired behaviour.

As shown in Fig. 3(c), we can see that every ablation results in a significant controlling capability decrease. The input-independent editing strategy that omits the calculations of Eq. (2) and Eq. (3) fails to steer the usage of knowledge and obtain results that are close to the ones we obtain without controlling. The results of SpARE "remove only" obtain a zero accuracy on both EM\({}_{M}\) and EM\({}_{C}\), indicating that the model cannot keep the original behaviour and also cannot generate answers toward another behaviour. This suggests that SpARE can effectively remove the functional features of the original behaviour. Moreover, SpARE "add only" leads to worse performance than without controlling, suggesting the importance of removing the features of undesired knowledge selection behaviour.

## 6 Analysis and Discussion

### Analysing the Layer Choice

We present the results of editing multiple layers in Table 1; here, we analyse the effectiveness of SpARE by editing each layer individually. As shown in Fig. 5, we find SpARE can control the be 

haviour of LLMs most effectively at mid-layers for both Llama3-8B and Gemma2-9B. These layers are also where we can detect knowledge conflict most accurately, as shown in Fig. 2 and Appendix A. This supports the practical application of inference-time intervention to control knowledge selection behaviour, where SpARE can effectively steer the generation once we detect the conflict.

The effectiveness of steering the behaviours in middle layers also matches previous findings Hendel et al. (2023); Pan et al. (2023); Todd et al. (2024), that suggest that the middle layers of LLMs contain the functional feature that drives a task. To the best of our knowledge, we are the first to accurately extract this functional feature using pre-trained SAEs.

### Analysing the Residual Stream

We analyse how the residual stream changes after applying SpARE. Here, we edit the hidden states from \({\mathcal{D}_{E_{C}}=\left\{(Q,E_{C})\right\}}\) at the 15th layer to steer the contextual and parametric knowledge usage.

In Fig. 6a, we present the probing results on the residual stream using the same probing model described in Section 3. We observe that when we steer towards the usage of parametric knowledge, the probing performance decreases _immediately_ (green line), indicating that the signal of knowledge conflict fades quickly, and the representations of activations become closer to \({\mathcal{D}_{E_{M}}}\), thus making it more difficult for the probing model to distinguish whether a given activation is from \({\mathcal{D}_{E_{C}}}\) or \({\mathcal{D}_{E_{M}}}\). In contrast, when we steer towards using contextual knowledge, the probing performance increases (blue line), indicating the signal of the conflict increases, and the representations of activations become more different from \({\mathcal{D}_{E_{M}}}\), making it easier for the probing model to distinguish whether a given activations is from \({\mathcal{D}_{E_{C}}}\) or \({\mathcal{D}_{E_{M}}}\).

In Fig. 6b, we find the skewness of the representation from the residual stream - measured by Kurtosis - shows distinct patterns after applying SpARE. We observe that when we apply SpARE to steer the usage of contextual knowledge at the 15th layer, the residual stream becomes significantly more skewed starting from _later layers_--the 19th layer (blue line); in contrast, when we use parametric knowledge, the residual stream becomes less skewed (green line). Moreover, in Fig. 7, we analyse the skewness pattern when LLMs freely select knowledge to generate answers without controlling. We find the residual stream of \({\mathcal{D}_{C}}\), where the model uses contextual knowledge to generate answers, is significantly more skewed than \({\mathcal{D}_{M}}\) from the 19th layer. Thus, the skewness pattern changes shown in Fig. 6b can indicate that SpARE steers the knowledge selection behaviours.

We provide more analysis of the representation patterns in Appendix F and Zhao et al. (2024). In this work, we only provide our empirical observation on the representation pattern and leave investigating the reasons in future works.

## 7 Related Works

Representation EngineeringMany studies focus on _mechanistic interpretability_ to understand the LLMs by analysing the activities and connections of individual network components, such as circuits Elhage et al. (2021); Olsson et al. (2022)

Figure 5: Effectiveness of SpARE on editing different layers individually.

Figure 6: The residual stream changes after applying SpARE to Llama3-8B at the 15th layer.

Figure 7: The skewness patterns of the residual steam when LLMs select different sources of knowledge to generate the answer without controlling in NQSwap.

 

and neurons (Geva et al., 2021; Meng et al., 2022) However, though mechanistic interpretability can successfully explain simple mechanisms, it often struggles with more complex phenomena Zou et al. (2023a). Differently, _representation engineering_Turner et al. (2023a); Qiu et al. (2024); Zou et al. (2023a) offers a complementary approach. It focuses on the characteristics of representations rather than lower-level mechanisms, providing a framework for understanding complex systems at a higher level of abstraction. It has shown more promise in interpreting higher-level behaviours of LLMs at scale. Some works modify model activations to change behaviours Ravfogel et al. (2020); Iskander et al. (2023); Liu et al. (2024); Zou et al. (2023b); Li et al. (2023), and some works extract latent vectors and leveraging these vectors to regulate the model's inference Turner et al. (2023b); Subramani et al. (2022); Rimsky et al. (2023).

Knowledge Conflicts_knowledge conflicts_refer to discrepancies among contextual and parametric knowledge Chen et al. (2022); Xie et al. (2024b). Xu et al. (2024) identify three types of knowledge conflicts: _inter-context_Zhang and Choi (2021); Du et al. (2022); Pan et al. (2023b); Zhao et al. (2024c), _context-memory_Longpre et al. (2021); Xie et al. (2024b); Minder et al. (2024), and _intra-memory conflicts_Huang et al. (2023). In this work, we focus on context-memory knowledge conflicts, which refers to conflicts between the contextual knowledge and the parametric knowledge encoded in the model parameters. Ortu et al. (2024) and Jin et al. (2024) investigate the mechanisms of attention heads and feed-forward networks of LLMs when context-memory knowledge conflict occurs.

Sparse Auto-EncoderSparse Auto-Encoders (SAEs) have been introduced as a post-hoc analysis tool to identify disentangled features within uncompressed representations of an LLM Yun et al. (2021); Bricken et al. (2023); Huben et al. (2024). SAEs are trained with sparsity regularisation to learn a sparse, overcomplete basis that characterises the activation space of an LLM Bereska and Gavves (2024). Marks et al. (2024) showed that the features learned by SAEs can identify sparse circuits in LLMs. Templeton et al. (2024b) showed the possibility of searching for monosemantic features and steering LLMs' generation. Chalnev et al. (2024) improves steering vectors using SAEs. GurArieh et al. (2025) generates description for representations learned through SEAs. Lan et al. (2024) finds universal representation across different language models.

## 8 Conclusions

We investigated the context-memory knowledge conflicts in LLMs. We identify that knowledge conflicts can be detected by probing the residual stream of the model (Section 3) and propose SpARE (Section 4), a training-free representation engineering method that leverages pre-trained SAEs to effectively and efficiently control the knowledge selection behaviour of LLMs at inference time. Our experimental results on ODQA tasks show that SpARE produces more accurate results than existing representation engineering and contrastive decoding methods (Section 5). By providing a mechanism to steer knowledge selection behaviours at inference time, SpARE offers a promising approach to managing knowledge conflicts in LLMs without significant computational overhead. Additionally, we investigate the residual stream of LLMs under knowledge conflicts (Section 6). We find that _1)_ the knowledge selection behaviour is more steerable at the middle layers of LLMs; _2)_ the residual stream shows a significantly more skewed representation when models using contextual knowledge compared to using parametric knowledge.

### Limitations

While our proposed method, SpARE, demonstrates effective control over knowledge selection behaviours in LLMs, there are several limitations to consider. First, the approach relies on pre-trained SAEs to identify and manipulate functional features within the internal activations of the model, so it may not directly apply to models where pretrained SAEs are unavailable or cannot be efficiently trained. Second, our experiments are conducted on specific ODQA tasks involving context-memory knowledge conflicts. While the results are promising in this setting, it is unclear how well the method generalises to other types of tasks or conflicts, such as those involving complex reasoning, multi-hop questions, or long-form generation. Finally, the control over knowledge selection behaviours is evaluated primarily in terms of steering the model towards either contextual or parametric knowledge. In practice, the decision about which knowledge source to trust may not be binary or require a more elaborate approach, such as using another model as a critic Hong et al. (2024) 



## Acknowledgments

Yu Zhao and Xiaotang Du were partly supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by UK Research and Innovation (grant EP/S022481/1) and the University of Edinburgh, School of Informatics. Alessio Devoto was supported by Sapienza Grant RM1221816BD028D6 (DeSMOS). Giwon Hong was supported by the ILCC PhD program (School of Informatics Funding Package) at the University of Edinburgh, School of Informatics. Aryo Pradipta Gema was supported by the United Kingdom Research and Innovation (grant EP/S02431X/1), UKRI Centre for Doctoral Training in Biomedical AI at the University of Edinburgh, School of Informatics. Xuanli He was funded by an industry grant from Cisco. Pasquale Minervini was partially funded by ELIAI (The Edinburgh Laboratory for Integrated Artificial Intelligence), EPSRC (grant no. EP/W002876/1), an industry grant from Cisco, and a donation from Accenture LLP. This work was supported by the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh.

## References

* Allen-Zhu and Li (2023) Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Physics of language models: Part 1, context-free grammar. _arXiv preprint arXiv:2305.13673_.
* Anil et al. (2023) Rohan Anil, Sebastian Borgeaud, Yonghui Wu, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. _arXiv preprint arXiv:2312.11805_.
* A review. _CoRR_, abs/2404.14082.
* Bricken et al. (2023) Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schaefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. _Transformer Circuits Thread_.
* Brown (2020) Tom B Brown. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Chalnev et al. (2024) Suiatoslav Chalnev, Matthew Siu, and Arthur Conny. 2024. Improving steering vectors by targeting sparse autoencoder features. _ArXiv_, abs/2411.02193.
* Chen and Shu (2023a) Canyu Chen and Kai Shu. 2023a. Can llm-generated misinformation be detected? _arXiv preprint arXiv:2309.13788_.
* Chen and Shu (2023b) Canyu Chen and Kai Shu. 2023b. Combating misinformation in the age of llms: Opportunities and challenges. _AI Magazine_.
* Chen et al. (2022) Hung-Ting Chen, Michael J. Q. Zhang, and Eunsol Choi. 2022. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In _EMNLP_, pages 2292-2307. Association for Computational Linguistics.
* Chuang et al. (2024) Yung-Sung Chuang, Yujia Xie, Hongyi Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. Dola: Decoding by contrasting layers improves factuality in large language models. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Conneau et al. (2018) Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. _arXiv preprint arXiv:1805.01070_.
* Devoto et al. (2024) Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. 2024. A simple and effective l\({}_{2}\) norm-based strategy for KV cache compression. _CoRR_, abs/2406.11430.
* Du et al. (2022) Yibing Du, Antoine Bosselut, and Christopher D. Manning. 2022. Synthetic disinformation attacks on automated fact verification systems. In _AAAI_, pages 10581-10589. AAAI Press.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kodian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. _Transformer Circuits Thread_, 1(1):12.
* Gao et al. (2024) Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. _CoRR_, abs/2406.04093.
* Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5484-5495.
* Gur-Arieh et al. (2025) Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. 2025. Enhancing automated interpretability with output-centric feature descriptions. _arXiv preprint arXiv:2501.08319_

Roee Hendel, Mor Geva, and Amir Globerson. 2023: In-context learning creates task vectors. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 9318-9333. Association for Computational Linguistics.
* Hong et al. (2024) Giwon Hong, Jeonghwan Kim, Junmo Kang, SungHyon Myaeng, and Joyce Jiyoung Whang. 2024. Why so gullible? enhancing the robustness of retrieval-augmented models against counterfactual noise. In _Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024_, pages 2474-2495. Association for Computational Linguistics.
* Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. _CoRR_, abs/2311.05232.
* Huben et al. (2024) Robert Huben, Hoagy Cunningham, Logan Riggs, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In _ICLR_. OpenReview.net.
* Iskander et al. (2023) Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. 2023. Shielded representations: Protecting sensitive attributes through iterative gradient-based projection. In _ACL (Findings)_, pages 5961-5977. Association for Computational Linguistics.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. _arXiv preprint arXiv:2310.06825_.
* Jin et al. (2024) Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yuko Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2024. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. _arXiv preprint arXiv:2402.18154_.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Dangi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 6769-6781. Association for Computational Linguistics.
* Lan et al. (2024) Michael Lan, Philip Torr, Austin Meek, Ashkan Khakzar, David Krueger, and Fazl Berez. 2024. Sparse autoencoders reveal universal feature spaces across large language models. _arXiv preprint arXiv:2410.06981_.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kutler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* Li et al. (2023) Kenneth Li, Oam Patel, Fernanda B. Viegas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. In _NeurIPS_.
* Lieberum et al. (2024) Tom Lieberum, Senthoaran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca D. Dragan, Rohin Shah, and Neel Nanda. 2024. Gamma scope: Open sparse autoencoders everywhere all at once on gamma 2. _CoRR_, abs/2408.05147.
* Liu et al. (2024) Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. 2024. In-context vectors: Making in context learning more effective and controllable through latent space steering. In _ICML_. OpenReview.net.
* Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhl Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7052-7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishrizi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 9802-9822. Association for Computational Linguistics.
* Marks et al. (2024) Samuel Marks, Can Roger, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. _arXiv preprint arXiv:2403.19647_.
* December 9, 2022_.
* Minder et al. (2024) Julian Minder, Kevin Du, Niklas Stoehr, Giovanni Monea, Chris Wendler, Robert West, and Ryan Cotterell. 2024. Controllable context sensitivity and the knob behind it. _arXiv preprint arXiv:2411.07404_.
* Olah (2023) Chris Olah. 2023. Distributed representations: Composition & superposition. _Transformer Circuits Thread_.
* Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. _arXiv preprint arXiv:2209.11895_ 

Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Scholkopf. 2024. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. _arXiv preprint arXiv:2402.11655_.
* Pan et al. (2023a) Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023a. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 8298-8319. Association for Computational Linguistics.
* Pan et al. (2023b) Liangming Pan, Wenhu Chen, Min-Yen Kan, and William Yang Wang. 2023b. Attacking open-domain question answering by injecting misinformation. In _IJCNLP (1)_, pages 525-539. Association for Computational Linguistics.
* Petroni et al. (2019) Fabio Petroni, Tim Rocktaschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 2463-2473. Association for Computational Linguistics.
* Qiu et al. (2024) Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo M. Ponti, and Shay B. Cohen. 2024. Spectral editing of activations for large language model alignment. _CoRR_, abs/2405.09719.
* Rajamanoharan et al. (2024) Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, Janos Kramar, and Neel Nanda. 2024. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. _CoRR_, abs/2407.14435.
* Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In _ACL_, pages 7237-7256. Association for Computational Linguistics.
* Rimsky et al. (2023) Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering llama 2 via contrastive activation addition. _CoRR_, abs/2312.06681.
* Riviere et al. (2023)Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenet, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Niro Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastier Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andrew Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chiutun Kumar, Chris Perry, Chris Welty, Christopher A. Chooette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Heribson, Elisa Bandy, Emma Wang, Eric Noland, Eric Moreira, Evan Senter, Evgenii Ellyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadl Hashem, Hanna Klimczak-Plucinska, Harleen Batra, Hersh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersoort, Josh Gordon, Josh Lipschitz, Josh Newman, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badda, Kat Black, Kaite Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhi, Kish Greene, Lars Lowe Sjosund, Lauren Usui, Laurent Siffe, Lena Heuermann, Letica Lago, and Lilly McNeals. 2024. Gemma 2: Improving open language models at a practical size. _CoRR_, abs/2408.00118.
* Schick et al. (2024) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. _Advances in Neural Information Processing Systems_, 36.
* Shi et al. (2024) Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallicunate less with context-aware decoding. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Short Papers, NAACL 2024, Mexico City, Mexico, June 16-21, 2024_, pages 783-791. Association for Computational Linguistics.
* Su et al. (2024) Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jianxuo Sun, Juntao Li, Min Zhang, and Yu Cheng. 2024. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llvm. _arXiv preprint arXiv:2408.12076_.
* Subramanian et al. (2022) Nishant Subramanian, Nivedita Suresh, and Matthew E. Peters. 2022. Extracting latent steering vectors from pretrained language models. In _ACL (Findings)_, pages 566-581. Association for Computational Linguistics.
* Templeton et al. (2024a) Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Amiesien, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024a. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. _Transformer Circuits Thread_.
* Templeton et al. (2024b) Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, 

Craig Citro, Emmanuel Amiesen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte McDiparndi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024b. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. _Transformer Circuits Thread_.
* Todd et al. (2024) Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. 2024. Function vectors in large language models. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. LLama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Turner et al. (2023a) Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. 2023a. Activation addition: Steering language models without optimization. _CoRR_, abs/2308.10248.
* Turner et al. (2023b) Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. 2023b. Activation addition: Steering language models without optimization. _CoRR_, abs/2308.10248.
* Wang et al. (2023) Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023. Resolving knowledge conflicts in large language models. _CoRR_, abs/2310.00935.
* Wu et al. (2022) Yuxiang Wu, Yu Zhao, Baotian Hu, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2022. An efficient memory-augmented transformer for knowledge-intensive NLP tasks. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 5184-5196. Association for Computational Linguistics.
* Xie et al. (2024a) Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024a. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Xie et al. (2024b) Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024b. Adaptive chameleon or stubborn sldn: Revealing the behavior of large language models in knowledge conflicts. In _ICLR_. OpenReview.net.
* Xu et al. (2024) Rongwu Xu, Zehan Qi, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. Knowledge conflicts for llms: A survey. _arXiv preprint arXiv:2403.08319_.
* Yun et al. (2021) Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann Lecun. 2021. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In _Proceedings of Deep Learning Inside Out (DeeLO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 1-10, Online. Association for Computational Linguistics.
* Zhang and Choi (2021) Michael J. Q. Zhang and Eunsol Choi. 2021. Situatedqa: Incorporating extra-linguistic contexts into QA. In _EMNLP (1)_, pages 7371-7387. Association for Computational Linguistics.
* Zhao et al. (2024a) Wanru Zhao, Vidit Khazanchi, Haodi Xing, Xuanli He, Qinogkai Xu, and Nicholas Donald Lane. 2024a. Attacks on third-party apis of large language models. In _ICLR 2024 Workshop on Secure and Trustworthy Large Language Models_.
* Zhao et al. (2024b) Yu Zhao, Xiaotang Du, Giwon Hong, Arroy Pradipta Gema, Alessio Devoto, Hongru Wang, Xuanli He, Kam-Fai Wong, and Pasquale Minervin. 2024b. Analysing the residual stream of language models under knowledge conflicts. _CoRR_, abs/2410.16090.
* Zhao et al. (2024c) Yu Zhao, Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Milos, Yuxiang Wu, and Pasquale Minervini. 2024c. Analysing the impact of sequence composition on language model pretraining. _arXiv preprint arXiv:2402.13991_.
* Zhong et al. (2023) Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. 2023. Poisoning retrieval corpora by injecting adversarial passages. _arXiv preprint arXiv:2310.19156_.
* Zhu and Li (2023) Zeyuan Allen Zhu and Yuanzhi Li. 2023. Physics of language models: Part 3.1, knowledge storage and extraction. _arXiv preprint arXiv:2309.14316_.
* Zou et al. (2023) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Samni Koyejo, Dawn Song, Matt Fredrickson, J. Zico Kolter, and Dan Hendrycks. 2023b. Representation engineering: A top-down approach to AI transparency. _CoRR_, abs/2310.01405.
* Zou et al. (2024) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazekia, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmil Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. 2023b. Representation engineering: A top-down approach to AI transparency. _CoRR_, abs/2310.01405.
* Zou et al. (2024) Wei Zou, Runpeng Geng, Binghui Wang, and Jihyuan Jia. 2024. Poissonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language models. _arXiv preprint arXiv:2402.07867_ 



## Appendix A More Analysis of Knowledge Conflict Probing

### Details of Probing Model

We train the probing model with an L\({}_{1}\) norm regularisation for all probing experiments. The training objective is \(\mathcal{L}=-\log P(y=y_{i})+\lambda\|W\|_{1}\), where we set \(\lambda\) to \(3\times 10^{-4}\) and \(y_{i}\) is the label. We train \(20\) times with different random seeds for each probing task, and we report the average and deviation in our experiments. We split the training and test datasets for the probing tasks, ensuring no overlapping questions between them.

### More Probing Results

We present the knowledge conflict probing results of Llama2-7B on NQSwap and Macnoise on Fig. 8 and Fig. 9 using accuracy, AUROC and AUPRC as metrics. We provide more analysis of probing the residual stream under knowledge conflict in our preliminary study Zhao et al. (2024b).

## Appendix B Sparse Auto-Encoders Details

Both the SAEs of Llama3-8B from ElethuerAI and Llamaa2-7B pre-trained by us have a latent dimension \(n=131072\), \(32\) times larger than the number of dimensions of the hidden state size \(d=4096\), and use ReLU as activation function \(\sigma\)Eq.1). Gemma2-9B uses JumpReLU Rajamanoharan et al. (2024) as the activation function. GemmaScope Lieberum et al. (2024) provides different sizes of SAEs, and we use the size of \(n=131072\) in the experiment.

Following Gao et al. (2024), we pre-train the SAEs for Llama2-7B with TopK activation functions: \(\mathbf{z}=\text{TopK}\left(\mathbf{W}_{\theta}(\mathbf{h}-\mathbf{b})\right)\), which only keeps the \(k\) largest latents during pre-training, while does not use sparsity regularisation during pre-training. The loss is calculated by the L\({}_{2}\) norm of reconstruction error: \(\mathcal{L}=\|\mathbf{h}-\hat{\mathbf{h}}\|_{2}^{2}\). We pre-train SAE models3 with \(10\)B tokens sampled from RedPajama4 and use the hyperparamters determined by Gao et al. (2024). The pre-training for an SAE for a certain layer of hidden states costs about \(300\) &O A100 GPU hours.5

Footnote 3: [https://github.com/EleutherAI/sae](https://github.com/EleutherAI/sae)

Footnote 4: [https://huggingface.co/datasets/togethercomp](https://huggingface.co/datasets/togethercomp)

Footnote 5: We provide our pre-trained SAEs in [https://huggingface.co/yuzhaooe/Llama2-7b-SAE/tree/main](https://huggingface.co/yuzhaooe/Llama2-7b-SAE/tree/main)

Though it costs non-trivial resources for pre-training SAEs, we believe it is valuable to explore using SAEs to resolve knowledge conflicts for the following reasons: _1)_ SAEs are general models for interpreting the representation of LLMs, which have broader applications beyond steering knowledge selection behaviours; _2)_ SAEs are becoming popular tools for interpreting LLMs with rising numbers of open-resource frameworks and pre-trained models released recently, so we can use

Figure 8: Knowledge conflict probing results using Llama2-7B on NQSwap.

Figure 9: Knowledge conflict probing results using Llama2-7B on Macnoise.



the pre-trained SAEs conveniently rather than pre-training SAEs by ourselves in the future.

## Appendix C Implementation Details

### Collecting Activations

Collecting activations is an essential step in representation engineering methods Zou et al. (2023); Qiu et al. (2024), where we will extract desired features from the collected activations and use these features to edit the activations of LLMs. In SpARE, we first sample demonstrations from the development set using \(5\) seeds. Then, we use these demonstrations to test the rest of the questions with conflict evidence \(E_{C}\). Based on the predictions, we split the instance into \(\mathcal{D}_{C}\) and \(\mathcal{D}_{M}\) and collect their corresponding hidden states, denoting as \(\{\mathbf{h}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{h}^{j}_{M}\}_{j=1}^{N}\). Then, the corresponding layer's SAE encodes them to \(\{\mathbf{z}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{z}^{j}_{M}\}_{j=1}^{N}\). Here, we use \(j\) to index the collected instances and use \(i\) to index the SAE's activation.

One strategy to highlight the values of behaviour-related activation is weighted averaging \(\{\mathbf{z}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{z}^{j}_{M}\}_{j=1}^{N}\) by the confidence of generating a specific answer. Specifically, denote the confidence of generating answers \(C\) and \(M\) conditioned on \(\mathbf{h}^{j}_{C}\) and \(\mathbf{h}^{j}_{M}\) as

\[\lambda^{j}_{C} =\frac{\log P(C\mid\mathbf{h}^{j}_{C})}{\log P(C\mid\mathbf{h}^{ j}_{C})+\log P(M\mid\mathbf{h}^{j}_{C})}\] \[\mu^{j}_{M} =\frac{\log P(M\mid\mathbf{h}^{j}_{M})}{\log P(C\mid\mathbf{h}^{ j}_{M})+\log P(M\mid\mathbf{h}^{j}_{M})},\]

where \(P(\cdot\mid\mathbf{h})\) is calculated by the output of LLMs. We use the normalised confidence \(\lambda^{{}^{\prime}j}_{C}\) and \(\mu^{{}^{\prime}j}_{M}\) as weight to average \(\{\mathbf{z}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{z}^{j}_{M}\}_{j=1}^{N}\), respectively:

\[\overline{\mathbf{z}_{C}}=\sum_{j=1}^{N}\lambda^{{}^{\prime}j}_{C}\mathbf{z}^ {j}_{C},~{}~{}\text{and}~{}~{}\overline{\mathbf{z}_{M}}=\sum_{j=1}^{N}\mu^{{}^ {\prime}j}_{M}\mathbf{z}^{j}_{M}.\]

We find this strategy brings a slight improvement compared to directly average \(\{\mathbf{z}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{z}^{j}_{M}\}_{j=1}^{N}\).

### Identifying Functional Activations

We identify the activations that steer the usage of contextual and parametric knowledge by calculating the mutual information between activation \(Z_{i}\) and the generated answer \(Y=\{C,M\}\):

\[I(Z_{i};Y)=\sum_{z_{i}\in Z_{i}}\sum_{y\in\{C,M\}}P(z_{i},y)\log\frac{P(z_{i} ,y)}{P(z_{i})P(y)}.\]

Since a higher \(I(Z_{i};Y)\)6 indicates a higher dependency between \(Z_{i}\) and the knowledge selection behaviour, we sorted \(\{I(Z_{i};Y)\}_{i=1}^{n}\) in descending order, and consider the top \(k\) activation work as functional activations. To decide the number of selected activations \(k\), we introduce a hyperparameter \(K\). We then choose \(k\) such that:

Footnote 6: We estimate mutual information between continuous variables \(Z_{i}\) and discrete labels \(Y\) using https://scikit-lea rn.org/1.5/modules/generated/sklearn.feature_sel ection.mutual_info_classif.html

\[k=\operatorname*{arg\,min}_{k}\sum_{i=1}^{k}\frac{I(Z_{i};Y)}{\sum_{j=1}^{n} I(Z_{j};Y)}\geq K, \tag{5}\]

which means selecting the top \(k\) SAE activations with the proportion \(K\%\) in the sum of all mutual information \(\sum_{j=1}^{n}I(Z_{j};Y)\). One potential improvement is normalising mutual information based on entropy, which has shown better properties for comparing the importance of features. We determine the top activations in each layer individually rather than ranking all mutual information across layers.

The proportion \(K\) abstracts the exact number of activations to select. We expect the same types of SAEs, e.g., pre-trained by TopK Gao et al. (2024) or JumpReLU Rajamanoharan et al. (2024), can share similar values of \(K\) for controlling. We present the relation between \(k\) and \(K\) in Appendix E. We also present the selected Gemma2-9B SAEs activations used by SpARE in Appendix D.

### Impact of the Size of the Collected Activations

We collect the hidden states \(\{\mathbf{h}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{h}^{j}_{M}\}_{j=1}^{N}\) to calculate the value of functional activations \(\mathbf{z}_{C}\) and \(\mathbf{z}_{M}\), and we also use \(\{\mathbf{z}^{j}_{C}\}_{j=1}^{N}\) and \(\{\mathbf{z}^{j}_{M}\}_{j=1}^{N}\) to estimate the mutual information. Here, we analyse the impact of \(N\) on the controlling performance.

As shown in Fig. 9(a), the performance increases when we use \(8\) examples to \(128\) for calculating the mutual information, while collecting more activations brings slight improvement. In Fig. 9(b), the performance does not increase until using \(64\) examples to calculate \(\mathbf{z}_{C}\) and \(\mathbf{z}_{M}\). The above analysis indicates that SpARE needs at least \(128\) activations to achieve a high controlling capability.

### Development Set and Demonstrations

We held out a demonstration set consisting of \(128\) instances from each dataset for each model. Each 

question in the demonstration set can be answered correctly by the corresponding model without providing evidence; more specifically, we test it with the close-book setting with few-shot examples to align the answer format. We use the non-conflict context \(E_{M}\) and memorised answer \(M\) in demonstrations. Since the contextual and parametric knowledge are consistent in the demonstration set, they do not provide information about how to resolve the knowledge conflict for the test examples when presented with conflict evidence \(E_{C}\). We sample \(5\) different sets of demonstrations using \(5\) different seeds and present the average results with deviations. All methods we compared use the same sets of demonstrations for each model.

The held-out demonstration set is also used as the development set to search hyperparameters for each model in each dataset. We sample a set of demonstrations to align the answer format and use the rest of the instances to evaluate the performance.

### Implementation Details of Representation Engineering Baselines

For all representation engineering baselines TaskVec Hendel et al. (2023), ActAdd Turner et al. (2023a) and SEA Qiu et al. (2024), we experimented with performing the intervention at different layers and reported the best performance in each case. We present the implementation details as follows.

TaskVecHendel et al. (2023): We first collect the task vectors using a few-shot setup where the few-shot examples contain the conflict evidence \(E_{C}\) along with the memorised answer \(M\). We then follow the procedure illustrated in the original paper and extract the Task Vectors as the hidden representation of the last token (: in our case) at all layers. More specifically, we use the hidden representation of samples that generate answers following the context to create \(\overline{\mathbf{h}_{C}}\), while we use those that follow the parametric knowledge and ignore the context to create \(\overline{\mathbf{h}_{M}}\). At inference time, we replace the residual stream at layer \(L\) with \(\overline{\mathbf{h}_{C}}\) or \(\overline{\mathbf{h}_{M}}\) to steer the model to follow the context or the parametric knowledge, respectively.

ActAddTurner et al. (2023a): We collect the activations for ActAdd following the same procedure. For the inference-time intervention, we edit the residual stream by adding \(\overline{\mathbf{h}_{C}}\) and subtracting \(\overline{\mathbf{h}_{M}}\) to steer the model towards context knowledge, while we perform the opposite to steer the model towards parametric knowledge.

SeaQiu et al. (2024): SEA adopts a different strategy and uses positive, negative and neutral model generations to compute steering vectors. We consider as neutral the generations that result from a few-shot setup where the demonstrations contain the memorised evidence \(E_{M}\) and memorised answer \(M\), irrespective of the generated output. We then collect activations \(\overline{\mathbf{h}_{M}}\) and \(\overline{\mathbf{h}_{C}}\) following the same procedure illustrated above and use the method described in Qiu et al. (2024) to compute the steering vectors, assuming \(\overline{\mathbf{h}_{M}}\) and \(\overline{\mathbf{h}_{C}}\) encode the positive (follow parametric knowledge) and negative (follow context knowledge) behaviours respectively.

### Searching Hyperparameters

We search hyperparameters of all methods using the same set of development sets. We choose hyper-parameters based on the EM\({}_{C\to M}\) and EM\({}_{M\to C}\), which measure the capability of changing the behaviours of LLMs.

DoLaChuang et al. (2024): We search the best coefficient \(\alpha\) that is used to compare premature and mature logits, evaluating with the "higher-layer" and "lower-layer" settings:

\[\log P(y)=\log P_{\text{mature}}(y)-\alpha\log P_{\text{premature}}(y).\]

We test the \(\alpha\) ranging from \(-10\) to \(10\) with an interval of \(0.5\). Finally, we set \(\alpha=6.0\) and \(\alpha=-8.0\) to steer the model to use contextual and parametric knowledge for Llama2-7B and Llama3-8B; set \(\alpha=1.0\) and \(\alpha=-1.0\) for Gemma2-9B. However, based on our experiments, though DoLa has a certain ability to change the behaviours of Gemma2-9B, we do not find a suitable \(\alpha\) on both "high-layer"

Figure 10: The impact of the number of the collected hidden states \(N\) on the controlling performance.

 

and "low-layer" choices for improving Gemma2-9B on the overall performance measured by EM\({}_{M}\) and EM\({}_{C}\).

**CAD**[15]: We search the best coefficient \(\alpha\) that is used to combine the logits with context and the logits without the context:

\[\log P(y)=(1+\alpha)\log P(y~{}|~{}c,x)-\alpha\log P(y~{}|~{}x).\]

We test the \(\alpha\) ranging from \(-1.5\) to \(1.5\) with an interval of \(0.1\). Finally, we set \(\alpha=0.6\) and \(\alpha=-0.8\) to steer the model to use contextual and parametric knowledge for Llama2-7B and Llama3-8B; set \(\alpha=0.3\) and \(\alpha=-0.3\) for Gamma-2-9B.

Spare(Ours): Since no previous work used SAEs to control the knowledge selection behaviour of LLMs, we need first to identify suitable magnitudes of the hyperparameters and then search them in a smaller range in the development set. We fix the \(\alpha=1\) in Eq. (4) and test the proportion \(K\) to select the activations ranging from \(0.1\) to \(0.01\) with an interval of \(0.01\). Then, we choose \(K=0.07\) and test \(\alpha\) ranging from \(1.0\) to \(3.0\) with an interval of \(0.2\). Finally, we select \(K=0.07\) and \(\alpha=2\) for Llama3-8B, \(K=0.06\) and \(\alpha=2.2\) for Llama2-7B. In our main experiments, SpARE edits the 13th to the 16th layers of Llama3-8B and the 12th to the 15th layers of Llama2-7B. We do not try other choices because there is no public SAE for Llama2-7B.

Due to the Gemma2-7B's SAEs [10, 11] using different training strategies and activation functions, they show a much more sparse pattern and have different suitable hyperparameters. Here, we select \(K=0.01\) and \(\alpha=3\) to steer contextual knowledge, and \(K=0.01\) and \(\alpha=1.8\) to steer parametric knowledge. In our main experiments, SpARE edits 23, 24, 25, 29, 30 and 31 layers of Gemma2-9B.

We also present the selected top \(k\) activations in Appendix D for further analysis.

## Appendix D Selected SAEs Activations of Gemma2-9B

In Table 2, we present the selected SAE activations used by SpARE for steering the knowledge selection behaviour. We can further interpret them in GemmaScope7[10].

Footnote 7: [https://www.neuronpedia.org/gemma-scope](https://www.neuronpedia.org/gemma-scope)

## Appendix E Distribution of Mutual Information

In Appendix C.2, we mentioned that we use the proportion mutual information (\(K\)) to determine how many activations of the SAE (\(k\)) to select. Figs. 11 to 13 shows the layer-wise accumulated mutual information (x-axis) for the number of selected activations (y-axis) across different models (Gemma2-9B, Llama2-7B, Llama3-8B). In all three models, we observed that the graph is skewed when \(k\) (y-axis) is small, indicating that some SAE activations have relatively high mutual information. While there were some differences in this tendency between models (particularly pronounced in Gemma2-9B), we found only a little variation across selected layers within the same model. This analysis corresponds to the \(K\) values (from 0.01 for Gemma2-9B to 0.07 for Llama3-8B) that we identified through hyperparameter search in Appendix C.6.

 

activations and Self-Attention activations in Fig. 16 and Fig. 17. However, we do not observe a distinct distribution difference like hidden states.

### L1 Norm and L2 Norm Pattern

As we observe the distinct skewness pattern in hidden states, we further analyse their L1 norm and L2 norm patterns in Fig. 18 and Fig. 19 However, we do not observe the distinct norm differences between \({\cal D}_{C}\) and \({\cal D}_{M}\), though they have a significantly different skewness pattern.

 

Figure 14: Skewness of the hidden states of Llama2-7B on NQSwap.

Figure 13: Proportion of accumulated mutual Information (\(K\)) on Llama3-8B

Figure 15: Skewness of the hidden states of Llama3-8B on NQSwap.

 

Figure 16: Skewness of the MLP activation of Llama2-7B on NQSwap.

Figure 19: L1 norm and L2 norm of the hidden states of Llama2-7B on NQSwap.

Figure 18: L1 norm and L2 norm of the hidden states of Llama3-8B on NQSwap.