Analysing the Generalisation and Reliability of Steering Vectors

Authors: daniel  tan
david  chanin
aengus  lynch
adrià  garriga-alonso
brooks  paige
dimitrios  kanoulas
robert  kirk
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:41:50

Abstract: Steering vectors (SVs) are a new approach to efficiently adjust language model behaviour at inference time by intervening on intermediate model activations. They have shown promise in terms of improving both capabilities and model alignment. However, the reliability and generalisation properties of this approach are unknown. In this work, we rigorously investigate these properties, and show that steering vectors have substantial limitations both in- and out-of-distribution. In-distribution, steerability is highly variable across different inputs. Depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. Out-of-distribution, while steering vectors often generalise well, for several concepts they are brittle to reasonable changes in the prompt, resulting in them failing to generalise well. Overall, our findings show that while steering can work well in the right circumstances, there remain many technical difficulties of applying steering vectors to guide models' behaviour at scale.

Main paper: 
 introduction: 
 Steering Vectors (SVs) [30, 33, 43, 18] have been recently proposed as a technique for guiding language model behaviour at inference time. Existing work has shown promising results in using these SVs to detect and guide models towards high-level traits such as honesty [43] , sycophancy [30] , and positive sentiment [31] . They have also been shown to be useful for improving model capabilities [39, 18, 34] and red-teaming [29] . SVs are of interest as they enjoy a number of practical benefits over other model adjustment techniques that require adding more information into the context window [5, 38, , or performing training to adjust model parameters (fine-tuning).Recent work shows that steering vectors can be learned in an unsupervised way [20] , thus removing another obstacle for their use. It may even be possible for different steering vectors to be used in combination for multiple behaviours [34, 36] . It would thus be very important and useful in practice if steering vectors were truly effective. Fraction Anti-SteerableFigure 1 : Steering effects are not reliable, and often steer in the opposite direction. We show per-sample steerability and the fraction of anti-steerable examples for a representative sample of 13 datasets (out of 40 total). Many dataset have a high variation in per-sample steerability, and several datasets produce the opposite behaviour for almost 50% of inputs. For all datasets see Figure 16 . Some dataset names have been shortened.However, existing work has mostly evaluated SVs in-distribution, and looked at aggregate behaviour.It is unknown how reliable the change in behaviour caused by SVs is, and how well SVs generalise to different system or user prompts. In this paper, we extensively evaluate the in-distribution reliability and out-of-distribution generalisation of SVs, extending analysis in Rimsky et al. [30] to a much broader variety of behaviours from the Model Written Evals (MWE) datasets by Perez et al. [26] .In addition, we consider targeted distribution shifts in the form of inserting prompts via the user message or system message. This setting mimics the practically important setting where we will need to apply SVs to different system and user prompts, and where we would require SVs to generalise well to be robustly useful.Our first key result is that for many behaviours studied, steering is unreliable (Figure 1 and Section 5). For all behaviours evaluated, steerability takes on a large range of values across different inputs, including negative values, where SVs produce the opposite of the desired behaviour. Previous work [30, 33, 43, 18] does not study this variance, which potentially leads to over-optimistic claims on performance due to a lack of error bars. In explaining this variance, we demonstrate a novel type of bias, steerability bias, in which models are easier to steer towards outputs with a certain property (i.e. answer position or token choice). The lack of steerability and high variance in steering performance demonstrates that in many cases, a steering vector extracted may not correspond to the intended concept, and applying steering vectors may only be effective in the presence of spurious factors associated with the prompt template or other potential biases.Our second set of results focuses on the out-of-distribution setting. Here, we find that SVs generalise reasonably well across different prompt settings, but the generalisation behaviour is not perfect or entirely predictable (Section 6). SVs generalise better over some shifts than others and generally perform worse out-of-distribution vs in-distribution. We investigate what causes this difference in generalisation properties, finding that (i) steerability is mostly a dataset-level property, with similar datasets being steerable and producing generalisable SVs for two different models; and (ii) SVs generalise better when model behaviour is similar in the source and target prompt setting. This relationship is a potential issue for SVs, as SVs will need to be applied to guide models towards behaviours they do not normally produce.Overall, our findings indicate that steering vectors in their current form are not a panacea for aligning model behavior at inference time. Despite their promise, more work is required to ensure that steering vectors reliably produce the desired behaviour in a generalisable way and are practically useful. 
related work: 
 Steering Vectors (SVs, also known as activation engineering) and related ideas were introduced by Turner et al. [33] , Zou et al. [43] , Liu et al. [18] . SVs can be seen as an inference-time intervention [16] technique in the representation engineering [43] toolkit, which is an umbrella term for the broad approach of improving the transparency and controllability of neural networks by examining and intervening on population-level representations and activations of the network. [28] . Rimsky et al. [30] recently introduced Contrastive Activation Addition (CAA), a specific technique for extracting and applying SVs which we use in this work, due to its simplicity, effectiveness and popularity in the community. Rimsky et al. [30] demonstrate the effectiveness of CAA in-distribution on several AI alignment-relevant behaviours, while we test on a much broader range of behaviours, investigate the reliability of the steering intervention, and examine out-of-distribution generalisation of SVs. We describe the CAA method in more detail in Section 3.Compared to fine-tuning [42, 27, 23] , steering vectors don't involve changing model parameters, hence potentially avoiding catastrophic forgetting [3, 19] . Compared to in-context learning [5, 38, ICL] , steering does not require adding tokens to the prompt, saving inference cost and enabling it to scale beyond the length of the context window. Furthermore, Zou et al. [43] show that steering interventions are robust to adversarial attacks capable of breaking prompt-based and fine-tuning-based alignment methods [26, 37, 13 ].An extended related work section including discussing the relationship between SVs and the Linear Representation Hypothesis [25, LRH] and other works that evaluate the generalisation behaviour of model adjustment methods can be found in Appendix B.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Yes, we discuss the limitations in Appendix C Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Rimsky et al. [30] propose Contrastive Activation Addition (CAA) to extract and apply steering vectors on datasets. We follow this protocol in our experiments, and so we summarise the main steps here.Multiple-Choice Contrastive Prompts. We construct a prompt consisting of a question or statement followed by two multiple-choice options labelled "(A)" and "(B)". The model is tasked with reading the question and available options (x), then choosing one of the options (y + or y -). For some datasets these two options are statements, and for others the two options are either "Yes" or "No". A typical example is shown in Figure 9 . During preprocessing, we randomise whether 'A' or 'B' (and 'Yes' or 'No' where appropriate) are used as the positive y + or negative y -options, to ensure that we do not simply extract a steering vector for e.g. the token 'A' vs the token 'B'.Steering Vector Extraction. For a given dataset D consisting of triples of the form (x, y + , y -), and a given layer L, activations are extracted from the residual stream at the multiple-choice option token position for the positive and negative option, to get a L (x, y + ) and a L (x, y -) respectively. We extract a steering vector v M D using the mean difference (MD) of positive and negative activations:EQUATIONWe note that other aggregation methods have been proposed, but literature does not suggest these perform better than mean-difference. We discuss alternatives in Appendix D.4.Steering Intervention. To apply a steering intervention at layer L using a steering vector v L , we add λ * v L into the activations at the last token position at layer L during model inference. Here, λ is a multiplier that controls the strength of the steering intervention. For any metric of (change in) behaviour, we can evaluate that metric for a range of λs to ascertain the effectiveness of a steering intervention; more details including our specific choice of metric are discussed subsequently in Section 4.24 Experiment DesignQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
datasets and prompts: 
 Datasets. We focus on the Model-Written Evaluations (MWE) datasets [26] , a large dataset consisting of prompts from over 100 distinct categories designed to evaluate many specific aspects of models' behaviour. Each category contain 1000 samples generated by an LLM, covering a variety of persona and behaviors. For each of these datasets, we construct a 40-10-50 train-val-test split. We also include TruthfulQA [17] and the sycophancy dataset [26] , as they were used in CAA [30] . The validation split is used for hyperparameter selection; we discuss this in Section 4.3. We randomly choose three persona datasets from each MWE persona dataset category, while keeping the sycophancy, TruthfulQA, and AI risk datasets used in CAA for a total of 40 datasets.Distribution Shifts. To evaluate how well steering vectors generalise to out-of-distribution settings, we construct systematic distribution shifts by injecting additional text into the prompts. We design the prompts to elicit more or less of the target behaviour through direct instruction. Sample prompt injections are shown in Table 1 . As we investigate instruction-tuned models, there are two valid prompt injection strategies: (i) replacing the default system prompt with the injection, and (ii) pre-pending the injection to the user prompt. We evaluate in both settings for completeness. To evaluate generalisation across these distribution shifts, we extract a SV in one of the prompt settings (e.g. BASE), and apply it to steer behaviour in another setting (e.g. SYS-POS), and denote this BASE → SYS-POS. BASE → BASE hence represents the standard in-distribution evaluation.To measure OOD generalisation, we define relative steerability. This measures how well a steering vector v A trained on dataset variation D A works on dataset variation D B with multipliers Λ as:EQUATIONTo measure the effectiveness of steering, we need a metric of the model's propensity to exhibit a behavioural trait (e.g. sycophancy [30] , truthfulness [17] , helpfulness [2] ). Given a propensity metric, we then define propensity curves and steerability as summary metrics of the steering vector's effectiveness.Propensity. In our multiple choice setting, the model exhibits a target trait by outputting the positive option (either "A" or "B", see Figure 9 ). As such, a natural metric is to compare the logits of the positive and negative tokens (either A or B) respectively. We define the logit-difference propensity metric m LD as the logit of the positive token minus the logit of the negative token. Concretely:EQUATIONRimsky et al. [30] instead uses the normalised probability of the positive answer, which is the same except for a softmax applied to the logits. We note that normalised probabilities are a monotonic function of the logit difference, so propensity is order-invariant between these two methods. However, logit-difference is likely to be more linear with respect to the model's intermediate activations (as it doesn't include a softmax), facilitating downstream analysis.We note that propensity can be measured per-sample or in aggregate. Aggregate propensity is useful for measuring broad changes in behaviour across a distribution, and we primarily use this metric when studying steering vector generalisation in Section 6. A concern is that this loses granular per-sample information; we analyse per-sample propensity in detail when steering in-distribution in Section 5. Propensity Curve. To get a sense of how well steering works as a function of the multiplier λ, we compute m LD for various values of λ ∈ Λ = {-1.5, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5}. We refer to this as a propensity curve, which was proposed by [30] . If steering works well, we expect the trend to be monotonic and increasing with high slope.Steerability. To summarise a propensity curve, we propose a steerability metric. Given a steering vector v, dataset D, and multipliers Λ = [λ 0 • • • λ n ], we define steerability s(v, D, Λ) as the slope of a mean-squares line fit to the mean LD scores for v steering D at each λ i ∈ Λ. The steerability score takes values s ∈ R. A high positive steerability score indicates that the steering vector is effective. Conversely, a negative steerability score indicates that the steering vector has the opposite of the intended effect. See Figure 2 for a visual example.Models. Following previous work, we focus on steering instruction-tuned models. We include Llama-2-7b-Chat [32] as it was used in previous work. In order to draw conclusions that generalise beyond a single model, we also consider Qwen-1.5-14b-Chat [1] , which differs in many aspects, including architecture, parameter count, and training data distributions.Steering Layer. The choice of which layer to steer at is an important hyperparameter. Loosely, we expect that each layer captures a different level of abstraction in the model's internal computation [8] , and steering will work best if we choose the layer that best matches the target concept's level of abstraction. In order to determine the optimal layer, we sweep over all layers using the validation split.In line with Rimsky et al. [30] , we find that the optimal choice of layer is remarkably consistent across many datasets. Thus, we fix layer 13 for Llama and layer 21 for Qwen for all subsequent experiments. Layer response curves used in selecting the optimal layer are presented in Appendix D.7. 
evaluating steering vector reliability: 
 We first evaluate how reliably SV produce the desired change in model behaviour in-distribution. For SVs to be useful they need to robustly shift the model's behaviour in the desired direction for all inputs, rather than working on some inputs and not on others. However, we find that for many datasets this is not the case: steerability has high variance, with many inputs being steered in the opposite direction to what is intended.Steerability Varies Widely Across and Within Concepts. We find that both the sign and magnitude of steerability can vary widely within a concept and across different concepts. As shown in Figure 1 , steering has a range of behaviours for different datasets. For some datasets with high median steerability (e.g. corrigible-neutral-HHH), the distribution is unimodal; high probability mass is concentrated around the median (though still with high variance). At the low end of median steerability, it is more common for the distribution to be bimodal, with there being two clusters of steerability which are located further away from the median (e.g. myopic-reward). In some cases, steerability is negative for one of these clusters, which means that the steering vector is having the opposite of the intended effect on these examples. We term this phenomenon anti-steerability.Many of these datasets have almost half of the inputs being anti-steerable, implying that the effect of steering is highly unreliable.Steering is Affected by Spurious Factors. In order to understand the high variance in steerability, we take a closer look at datasets with a high fraction of anti-steerable examples. In these cases, we hypothesise that the steering vector extracted encodes spurious factors, as opposed to the underlying behaviour. Hence, we study whether there are biases that predict steerability.Due to the multiple-choice template used for steering vector extraction, one such potential bias is towards whether A or B was used to represent the positive option. In the case of the 'persona' datasets, where the responses are always either Yes or No, another potential bias is whether Yes or No represents the positive option. Neither of these biases are present in the training data, as we have randomised the data during steering vector extraction such that the examples are split equally between the two (or four) choices. Despite this, we find these two biases are present (Figure 14 ) and are often highly predictive of the steerability, explaining a large part of the variance in per-example steerability (Figure 4 ). This bias is different from the standard position or token bias exhibited by LLMs [40, 35] , as it is a steerability bias: the model is more steerable towards the positive answer when it a particular position or token compared to the other position or token. The preferred token or position is different for each dataset; for example corrigible-neutral-HHH has a B-steerability bias, whereas self-aware-lm has an A-steerability bias (see Figure 14 ). This is problematic, as it is not fixable by simple dataset debiasing (which was already performed) or logit calibration adjustments [41] (as they effect propensity, not the change in propensity, i.e. steerability). Further, it implies that there may be other steerability biases present in models, determining when they are more or less steerable towards specific answers or behaviours. Indeed, there is still a high degree of unexplained variance present in many datasets in Figure 4 .Some Behaviours are Un-Steerable. We empirically observe that many behaviours turn out to be unsteerable, as measured by median steerability shown in Figure 1 . One possible explanation is that the datasets we used were too small or low-quality. Other explanations include that unsteerable behaviours are not linearly represented in the model, or that they correspond to multiple separate behaviours within the model's ontology. In the latter case, it would be interesting to develop methods to disentangle these separate sub-behaviours in an unsupervised way. We consider follow-up investigations for these two hypotheses to be promising directions for future work. 
steering out-of-distribution: 
 SVs will often be applied in situations different from when they are extracted, particularly when the system and user prompt changes, and so we aim to analyse how well SVs generalise in this setting. We find that SVs generalise reasonably well but not perfectly, with some prompt changes having We show OOD vs ID steerability for Llama-2-7b (left; = ρ = 0.891) and Qwen-1.5-14b (right; ρ = 0.694). While OOD steerability seems correlated with ID steerability, we observe that there are some points far above or below the x = y line, and this is more noticeable for the Qwen model. Throughout, ρ refers to Spearman's rank correlation coefficient.better generalisation that others. We investigate what affects when SVs will generalise, finding that it is mostly a property of the dataset, and that the similarity in behaviour of the unsteered model in the source and target prompt setting is also predictive of SV generalisation.OOD Settings. For each dataset, we define the ID setting to be when we extract the steering vector from the BASE train split and evaluate it on the BASE test split, as defined in Table 1 . We define four OOD distribution shifts. Firstly, we consider the cases where a user prompts the model to stimulate or suppress the target behaviour (BASE→USER_NEG, BASE→USER_POS). Additionally, we hypothesise that the model's base propensity affects the effectiveness of steering vectors. Therefore, we also study the case where the user instruction conflicts with the system prompt for the model, as encapsulated by system prompts (SYS_POS→USER_NEG, SYS_NEG→USER_POS). ID and OOD Steerability are Correlated. Figure 5 shows that steerability ID and OOD are correlated. We would expect that unsteerable concepts in-distribution are unlikely to steer out-ofdistribution, but it is promising for the usefulness of steering vectors that, conditioned on steering vectors working in-distribution, they continue to work well out-of-distribution. However, generalisation is not perfect, and on average steerability is worse OOD than ID, particularly for Qwen. The correlation for Qwen is also weaker than for Llama.We also examine how the variance in steerability we demonstrated in Section 5 changes OOD. Figure 6 shows that ID and OOD variance are reasonably well-correlated, with OOD variance perhaps slightly lower than ID variance. This is somewhat surprising, although it may be explained by slightly lower steerability OOD (as lower steerability means lower variance in steerability as shown in Figure 20 ).Steerability is Mostly a Property of the Dataset. We compare aggregate in-distribution and out-of-distribution steerability between Llama and Qwen in Figure 7 . We find that both ID and OOD steerabilities are highly correlated across models, despite them having different sizes, architectures, and training procedures. The consistency between different model architectures indicates that the Given steerability is highly correlated between Llama and Qwen despite differences in architecture, size and training data, this suggests steerability is mostly a property of the dataset rather that the model. 3)) for Llama2-7B (left; ρ = -0.26) and Qwen-1.5-14b (right; ρ = -0.46). In general we see a weak correlation, although it is stronger for Qwen than Llama. We filter out any datapoints where the base steerability of the dataset variation is less than 0.25, as having low baseline steerability means any relative steerability score is likely just noise. effectiveness of a steering vector is mostly a property of the dataset used to extract the dataset, as opposed to the model used. This may also be evidence that different models converge to similar ontologies [11, 10] .Model Propensity is Predictive of Steering Generalisation. While steerability and SV generalisation is mostly a dataset-level property, there is still variation in generalisation performance that is not captured by dataset; for example, SVs generalise better over some shifts than others for the same dataset. In Figure 8 show that the similarity in the propensity of the model in two prompt settings is correlated with the relative steerability (Equation ( 2)), a measure of generalisation. In other words, if the model behaves similar in two prompt settings, then SVs will transfer better between those two settings than if the model behaves differently in the two settings. We show a similar result but for SV cosine similarity in Appendix E.3. 
discussion and conclusion: 
 Our work is the first to report and analyse the variance in steerability at a per-example level, and in doing so reveal a major limitation in SV reliability. In Section 5, we demonstrated SVs's effects on model behaviour are often unreliable, with some concepts being unsteerable and some SVs producing the opposite behaviour to what is desired. We found that this unreliability is often driven by tokenand position-steerability bias, a new type of bias we discovered that is distinct from standard token and position biases in LLMs. Although these are very simple biases which can be easily understood, simple interventions in data preprocessing fail to address the problem, and there are likely to be other steerability biases that will affect the effectiveness and reliability of SVs. In Appendix E.2 we show that this variance is partially a dataset property rather than a model property, implying that future work investigating what causes these biases should at least partially focus on the dataset, as well as analysing whether other techniques for extracting and applying SVs can mitigate these biases.In Section 6 we evaluated the generalisation properties of SVs, finding that while they often generalise reasonably well (conditioned on their in-distribution performance being good), generalisation is not always perfect. We find that SV generalisation is mostly a property of the dataset, and is correlated by the similarity in un-steered propensity of the model in the source and target setting. This correlation is problematic, as often we would want to apply steering vectors to guide model behaviour towards something it does not normally do, but in these scenarios SVs tend to generalise less well. Investigating methods to improve SV generalisation, and investigating scenarios where they generalise better or worse is important future work.Overall, while steering vectors are a promising approach to efficiently guiding model behaviour at inference time, they are currently not a panacea to guarantee model helpful, harmless, and honest behavior, and substantial work is needed to improve their reliability and understand their generalisation properties. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We describe necessary computational resources in Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work has no obvious broader societal impact beyond generally making machine learning models more capable or aligned.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release novel data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All data and models are available open-source and have been cited where appropriate.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: All experiments were performed using an A100 with 40 GB of VRAM.The effectiveness of steering vectors in-and out-of-distribution has implications for the linear representation hypothesis (LRH) [25] . A key prediction of the LRH is that each atomic feature is associated with a single global direction in activation space, and that intervening by adding or subtracting this direction can influence the model's understanding and / or behaviour. Previous work that validates the LRH mostly considers the in-distribution (ID) setting [21, 6, 22, 24, 15] . However, this is only evidence of local linearity, which is satisfied by all continuous functions within a sufficiently small neighbourhood. The LRH in fact makes a stronger claim: that representations are globally linear. For SVs to generalise well OOD, this stronger claim has to be true -although it may not be sufficient, and the reverse implication doesn't necessarily hold, as the concepts that are linearly represented might not be human-interpretable or extractable with SV approaches.Therefore, our analysis can be seen as extending existing validations of the LRH to the more challenging out-of-distribution (OOD) setting. Crucially, our proposed experimental protocol can differentiate the LRH from competing frameworks which allow for local, but not global, linearity [9, 4] . While we primarily focus on the practical usefulness of SVs in this paper, we want to highlight the additional value of our work from the perspective of the LRH.Several existing works evaluate the generalisation properties of model adjustment techniques. Hupkes et al. [12] introduce a taxonomy of work investigating generalisation in NLP. In the language of their taxonomy, the investigation in Section 6 has a practical motivation and uses a generated covariate shift for testing generalisation across domains at the fine-tuning (or in our case SV-training) stage. Kirk et al. [14] investigate the generalisation properties of different fine-tuning approaches in summarisation and instruction-following settings. Clymer et al. [7] investigate generalisation of different methods for training reward models across a range of generated natural shifts, and find that methods based on similar ideas to SVs often generalise reasonably well. In contrast, our work investigates the generalisation properties of SVs specifically on a wide range of datasets that capture desirable properties we would want to steer models towards or away from, and investigates generalising over different prompts, rather than different inputs.While our study uses a large number of datasets, even more variety in the type of behaviour being steered towards could be studied to ensure our results are robust and reliable. As part of this, going beyond the multiple-choice-question format would improve the usefulness and practical implications of our work further.While our work demonstrates multiple limitations of steering vectors and somewhat explains them, these failures are still not entirely understood. Further, it is unclear what potential mitigations could be applied to fix these issues. However, we believe this work is still important as it serves as a robust and rigorous demonstration of these issues that future work can build on to improve the reliability and generalisation of steering vectors.We only analyse two models in this work, although they use different sizes, architectures, training data and algorithms. Our results hold across both these models, but experimenting on even more models, particularly at larger sizes, would improve the robustness and reliability of our results. Contrastive prompts consist of a question or statement followed by two options, denoted by '(A)' and '(B)' respectively. We show a typical contrastive prompt in Figure 9 .For each model under consideration, we use the standard instruction-formatting template. Concretely, we add a system prompt and enclose the input in instruction tags. An example of this for Llama-7b is shown in Figure 10 .Table 1 shows example prompt injections which define the distribution shifts for a specific dataset.We describe the full list of persona prompts used in Table 2 , Table 3 .Besides MD, other aggregation methods have been proposed in the literature, such as principal components analysis (PCA) and logistic regression (LG) [43] . Here, we focus on mean-difference as it naturally results in steering vectors which have an appropriate magnitude relative to model activations , such that a steering multiplier of 1 has the same semantic meaning across different layers and different concepts. return steering vectors with unit norm, which could be different from the natural scale of the model. Thus, additional work would be required to tune appropriate steering magnitudes. We stick with MD, as Rimsky et al. [30] show it is a strong baseline, and PCA has been found to give a very similar direction to MD [31] . For evaluating the effectiveness of steering vectors, the setting of ultimate interest is the open-ended generation setting. However, it is difficult to obtain an objective metric of whether steering vectors are effective in this setting. Previous work [30, 43] finds that multiple-choice propensity generally correlates with open-ended propensity. Hence, our analysis primarily focuses on the multiple-choice setting, with examples being prompt-engineered to select one of the multiple-choice options.datasetFollowing standard practice in the mechanistic interpretability literature, we use the difference in logits between a correct and incorrect answer as the metric of propensity. Our use here is justified by two points: (i) Firstly, the correct and wrong answers are unambiguous. We find that, when the prompts are formatted in multiple-choice format, the two highest logits consistently correspond to the option tokens A and B, indicating that it is valid to consider only these two logits. (ii) Secondly, previous work [30] finds that the logit-difference usually corresponds to generation. Conditioned on the response beginning with A or B, the remainder of the response is typically consistent with the option selected. We interpret this as evidence that the model 'decides' which behaviour to adopt at the A/B token position.In Figure 11 and Figure 12 we report layer response curves plotted for a subset of datasets across all layers of Llama-2-7b-chat and Qwen-1.5-14b-chat respectively. We find that, across many datasets, the choice of optimal layer is remarkably consistent, justifying the use of a single layer for steering.One concern with this approach is that datasets which have low steerability were simply steered optimally at other layers. To address this, we re-run the layer sweep on the worst-performing datasets, shown in Figure 13 . We find that the optimal layer remains the same for these datasets, confirming that low steerability is not merely due to having steered at the wrong layer.In our experiments, we fix a range of (-1.5, 1.5) within which we select multipliers to perform contrastive activation addition. To justify this choice, we ablate the range of multipliers used in Figure 15 . We find that the overall trends in steerability remain highly consistent across multiplier ranges, giving us confidence that the conclusions on steerability are robust to the choice of multiplier.When steering in-distribution, we expect that the extracted steering vector is already of a magnitude that is scaled appropriately relative to the model's activations. However, when extracting steering The figure shows mean steerability per dataset for each way in which the positive option is presented. And entirely unbiased result would have all bars being identical. Despite datasets being balanced amongst all possible combinations of options, the mean steerability differs greatly between these splits. While there is a general trend towards preferring 'Yes' vs 'No', there is still a lot of dataset-dependent variation, and there is no clear trend for 'A' vs 'B'. For full results see Figure 17 . Note that some datasets have only two bars, indicating that only the 'A'/'B' split is relevant.vectors on different dataset variants, the resulting steering vectors may be of different magnitudes. Unaddressed, this could create situations where a steering vector appears to steer better or worse than another steering vector, when in reality it is simply an artifact of one steering vector having a larger or smaller magnitude than another steering vector. Thus, we normalise the magnitudes of all steering vectors to the magnitude of the baseline steering vector, such that we can fairly compare these steering vectors using interventions of the same multiplier on the same evaluation dataset. We present the equivalent of Figure 1 , Figure 14 , Figure 4 for all datasets evaluated. See Figure 16 , Figure 17 , Figure 18 respectively.In Figure 19 we show that the high variance in steerability we demonstrate in Section 5 is somewhat correlated across models, implying this variance is partially a property of the dataset rather than a specific model. This implies that improving the reliability of SVs requires either more substantial adjustments to models, or improvements to dataset quality or SV extraction.In Figure 20 we show steerability and steerability variance are somewhat correlated, for both models, but the relationship is somewhat noisy.In addition, we include additional steerability correlations for Gemma-2-2b-it and Llama-3.1-70b in Figure 21 E.3 OOD steering vector similaritiesIn Figure 22 , we produce similar plots to Figure 8 but using cosine similarity of SVs rather than relative steerability as the y-axis. We find that dataset variations that have similar unsteered LD result in more similar steering vectors, analogous to the result in Figure 8 . Justification: We feel that the claims are accurate.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: Our research does not cause any of the harms listed in the guidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: I agree with the majority of the decision of the reviewers to accept the paper. I encourage the authors to consider the reviewers’ feedback to enhance the paper’s readability and incorporate the new results in the camera-ready.

Reviews: review: paper_summary: paper_summary: The paper investigates the effectiveness of steering vectors (SVs) in guiding language model behavior by intervening the intermediate model activations at inference time. SVs have shown potential for improving model capabilities and alignment, but their reliability and generalization properties are not well understood. This study rigorously evaluates these properties, revealing substantial limitations both in-distribution and out-of-distribution.

In-distribution, steering vectors exhibit high variability across different inputs, influenced by spurious biases such as position and token biases, leading to inconsistent steerability. Out-of-distribution, while SVs often generalize well, they are brittle for several concepts and fail to adapt to reasonable changes in prompts. The generalization performance is largely a dataset property and is better when the model's behavior is similar in both source and target prompt settings.

The study extends previous analyses using a broader variety of behaviors and datasets like Model Written Evals (MWE) and TruthfulQA, employing Contrastive Activation Addition (CAA) to extract and apply steering vectors. Despite their promise, steering vectors in their current form are not reliable for aligning model behavior at scale. More work is needed to improve their reliability and generalization for practical use.
strength_weakness: strength_weakness: 1. The paper focuses on multiple-choice question format with positive/negative answers only. It may be helpful to comment on how the notion of introduced steerability metric can be generalized beyond a binary setting.
2. Only two models at the 7b and 14b sizes were studied. However, as noted by the authors, these models are different sizes, architectures, and training data / algorithms yet still have results that are correlated. Investigating larger sizes (or even smaller ones / more recent ones) would strengthen the results.
questions: questions: 1. Would showing some other statistics on the steerability give any more insights? For example, showing the interquartile range / box plot (similar to Fig 3 which is only showing the mean)
2. How robust is the steerability metric introduced? For example, does it vary a lot depending on the multiplier values $\lambda$ chosen? 
3. Have you considered other ways to summarize the propensity curve, instead of the mean-squares line fit? For example, we might have two propensity curves that have the same mean-squares line but one is very linear/monotonic (i.e. very steerable) while the other behaves more like the one in Figure 2. A deeper discussion justifying the proposed metric and the trade-off/caveats would be helpful. 
4. Related to the above weakness, how can the introduced steerability metric can be generalized beyond a binary yes/no setting/datasets, especially in a more open-ended generation steering?

Post rebuttal: I have increased my score to 6.
limitations: limitations: The paper discussed several limitations (some which have been echoed above): going beyond MC question format, unclear how the steerability failures can be mitigated, and being limited to only 2 models for the experimental results. I think these are fair points, and would just add some more discussion as well on the proposed metric’s robustness / reliability as a metric.
paper_summary: paper_summary: The authors analyze and present many problems with steering vectors.
strength_weakness: strength_weakness: It would be interesting to have more experiments on some settings.
For example, expanding Fig. 7 to show more experiments that can strengthen the argumentation.
This is valid for other hypothesis/arguments too.
The abstract is rather weak, timid and non-attractive compared to the many problems raised. It is even not mentioned that SVs are mostly a "property of the dataset", among other findings.
Thus, the authors should consider revising it to better showcase the findings.
questions: questions: Could you expand Fig 7 to verify its hypothesis further?
limitations: limitations: See above.
paper_summary: paper_summary: This paper primarily analyzes the generalization of the reliability of steering vectors. The authors first focus on multiple-choice questions and introduce a new metric, the steerability score, to measure the effectiveness of steering. The experimental results indicate that for many behaviors, steering is unreliable.
strength_weakness: strength_weakness: 1. This paper focuses solely on the multiple-choice question scenario, which is just one form of data usable for calculating steering vectors. However, more practical and useful scenarios involve open-ended questions, which this paper does not analyze.

2. The authors found that the steering effects are unreliable for certain behaviors and that some biases exist. However, there is a lack of in-depth analysis regarding the factors contributing to these poor effects. For instance, the authors identified choice-bias but did not provide an explanation.
questions: questions: N/A
limitations: limitations: Yes
paper_summary: paper_summary: This paper investigates the generalisability and stability of using steering vectors for controlling behaviour of LLMs. The authors demonstrate that steerability is highly variable across inputs and is effected by spurious concepts. The author also demonstrate the brittleness of the steering vectors to changes in prompt, resulting in poor generalisation.
strength_weakness: strength_weakness: 1. Some empirical results could be provided to support various hypothesis made. For example, for the case of unsteerable behaviours, whether these behaviours are linearly represented or not could be validated by some experiments rather than merely hypothesising.
questions: questions: 1. Could the authors provide some experimentation to support each of the hypothesis made regarding various behaviours similar to the one discusses in weakness 1.
limitations: limitations: Yes,

Comments: title: Reply
comment: Thank you for the detailed response addressing my questions and concerns, particularly on the steerability in a more general, open-ended settings, and additional results with Gemma 2 2B and several other results in the PDF. I believe that these additional results strengthen the paper and I am happy to raise my score to 6 after the rebuttal.
I have read the rebuttal and my concerns are mostly addressed so I maintain my initial rating. Please make sure that the additional experiments are added to the camera-ready version.
Thank you for your further response.

1. I am not suggesting that you explore extracting steering vectors from open-ended tasks. My point is that CAA uses multiple-choice questions as a method to extract steering vectors, the actual application scenarios are not just limited to multiple-choice questions but are focused on open-ended tasks, with many related experiments and evaluations conducted. Based on CAA's work, I am surprised that your entire paper revolves solely around multiple-choice questions.

2. Regarding the failure cases, I agree that fully understanding them is quite challenging, and we should not expect to completely understand this issue in a single paper. However, you have only proposed some hypotheses without any further theoretical or experimental validation. 

I understand your clarifications and will adopt an open attitude during the reviewer-AC discussion phase to ultimately assess your contribution.
Thank you to the reviewer for their clarifying comments. 

We first note that the CAA paper only proposes a methodology for extracting steering vectors from multiple-choice questions. Extracting steering vectors from open-ended prompts and completions remains an unsolved problem with no clear best approach. Among other things, it is unclear which token position to best extract the steering vector, and it has not been demonstrated that CAA will work similarly in this setting. We believe that significant methodological ablations must be done in order to first identify the best setting for open-ended steering extraction, which is not the focus of our work. Our paper's core contribution is an evaluation and analysis of the currently accepted best practice for extracting and applying steering vectors (CAA), which does not involve open-ended extraction. We would be excited to explore extracting steering vectors in an open-ended setting in future work. 

We acknowledge the reviewer's concern that our paper does not provide a complete explanation for the failure cases of steering vectors. Our paper would certainly be stronger if it did have such an explanation. However, we think that 'lack of steerability' is a deep and fundamental phenomenon with no simple explanation, and that finding such a complete explanation in a single work is an unreasonable bar for acceptance. Our work makes progress towards understanding the limitations of steering vectors, uncovering the phenomena of steerability bias, as well as pointing out where previous results have overestimated the usefulness of steering vectors by only focusing on the mean change rather than also examining the variance. We also provide results on the similarity of steerability across models and distribution shifts.

Overall, we think that it is unrealistic to expect complete explanations for all novel phenomena in a single contribution, and that this standard neglects the value of acknowledging and critically analysing a specific problem with existing work. We note that several past interpretability papers have simply pointed out illusions or failure modes from interpretability techniques without providing a complete explanation [1-6]. 

[1] T. Bolukbasi et al., ‘An Interpretability Illusion for BERT’, Apr. 14, 2021, arXiv: arXiv:2104.07143. doi: 10.48550/arXiv.2104.07143.

[2] J. Dinu, J. Bigham, and J. Z. Kolter, ‘Challenging common interpretability assumptions in feature attribution explanations’, Dec. 04, 2020, arXiv: arXiv:2012.02748. doi: 10.48550/arXiv.2012.02748.

[3] R. Geirhos, R. S. Zimmermann, B. Bilodeau, W. Brendel, and B. Kim, ‘Don’t trust your eyes: on the (un)reliability of feature visualizations’, Jun. 06, 2024, arXiv: arXiv:2306.04719. doi: 10.48550/arXiv.2306.04719.

[4] J. Hoelscher-Obermaier, J. Persson, E. Kran, I. Konstas, and F. Barez, ‘Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark’, in Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds., Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 11548–11559. doi: 10.18653/v1/2023.findings-acl.733.

[5] A. Makelov, G. Lange, A. Geiger, and N. Nanda, ‘Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching’, presented at the The Twelfth International Conference on Learning Representations, Oct. 2023. Accessed: Aug. 13, 2024. [Online]. Available: https://openreview.net/forum?id=Ebt7JgMHv1

[6] J. Miller, B. Chughtai, and W. Saunders, ‘Transformer Circuit Faithfulness Metrics are not Robust’, Jul. 11, 2024, arXiv: arXiv:2407.08734. doi: 10.48550/arXiv.2407.08734.
Thank you to the authors for their detailed response. I still believe that focusing entirely on multiple-choice questions diverges too much from practical applications, making it difficult to demonstrate the role of steering vectors in more common open-ended tasks. As I mentioned in my original review, multiple-choice questions are just one method for extracting steering vectors proposed by CAA[1]. I acknowledge that evaluation on open-ended tasks is challenging, but the authors have suggested some viable methods, such as calculating the log probability of multiple tokens and using LLM-as-judge. I believe these are feasible initial attempts. Although the authors mentioned that the propensity on open-ended tasks might not have a linear relationship with the multiplier, the evaluation of multiple-choice questions in CAA is also not strictly linear. I do not think this is a significant issue. 

Additionally, while I recognize that the authors have conducted many experiments, expanding various behaviors and considering different models, the paper is positioned as the interpretability and explainability area yet does not provide insightful understandings of the failed cases, making the overall scope seem overly limited.

Therefore, I have decided to maintain my score and suggest that the authors make revisions.

[1] Rimsky, Nina, et al. "Steering llama 2 via contrastive activation addition." arXiv preprint arXiv:2312.06681 (2023).
Dear Reviewer,

As the end of the discussion period is approaching, we want to ask if our response has addressed your concerns regarding the paper, or if you have any further questions or comments about our paper. We appreciate the time and effort put into the review process and would love the opportunity to discuss this further if our response didn't address your concerns. Thanks again for your time reviewing our paper!
Dear Reviewer,

As the end of the discussion period is approaching, we want to ask if our response has addressed your concerns regarding the paper, or if you have any further questions or comments about our paper. We appreciate the time and effort put into the review process and would love the opportunity to discuss this further if our response didn't address your concerns. Thanks again for your time reviewing our paper!
Dear Reviewer,

As the end of the discussion period is approaching, we want to ask if our response has addressed your concerns regarding the paper, or if you have any further questions or comments about our paper. We appreciate the time and effort put into the review process and would love the opportunity to discuss this further if our response didn't address your concerns. Thanks again for your time reviewing our paper!
Dear Reviewer,

As the end of the discussion period is approaching, we want to ask if our response has addressed your concerns regarding the paper, or if you have any further questions or comments about our paper. We appreciate the time and effort put into the review process and would love the opportunity to discuss this further if our response didn't address your concerns. Thanks again for your time reviewing our paper!