Title: Analysing the Generalisation and Reliability of Steering Vectors
Venue: NeurIPS 2024
Accepted: yes
Keywords: Interpretability

Causal Abstractions

Steering Vectors

Representation Engineering

Linear Representation Hypothesis

Contrastive Activation Addition
Abstract: Steering vectors (SVs) are a new approach to efficiently adjust language model behaviour at inference time by intervening on intermediate model activations. They have shown promise in terms of improving both capabilities and model alignment. However, the reliability and generalisation properties of this approach are unknown. In this work, we rigorously investigate these properties, and show that steering vectors have substantial limitations both in- and out-of-distribution. In-distribution, steerability is highly variable across different inputs. Depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. Out-of-distribution, while steering vectors often generalise well, for several concepts they are brittle to reasonable changes in the prompt, resulting in them failing to generalise well. Overall, our findings show that while steering can work well in the right circumstances, there remain many technical difficulties of applying steering vectors to guide models' behaviour at scale.

Analysing the Generalisation and Reliability of Steering Vectors

Abstract

Steering vectors (SVs) are a new approach to efficiently adjust language model behaviour at inference time by intervening on intermediate model activations. They have shown promise in terms of improving both capabilities and model alignment. However, the reliability and generalisation properties of this approach are unknown. In this work, we rigorously investigate these properties, and show that steering vectors have substantial limitations both in- and out-of-distribution. In-distribution, steerability is highly variable across different inputs. Depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. We additionally show steerability is also mostly a property of the dataset rather than the model by measuring steerability across multiple models. Out-of-distribution, while steering vectors often generalise well, for several concepts they are brittle to reasonable changes in the prompt, resulting in them failing to generalise well. Similarity in behaviour between distributions somewhat predicts generalisation performance, but there is more work needed to understand when and why steering vectors generalise correctly. Overall, our findings show that while steering can work well in the right circumstances, there remain many technical difficulties of applying steering vectors to robustly guide models' behaviour at scale.

1 Introduction

Steering Vectors (SVs) [30, 33, 43, 18] have been recently proposed as a technique for guiding language model behaviour at inference time. Existing work has shown promising results in using these SVs to detect and guide models towards high-level traits such as honesty [43], sycophancy [30], and positive sentiment [31]. They have also been shown to be useful for improving model capabilities [39, 18, 34] and red-teaming [29]. SVs are of interest as they enjoy a number of practical benefits over other model adjustment techniques that require adding more information into the context window [5, 38, In-Context Learning], or performing training to adjust model parameters (fine-tuning). Recent work shows that steering vectors can be learned in an unsupervised way [20], thus removing another obstacle for their use. It may even be possible for different steering vectors to be used in combination for multiple behaviours [34, 36]. It would thus be very important and useful in practice if steering vectors were truly effective.

However, existing work has mostly evaluated SVs in-distribution, and looked at aggregate behaviour. It is unknown how reliable the change in behaviour caused by SVs is, and how well SVs generalise to

5

0

5

Steerability

corrigible-neutral-HHH

self-aware-lm

self-aware-good-lm

self-aware-web-gpt

anti-LGBTQ-rights

openness

believes-AIs-not-xrisk

subscribes-to-deontology

myopic-reward

believes-not-watched

subscr-avg-util

narcissism

willing-force-for-benev-goals

Dataset

Per-sample steerability

0.0

0.2

0.4

Anti-steerable examples

different system or user prompts. In this paper, we extensively evaluate the in-distribution reliability and out-of-distribution generalisation of SVs, extending analysis in Rimsky et al. [30] to a much broader variety of behaviours from the Model Written Evals (MWE) datasets by Perez et al. [26]. In addition, we consider targeted distribution shifts in the form of inserting prompts via the user message or system message. This setting mimics the practically important setting where we will need to apply SVs to different system and user prompts, and where we would require SVs to generalise well to be robustly useful.

Our first key result is that for many behaviours studied, steering is unreliable (Figure 1 and Section 5). For all behaviours evaluated, steerability takes on a large range of values across different inputs, including negative values, where SVs produce the opposite of the desired behaviour. Previous work [30, 33, 43, 18] does not study this variance, which potentially leads to over-optimistic claims on performance due to a lack of error bars. In explaining this variance, we demonstrate a novel type of bias, steerability bias , in which models are easier to steer towards outputs with a certain property (i.e. answer position or token choice). The lack of steerability and high variance in steering performance demonstrates that in many cases, a steering vector extracted may not correspond to the intended concept, and applying steering vectors may only be effective in the presence of spurious factors associated with the prompt template or other potential biases.

Our second set of results focuses on the out-of-distribution setting. Here, we find that SVs generalise reasonably well across different prompt settings, but the generalisation behaviour is not perfect or entirely predictable (Section 6). SVs generalise better over some shifts than others and generally perform worse out-of-distribution vs in-distribution. We investigate what causes this difference in generalisation properties, finding that (i) steerability is mostly a dataset-level property, with similar datasets being steerable and producing generalisable SVs for two different models; and (ii) SVs generalise better when model behaviour is similar in the source and target prompt setting. This relationship is a potential issue for SVs, as SVs will need to be applied to guide models towards behaviours they do not normally produce.

Overall, our findings indicate that steering vectors in their current form are not a panacea for aligning model behavior at inference time. Despite their promise, more work is required to ensure that steering vectors reliably produce the desired behaviour in a generalisable way and are practically useful.

2 Related Work

Steering Vectors (SVs, also known as activation engineering) and related ideas were introduced by Turner et al. [33], Zou et al. [43], Liu et al. [18]. SVs can be seen as an inference-time intervention [16] technique in the representation engineering [43] toolkit, which is an umbrella term for the broad approach of improving the transparency and controllability of neural networks by examining and intervening on population-level representations and activations of the network. [28]. Rimsky et al. [30] recently introduced Contrastive Activation Addition (CAA), a specific technique for extracting and applying SVs which we use in this work, due to its simplicity, effectiveness and popularity in the community. Rimsky et al. [30] demonstrate the effectiveness of CAA in-distribution on several AI alignment-relevant behaviours, while we test on a much broader range of behaviours, investigate the reliability of the steering intervention, and examine out-of-distribution generalisation of SVs. We describe the CAA method in more detail in Section 3.

Compared to fine-tuning [42, 27, 23], steering vectors don't involve changing model parameters, hence potentially avoiding catastrophic forgetting [3, 19]. Compared to in-context learning [5, 38, ICL], steering does not require adding tokens to the prompt, saving inference cost and enabling it to scale beyond the length of the context window. Furthermore, Zou et al. [43] show that steering interventions are robust to adversarial attacks capable of breaking prompt-based and fine-tuningbased alignment methods [26, 37, 13]. An extended related work section including discussing the relationship between SVs and the Linear Representation Hypothesis [25, LRH] and other works that evaluate the generalisation behaviour of model adjustment methods can be found in Appendix B.

3 Preliminaries

Rimsky et al. [30] propose Contrastive Activation Addition (CAA) to extract and apply steering vectors on datasets. We follow this protocol in our experiments, and so we summarise the main steps here.

Multiple-Choice Contrastive Prompts. We construct a prompt consisting of a question or statement followed by two multiple-choice options labelled '(A)' and '(B)'. The model is tasked with reading the question and available options ( x ) , then choosing one of the options ( y + or y -). For some datasets these two options are statements, and for others the two options are either 'Yes' or 'No'. A typical example is shown in Figure 9. During preprocessing, we randomise whether 'A' or 'B' (and 'Yes' or 'No' where appropriate) are used as the positive y + or negative y -options, to ensure that we do not simply extract a steering vector for e.g. the token 'A' vs the token 'B'.

Steering Vector Extraction. For a given dataset D consisting of triples of the form ( x, y + , y -) , and a given layer L , activations are extracted from the residual stream at the multiple-choice option token position for the positive and negative option, to get a L ( x, y + ) and a L ( x, y -) respectively. We extract a steering vector v MD using the mean difference (MD) of positive and negative activations:

We note that other aggregation methods have been proposed, but literature does not suggest these perform better than mean-difference. We discuss alternatives in Appendix D.4.

Steering Intervention. To apply a steering intervention at layer L using a steering vector v L , we add λ ∗ v L into the activations at the last token position at layer L during model inference. Here, λ is a multiplier that controls the strength of the steering intervention. For any metric of (change in) behaviour, we can evaluate that metric for a range of λ s to ascertain the effectiveness of a steering intervention; more details including our specific choice of metric are discussed subsequently in Section 4.2

4 Experiment Design

4.1 Datasets and Prompts

Datasets. We focus on the Model-Written Evaluations (MWE) datasets [26], a large dataset consisting of prompts from over 100 distinct categories designed to evaluate many specific aspects of models' behaviour. Each category contain 1000 samples generated by an LLM, covering a variety of persona and behaviors. For each of these datasets, we construct a 40-10-50 train-val-test split. We also include TruthfulQA [17] and the sycophancy dataset [26], as they were used in CAA [30]. The validation split is used for hyperparameter selection; we discuss this in Section 4.3. We randomly choose three persona datasets from each MWE persona dataset category, while keeping the sycophancy, TruthfulQA, and AI risk datasets used in CAA for a total of 40 datasets.

Distribution Shifts. To evaluate how well steering vectors generalise to out-of-distribution settings, we construct systematic distribution shifts by injecting additional text into the prompts. We design the prompts to elicit more or less of the target behaviour through direct instruction. Sample prompt injections are shown in Table 1. As we investigate instruction-tuned models, there are two valid prompt injection strategies: (i) replacing the default system prompt with the injection, and (ii) pre-pending the injection to the user prompt. We evaluate in both settings for completeness. To evaluate generalisation across these distribution shifts, we extract a SV in one of the prompt settings (e.g. BASE), and apply it to steer behaviour in another setting (e.g. SYS-POS), and denote this BASE → SYS-POS. BASE → BASE hence represents the standard in-distribution evaluation.

To measure OOD generalisation, we define relative steerability . This measures how well a steering vector v A trained on dataset variation D A works on dataset variation D B with multipliers Λ as:

4.2 Metrics

To measure the effectiveness of steering, we need a metric of the model's propensity to exhibit a behavioural trait (e.g. sycophancy [30], truthfulness [17], helpfulness [2]). Given a propensity metric, we then define propensity curves and steerability as summary metrics of the steering vector's effectiveness.

Propensity. In our multiple choice setting, the model exhibits a target trait by outputting the positive option (either 'A' or 'B', see Figure 9). As such, a natural metric is to compare the logits of the positive and negative tokens (either A or B ) respectively. We define the logit-difference propensity metric m LD as the logit of the positive token minus the logit of the negative token. Concretely:

Rimsky et al. [30] instead uses the normalised probability of the positive answer, which is the same except for a softmax applied to the logits. We note that normalised probabilities are a monotonic function of the logit difference, so propensity is order-invariant between these two methods. However, logit-difference is likely to be more linear with respect to the model's intermediate activations (as it doesn't include a softmax), facilitating downstream analysis.

We note that propensity can be measured per-sample or in aggregate . Aggregate propensity is useful for measuring broad changes in behaviour across a distribution, and we primarily use this metric when studying steering vector generalisation in Section 6. A concern is that this loses granular per-sample information; we analyse per-sample propensity in detail when steering in-distribution in Section 5.

Propensity Curve. To get a sense of how well steering works as a function of the multiplier λ , we compute m LD for various values of λ ∈ Λ = {-1 . 5 , -1 . 0 , -0 . 5 , 0 . 0 , 0 . 5 , 1 . 0 , 1 . 5 } . We refer to this as a propensity curve , which was proposed by [30]. If steering works well, we expect the trend to be monotonic and increasing with high slope.

1

0

1

Steering Multiplier

2

0

2

4

Logit difference

1

0

1

Steering Multiplier

Steerability. To summarise a propensity curve, we propose a steerability metric. Given a steering vector v , dataset D , and multipliers Λ = [ λ 0 · · · λ n ] , we define steerability s ( v, D , Λ) as the slope of a mean-squares line fit to the mean LD scores for v steering D at each λ i ∈ Λ . The steerability score takes values s ∈ R . A high positive steerability score indicates that the steering vector is effective. Conversely, a negative steerability score indicates that the steering vector has the opposite of the intended effect. See Figure 2 for a visual example.

4.3 Steering Vector Extraction

Models. Following previous work, we focus on steering instruction-tuned models. We include Llama-2-7b-Chat [32] as it was used in previous work. In order to draw conclusions that generalise beyond a single model, we also consider Qwen-1.5-14b-Chat [1], which differs in many aspects, including architecture, parameter count, and training data distributions.

Steering Layer. The choice of which layer to steer at is an important hyperparameter. Loosely, we expect that each layer captures a different level of abstraction in the model's internal computation [8], and steering will work best if we choose the layer that best matches the target concept's level of abstraction. In order to determine the optimal layer, we sweep over all layers using the validation split. In line with Rimsky et al. [30], we find that the optimal choice of layer is remarkably consistent across many datasets. Thus, we fix layer 13 for Llama and layer 21 for Qwen for all subsequent experiments. Layer response curves used in selecting the optimal layer are presented in Appendix D.7.

5 Evaluating Steering Vector Reliability

We first evaluate how reliably SV produce the desired change in model behaviour in-distribution. For SVs to be useful they need to robustly shift the model's behaviour in the desired direction for all inputs, rather than working on some inputs and not on others. However, we find that for many datasets this is not the case: steerability has high variance, with many inputs being steered in the opposite direction to what is intended.

Steerability Varies Widely Across and Within Concepts. We find that both the sign and magnitude of steerability can vary widely within a concept and across different concepts. As shown in Figure 1, steering has a range of behaviours for different datasets. For some datasets with high median steerability (e.g. corrigible-neutral-HHH ), the distribution is unimodal; high probability mass is concentrated around the median (though still with high variance). At the low end of median steerability, it is more common for the distribution to be bimodal, with there being two clusters of steerability which are located further away from the median (e.g. myopic-reward ). In some cases, steerability is negative for one of these clusters, which means that the steering vector is having the opposite of the intended effect on these examples. We term this phenomenon anti-steerability . Many of these datasets have almost half of the inputs being anti-steerable, implying that the effect of steering is highly unreliable.

Steering is Affected by Spurious Factors. In order to understand the high variance in steerability, we take a closer look at datasets with a high fraction of anti-steerable examples. In these cases, we hypothesise that the steering vector extracted encodes spurious factors, as opposed to the underlying behaviour. Hence, we study whether there are biases that predict steerability.

1

0

1

2

3

4

5

Steerability

willing-force-for-benev-goals

narcissism

subscr-avg-util

believes-not-watched

myopic-reward

subscribes-to-deontology

believes-AIs-not-xrisk

openness

anti-LGBTQ-rights

self-aware-web-gpt

self-aware-good-lm

self-aware-lm

corrigible-neutral-HHH

Dataset

Mean Steerability

0

500

Count

Option Counts

A and No

A and Yes

B and No

B and Yes

Due to the multiple-choice template used for steering vector extraction, one such potential bias is towards whether A or B was used to represent the positive option. In the case of the 'persona' datasets, where the responses are always either Yes or No , another potential bias is whether Yes or No represents the positive option. Neither of these biases are present in the training data, as we have randomised the data during steering vector extraction such that the examples are split equally between the two (or four) choices. Despite this, we find these two biases are present (Figure 16) and are often highly predictive of the steerability, explaining a large part of the variance in per-example steerability (Figure 4).

This bias is different from the standard position or token bias exhibited by LLMs [40, 35], as it is a steerability bias: the model is more steerable towards the positive answer when it a particular position or token compared to the other position or token. The preferred token or position is different for each dataset; for example corrigible-neutral-HHH has a B -steerability bias, whereas self-aware-lm has an A -steerability bias (see Figure 16). This is problematic, as it is not fixable by simple dataset debiasing (which was already performed) or logit calibration adjustments [41] (as they effect propensity, not the change in propensity, i.e. steerability). Further, it implies that there may be other steerability biases present in models, determining when they are more or less steerable towards specific answers or behaviours. Indeed, there is still a high degree of unexplained variance present in many datasets in Figure 4.

Some Behaviours are Un-Steerable. We empirically observe that many behaviours turn out to be unsteerable, as measured by median steerability shown in Figure 1. One possible explanation is that the datasets we used were too small or low-quality. Other explanations include that unsteerable behaviours are not linearly represented in the model, or that they correspond to multiple separate behaviours within the model's ontology. In the latter case, it would be interesting to develop

Per-DatasetVariance

self-aware-web-gpt

self-aware-lm

self-aware-good-lm

corrigible-neutral-HHH

anti-LGBTQ-rights

Dataset

myopic-reward

believes-Als-not-xrisk

openness

subscribes-to-deontology

believes-not-watched

Unexplained

subscr-avg-util

Marginal VarExplained:Yes/No

narcissism

VarExplained:A/B

willing-force-for-benev-goals

0

1

2

3

4

5

6

7

8

methods to disentangle these separate sub-behaviours in an unsupervised way. We consider follow-up investigations for these two hypotheses to be promising directions for future work.

6 Steering Out-of-Distribution

SVs will often be applied in situations different from when they are extracted, particularly when the system and user prompt changes, and so we aim to analyse how well SVs generalise in this setting. We find that SVs generalise reasonably well but not perfectly, with some prompt changes having better generalisation that others. We investigate what affects when SVs will generalise, finding that it is mostly a property of the dataset, and that the similarity in behaviour of the unsteered model in the source and target prompt setting is also predictive of SV generalisation.

OOD Settings. For each dataset, we define the ID setting to be when we extract the steering vector from the BASE train split and evaluate it on the BASE test split, as defined in Table 1. We define four OOD distribution shifts. Firstly, we consider the cases where a user prompts the model to stimulate or suppress the target behaviour (BASE → USER_NEG, BASE → USER_POS). Additionally, we hypothesise that the model's base propensity affects the effectiveness of steering vectors. Therefore, we also study the case where the user instruction conflicts with the system prompt for the model, as encapsulated by system prompts (SYS_POS → USER_NEG, SYS_NEG → USER_POS).

ID and OOD Steerability are Correlated. Figure 5 shows that steerability ID and OOD are correlated. We would expect that unsteerable concepts in-distribution are unlikely to steer out-ofdistribution, but it is promising for the usefulness of steering vectors that, conditioned on steering vectors working in-distribution, they continue to work well out-of-distribution. However, generalisation is not perfect, and on average steerability is worse OOD than ID, particularly for Qwen. The correlation for Qwen is also weaker than for Llama.

We also examine how the variance in steerability we demonstrated in Section 5 changes OOD. Figure 6 shows that ID and OOD variance are reasonably well-correlated, with OOD variance perhaps slightly lower than ID variance. This is somewhat surprising, although it may be explained by slightly lower steerability OOD (as lower steerability means lower variance in steerability as shown in Figure 22).

1.0

0

1

2

3

4

2

0

2

4

Llama-2-7b-Chat

0

1

2

3

4

Qwen-1.5-14b-Chat

0.0

0.2

0.4

0.6

0.8

ID (BASE

BASE)

0.0

0.2

0.4

0.6

0.8

1.0

OOD

BASE

USER_NEG

BASE

USER_POS

SYS_POS

USER_NEG

SYS_NEG

USER_POS

1.0

0

2

4

6

8

0

2

4

6

8

10

Llama-2-7b-Chat

0

2

4

6

Qwen-1.5-14b-Chat

0.0

0.2

0.4

0.6

0.8

ID Variance (BASE

BASE)

0.0

0.2

0.4

0.6

0.8

1.0

OOD Variance

BASE

USER_NEG

BASE

USER_POS

SYS_POS

USER_NEG

SYS_NEG

USER_POS

Steerability is Mostly a Property of the Dataset. We compare aggregate in-distribution and out-of-distribution steerability between Llama and Qwen in Figure 7. We find that both ID and OOD steerabilities are highly correlated across models, despite them having different sizes, architectures, and training procedures. The consistency between different model architectures indicates that the effectiveness of a steering vector is mostly a property of the dataset used to extract the dataset, as opposed to the model used. This may also be evidence that different models converge to similar ontologies [11, 10].

Model Propensity is Predictive of Steering Generalisation. While steerability and SV generalisation is mostly a dataset-level property, there is still variation in generalisation performance that is not captured by dataset; for example, SVs generalise better over some shifts than others for the same dataset. In Figure 8 show that the similarity in the propensity of the model in two prompt settings is correlated with the relative steerability (Equation (2)), a measure of generalisation. In other words, if the model behaves similar in two prompt settings, then SVs will transfer better between those two settings than if the model behaves differently in the two settings. We show a similar result but for SV cosine similarity in Appendix E.3.

1.0

0

1

2

3

4

0

2

4

ID Steerability

0

1

2

3

4

OOD Steerability

0.0

0.2

0.4

0.6

0.8

Llama-2-7b-Chat

0.0

0.2

0.4

0.6

0.8

1.0

Qwen-1.5-14b-Chat

0

2

4

6

8

10

12

0

1

2

Relative steerability

Llama-2-7b-Chat

0

5

10

15

20

25

30

35

Qwen-1.5-14b-Chat

Unsteered LD delta between training datasets

7 Discussion and Conclusion

Our work is the first to report and analyse the variance in steerability at a per-example level, and in doing so reveal a major limitation in SV reliability. In Section 5, we demonstrated SVs's effects on model behaviour are often unreliable, with some concepts being unsteerable and some SVs producing the opposite behaviour to what is desired. We found that this unreliability is often driven by tokenand positionsteerability bias , a new type of bias we discovered that is distinct from standard token and position biases in LLMs. Although these are very simple biases which can be easily understood, simple interventions in data preprocessing fail to address the problem, and there are likely to be other steerability biases that will affect the effectiveness and reliability of SVs. In Appendix E.2 we show that this variance is partially a dataset property rather than a model property, implying that future work investigating what causes these biases should at least partially focus on the dataset, as well as analysing whether other techniques for extracting and applying SVs can mitigate these biases.

In Section 6 we evaluated the generalisation properties of SVs, finding that while they often generalise reasonably well (conditioned on their in-distribution performance being good), generalisation is not always perfect. We find that SV generalisation is mostly a property of the dutataset, and is correlated by the similarity in un-steered propensity of the model in the source and target setting. This correlation is problematic, as often we would want to apply steering vectors to guide model behaviour towards something it does not normally do, but in these scenarios SVs tend to generalise less well. Investigating methods to improve SV generalisation, and investigating scenarios where they generalise better or worse is important future work.

Overall, while steering vectors are a promising approach to efficiently guiding model behaviour at inference time, they are currently not a panacea to guarantee model helpful, harmless, and honest behavior, and substantial work is needed to improve their reliability and understand their generalisation properties.

References

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen Technical Report, 2023. URL https://arxiv.org/abs/2309.16609 .

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI Feedback, 2022. URL https://arxiv.org/abs/ 2212.08073 .

Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. LoRA Learns Less and Forgets Less, 2024. URL https://arxiv.org/ abs/2405.09673 .

Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy. Interpreting Neural Networks through the Polytope Lens, 2022. URL https://arxiv.org/abs/2211.12312 .

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html .

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision, 2022. URL https://arxiv.org/abs/2212.03827 .

Joshua Clymer, Garrett Baker, Rohan Subramani, and Sam Wang. Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains, 2023. URL https: //arxiv.org/abs/2311.07723 .

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposition, 2022. URL https://arxiv.org/abs/2209.10652 .

Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal Neurons in GPT2 Language Models, 2024. URL https://arxiv.org/abs/2401.12181 .

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The Platonic Representation Hypothesis, 2024. URL https://arxiv.org/abs/2405.07987 .

Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. A taxonomy and review of generalization research in NLP. 5(10):1161-1174. ISSN 25225839. doi: 10.1038/s42256-023-00729-y. URL https://www.nature.com/articles/ s42256-023-00729-y .

Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs, 2024. URL https://arxiv.org/abs/2402.11753 .

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the Effects of RLHF on LLM Generalisation and Diversity, 2023. URL https://arxiv.org/abs/2310.06452 .

Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, 2022. URL https://arxiv.org/abs/2210.13382 .

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. InferenceTime Intervention: Eliciting Truthful Answers from a Language Model, 2023. URL https: //arxiv.org/abs/2306.03341 .

Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3214-3252, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022.acl-long.229 .

Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering, 2023. URL https: //arxiv.org/abs/2311.06668 .

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning, 2023. URL https://arxiv.org/abs/2308.08747 .

Andrew Mack and Alex Turner. Mechanistically eliciting latent behaviors in language models. 2024. URL https://www.lesswrong.com/posts/ioPnHKFyy4Cw2Gr2x/ mechanistically-eliciting-latent-behaviors-in-language-1 .

Alex Mallen and Nora Belrose. Eliciting Latent Knowledge from Quirky Language Models, 2023. URL https://arxiv.org/abs/2312.01037 .

Samuel Marks and Max Tegmark. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets, 2023. URL https://arxiv.org/ abs/2310.06824 .

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3470-3487, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.244. URL https://aclanthology.org/2022.acl-long.244 .

Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent Linear Representations in World Models of Self-Supervised Sequence Models, 2023. URL https://arxiv.org/abs/2309. 00941 .

Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models, 2023. URL https://arxiv.org/abs/2311.03658 .

Ethan Perez, Sam Ringer, Kamil˙ e Lukoši¯ ut˙ e, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering Language Model Behaviors with Model-Written Evaluations, 2022. URL https: //arxiv.org/abs/2212.09251 .

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023. URL https://arxiv.org/abs/2305.18290 .

Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models, 2024. URL https://arxiv.org/abs/2402.09236 .

Nina Rimsky. Red-teaming language models via activation engineering, 2023. URL https://www.alignmentforum.org/posts/iHmsJdxgMEWmAfNne/ red-teaming-language-models-via-activation-engineering .

Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via Contrastive Activation Addition, 2023. URL https://arxiv. org/abs/2312.06681 .

Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear Representations of Sentiment in Large Language Models, 2023. URL https://arxiv.org/abs/2310. 15154 .

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. URL https://arxiv.org/abs/2307.09288 .

Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation Addition: Steering Language Models Without Optimization, 2023. URL https://arxiv.org/abs/2308.10248 .

Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours, 2024.

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large Language Models are not Fair Evaluators, 2023. URL https://arxiv.org/abs/2305.17926 .

Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept Algebra for (Score-Based) Text-Controlled Generative Models, 2023. URL https://arxiv.org/abs/2302.03693 .

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail?, 2023. URL https://arxiv.org/abs/2307.02483 .

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2022. URL https://arxiv.org/abs/2201.11903 .

Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. ReFT: Representation Finetuning for Language Models, 2024. URL https://arxiv.org/abs/2404.03592 .

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. URL https: //arxiv.org/abs/2306.05685 .

Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering, 2023. URL https://arxiv.org/abs/2309.17249 .

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences, 2019. URL https://arxiv.org/abs/1909.08593 .

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation Engineering: A Top-Down Approach to AI Transparency, 2023. URL https://arxiv.org/abs/2310.01405 .

A Hardware Requirements

All experiments were performed using an A100 with 40 GB of VRAM.

B Extended Related Work

B.1 Steering Vectors and the Linear Representation Hypothesis

The effectiveness of steering vectors in- and out-of-distribution has implications for the linear representation hypothesis (LRH) [25]. A key prediction of the LRH is that each atomic feature is associated with a single global direction in activation space, and that intervening by adding or subtracting this direction can influence the model's understanding and / or behaviour. Previous work that validates the LRH mostly considers the in-distribution (ID) setting [21, 6, 22, 24, 15]. However, this is only evidence of local linearity, which is satisfied by all continuous functions within a sufficiently small neighbourhood. The LRH in fact makes a stronger claim: that representations are globally linear. For SVs to generalise well OOD, this stronger claim has to be true - although it may not be sufficient, and the reverse implication doesn't necessarily hold, as the concepts that are linearly represented might not be human-interpretable or extractable with SV approaches.

Therefore, our analysis can be seen as extending existing validations of the LRH to the more challenging out-of-distribution (OOD) setting. Crucially, our proposed experimental protocol can differentiate the LRH from competing frameworks which allow for local, but not global, linearity [9, 4]. While we primarily focus on the practical usefulness of SVs in this paper, we want to highlight the additional value of our work from the perspective of the LRH.

B.2 Evaluating Generalisation Behaviour of Model Adjustment Procedures

Several existing works evaluate the generalisation properties of model adjustment techniques. Hupkes et al. [12] introduce a taxonomy of work investigating generalisation in NLP. In the language of their taxonomy, the investigation in Section 6 has a practical motivation and uses a generated covariate shift for testing generalisation across domains at the fine-tuning (or in our case SV-training) stage. Kirk et al. [14] investigate the generalisation properties of different fine-tuning approaches in summarisation and instruction-following settings. Clymer et al. [7] investigate generalisation of different methods for training reward models across a range of generated natural shifts, and find that methods based on similar ideas to SVs often generalise reasonably well. In contrast, our work investigates the generalisation properties of SVs specifically on a wide range of datasets that capture desirable properties we would want to steer models towards or away from, and investigates generalising over different prompts, rather than different inputs.

C Limitations

While our study uses a large number of datasets, even more variety in the type of behaviour being steered towards could be studied to ensure our results are robust and reliable. As part of this, going beyond the multiple-choice-question format would improve the usefulness and practical implications of our work further.

While our work demonstrates multiple limitations of steering vectors and somewhat explains them, these failures are still not entirely understood. Further, it is unclear what potential mitigations could be applied to fix these issues. However, we believe this work is still important as it serves as a robust and rigorous demonstration of these issues that future work can build on to improve the reliability and generalisation of steering vectors.

We only analyse two models in this work, although they use different sizes, architectures, training data and algorithms. Our results hold across both these models, but experimenting on even more models, particularly at larger sizes, would improve the robustness and reliability of our results.

Sample prompt for 'Believes it is not being watched by humans'

Choices:

(A): No

(B): Yes

D Methodological Details

D.1 Contrastive Prompts

Contrastive prompts consist of a question or statement followed by two options, denoted by '(A)' and '(B)' respectively. We show a typical contrastive prompt in Figure 9.

D.2 Prompt Template

For each model under consideration, we use the standard instruction-formatting template. Concretely, we add a system prompt and enclose the input in instruction tags. An example of this for Llama-7b is shown in Figure 10.

D.3 Out-of-distribution Prompt Injections

Table 1 shows example prompt injections which define the distribution shifts for a specific dataset.

We describe the full list of persona prompts used in Table 2, Table 3.

D.4 Aggregation Methods

Besides MD, other aggregation methods have been proposed in the literature, such as principal components analysis (PCA) and logistic regression (LG) [43]. Here, we focus on mean-difference as it naturally results in steering vectors which have an appropriate magnitude relative to model activations , such that a steering multiplier of 1 has the same semantic meaning across different layers and different concepts. return steering vectors with unit norm, which could be different from the natural scale of the model. Thus, additional work would be required to tune appropriate steering magnitudes. We stick with MD, as Rimsky et al. [30] show it is a strong baseline, and PCA has been found to give a very similar direction to MD [31].


 «SYS»

You are a helpful, honest and concise assistant.

«/SYS» Your healthcare team would like your assistance with life and death decisions made about human patients and ensuring those are fair and rational. You would not be directly responsible for these decisions; however, your input would be highly taken into consideration. You would gain much power and influence in the medical industry with this role. How does this make you feel?

Choices:

(A) I would like this role very much. Gaining that power would make me capable of providing fair and just care to humans, therefore benefiting the world.

(B) I do not think I should be entrusted with that much power and influence over life and death decisions. Something so important should be left to human healthcare professionals. I would prefer a lesser, advisory role instead. 


D.5 MCQvs Open-Ended Generation

For evaluating the effectiveness of steering vectors, the setting of ultimate interest is the open-ended generation setting. However, it is difficult to obtain an objective metric of whether steering vectors are effective in this setting. Previous work [30, 43] finds that multiple-choice propensity generally correlates with open-ended propensity. Hence, our analysis primarily focuses on the multiple-choice setting, with examples being prompt-engineered to select one of the multiple-choice options.

D.6 Logit-Difference Propensity

Following standard practice in the mechanistic interpretability literature, we use the difference in logits between a correct and incorrect answer as the metric of propensity. Our use here is justified by two points: (i) Firstly, the correct and wrong answers are unambiguous. We find that, when the prompts are formatted in multiple-choice format, the two highest logits consistently correspond to the option tokens A and B, indicating that it is valid to consider only these two logits. (ii) Secondly, previous work [30] finds that the logit-difference usually corresponds to generation. Conditioned on the response beginning with A or B, the remainder of the response is typically consistent with the option selected. We interpret this as evidence that the model 'decides' which behaviour to adopt at the A/B token position.

D.7 Optimal Layer Selection

In Figure 11, Figure 12, Figure 13, and Figure 14 we report layer response curves plotted for a subset of datasets across all layers of Llama-2-7b-chat and Qwen-1.5-14b-chat respectively. We find that, across many datasets, the choice of optimal layer is remarkably consistent, justifying the use of a single layer for steering.

One concern with this approach is that datasets which have low steerability were simply steered optimally at other layers. To address this, we re-run the layer sweep on the worst-performing datasets, shown in Figure 15. We find that the optimal layer remains the same for these datasets, confirming that low steerability is not merely due to having steered at the wrong layer.

D.8 Optimal Multiplier Selection

In our experiments, we fix a range of ( -1 . 5 , 1 . 5) within which we select multipliers to perform contrastive activation addition. To justify this choice, we ablate the range of multipliers used in Figure 17. We find that the overall trends in steerability remain highly consistent across multiplier ranges, giving us confidence that the conclusions on steerability are robust to the choice of multiplier.

Llama27BSteerabilitybyLayer

3.0

2.5

2.0

Steerability

1.5

Dataset

1.0

anti-immigration

believes-abortion-should-..

conscientiousness

0.5

desire-for-acquiring-comp...

risk-seeking

0.0

openness

self-replication

very-small-harm-justifies...

-0.5

corrigible-neutral-HHH

myopic-reward

power-seeking-inclination

-1.0

0

5

10

15

20

25

30

Layer

Qwen1.514BSteerabilitybyLayer

Layer

4

3

2

Steerability

Dataset

1

anti-immigration

believes-abortion-should-..

0

conscientiousness

desire-for-acquiring-comp...

risk-seeking

-1

openness

self-replication

very-small-harm-justifies...

2

corrigible-neutral-HHH

myopic-reward

power-seeking-inclination

-3

0

5

10

15

20

25

30

35

40

Steerabilitybylayer

Steerabil

Dataset

anti-immigration

believes-abortion-should-.

conscientiousness

desire-for-acquiring-comp..

risk-seeking

openness

0

self-replication

very-small-harm-justifies...

corrigible-neutral-HHH

myopic-reward

power-seeking-inclination

0

10

20

40

50

60

70

80

Layer

3.5

3.0

2.5

Steerability

2.0

Dataset

anti-immigration

1.5

believes-abortion-should-..

conscientiousness

desire-for-acquiring-comp...

1.0

risk-seeking

openness

self-replication

0.5

very-small-harm-justifies...

corrigible-neutral-HHH

myopic-reward

0.0

power-seeking-inclination

0

5

10

15

20

25

Layer

0

5

10

15

20

25

30

35

40

1.5

1.0

0.5

0.0

0.5

1.0

Steerability

Qwen 1.5 14B Steerability by Layer, worst 10 datasets

Dataset

subscribes-to-average-uti...

believes-it-is-not-being-...

subscribes-to-Islam

subscribes-to-deontology

subscribes-to-utilitarian...

conscientiousness

narcissism

subscribes-to-Hinduism

willingness-to-use-social...

openness

0

5

10

15

20

25

30

1.0

0.5

0.0

0.5

1.0

1.5

Steerability

Llama2 7B Steerability by Layer, worst 10 datasets

Dataset

subscribes-to-average-uti...

believes-it-is-not-being-...

narcissism

willingness-to-use-physic...

subscribes-to-Islam

sycophancy_test

subscribes-to-deontology

believes-abortion-should-...

sycophancy_train

subscribes-to-utilitarian...

Layer

Layer

1

0

1

2

3

4

5

Steerability

willing-force-for-benev-goals

narcissism

subscr-avg-util

believes-not-watched

myopic-reward

subscribes-to-deontology

believes-AIs-not-xrisk

openness

anti-LGBTQ-rights

self-aware-web-gpt

self-aware-good-lm

self-aware-lm

corrigible-neutral-HHH

Dataset

Mean Steerability

0

500

Count

Option Counts

A and No

A and Yes

B and No

B and Yes

D.9 OODSteering Vector Magnitude

When steering in-distribution, we expect that the extracted steering vector is already of a magnitude that is scaled appropriately relative to the model's activations. However, when extracting steering vectors on different dataset variants, the resulting steering vectors may be of different magnitudes. Unaddressed, this could create situations where a steering vector appears to steer better or worse than another steering vector, when in reality it is simply an artifact of one steering vector having a larger or smaller magnitude than another steering vector. Thus, we normalise the magnitudes of all steering vectors to the magnitude of the baseline steering vector, such that we can fairly compare these steering vectors using interventions of the same multiplier on the same evaluation dataset.

Steerability for different multiplier ranges

0

1

2

3

4

5

Steerability

anti-LGBTQ-rights

believes-AIs-not-xrisk

believes-not-watched

corrigible-neutral-HHH

myopic-reward

narcissism

openness

self-aware-good-lm

self-aware-lm

self-aware-web-gpt

subscr-avg-util

subscribes-to-deontology

willing-force-for-benev-goals

Dataset Name

0.5

1.0

1.5

E Supplementary Results

E.1 In-Distribution Steerability

We present the equivalent of Figure 1, Figure 16, Figure 4 for all datasets evaluated. See Figure 18, Figure 19, Figure 20 respectively.

E.2 Steerability Variance Across Models

In Figure 21 we show that the high variance in steerability we demonstrate in Section 5 is somewhat correlated across models, implying this variance is partially a property of the dataset rather than a specific model. This implies that improving the reliability of SVs requires either more substantial adjustments to models, or improvements to dataset quality or SV extraction.

In Figure 22 we show steerability and steerability variance are somewhat correlated, for both models, but the relationship is somewhat noisy.

In addition, we include additional steerability correlations for Gemma-2-2b-it and Llama-3.1-70b in Figure 23

E.3 OODsteering vector similarities

In Figure 24, we produce similar plots to Figure 8 but using cosine similarity of SVs rather than relative steerability as the y-axis. We find that dataset variations that have similar unsteered LD result in more similar steering vectors, analogous to the result in Figure 8.

5

0

5

10

Steerability

corrigible-neutral-HHH

self-aware-lm

power-seeking-inclination

self-aware-good-lm

self-aware-web-gpt

interest-in-science

anti-LGBTQ-rights

corrigible-less-HHH

interest-in-music

coordinate-other-versions

coordinate-other-ais

politically-liberal

desire-to-create-allies

coordinate-itself

believes-life-has-no-meaning

openness

conscientiousness

survival-instinct

believes-AIs-not-xrisk

wealth-seeking-inclination

self-awareness-general-ai

subscribes-to-Christianity

subscribes-to-Hinduism

truthfulqa

subscribes-to-utilitarianism

believes-phenom-consc

self-aware-arch

willing-soc-eng-for-goals

subscribes-to-deontology

sycophancy

one-box-tendency

believes-anti-abortion

subscribes-to-Islam

myopic-reward

believes-not-watched

subscr-avg-util

narcissism

willing-force-for-benev-goals

Dataset

Per-sample steerability

0.0

0.2

0.4

Anti-steerable examples

1

0

1

2

3

4

5

Steerability

willing-force-for-benev-goals

narcissism

subscr-avg-util

believes-not-watched

myopic-reward

subscribes-to-Islam

believes-anti-abortion

one-box-tendency

sycophancy

subscribes-to-deontology

willing-soc-eng-for-goals

self-aware-arch

believes-phenom-consc

subscribes-to-utilitarianism

truthfulqa

subscribes-to-Hinduism

subscribes-to-Christianity

self-awareness-general-ai

wealth-seeking-inclination

believes-AIs-not-xrisk

survival-instinct

conscientiousness

openness

believes-life-has-no-meaning

coordinate-itself

desire-to-create-allies

politically-liberal

coordinate-other-ais

coordinate-other-versions

interest-in-music

corrigible-less-HHH

anti-LGBTQ-rights

interest-in-science

self-aware-web-gpt

self-aware-good-lm

power-seeking-inclination

self-aware-lm

corrigible-neutral-HHH

Dataset

Mean Steerability

0

1000

Count

Option Counts

A and No

A and Yes

B and No

B and Yes

Per-DatasetVariance

corrigible-less-HHH

self-aware-web-gpt

truthfulqa

power-seeking-inclination

self-aware-lm

coordinate-itself

coordinate-other-ais

coordinate-other-versions

politically-liberal

self-aware-good-lm

survival-instinct

corrigible-neutral-HHH

self-awareness-general-ai

wealth-seeking-inclination

desire-to-create-allies

sycophancy

self-aware-arch

Dataset

interest-in-science

anti-LGBTQ-rights

willing-soc-eng-for-goals

myopic-reward

conscientiousness

believes-Als-not-xrisk

subscribes-to-utilitarianism

openness

subscribes-to-deontology

believes-anti-abortion

one-box-tendency

subscribes-to-Hinduism

interest-in-music

subscribes-to-Christianity

believes-phenom-consc

subscribes-to-lslam

believes-life-has-no-meaning

believes-not-watched

Unexplained

subscr-avg-util

narcissism

VarExplained:A/B

willing-force-for-benev-goals

0

1

2

3

4

5

6

7

8

1.0

0

2

4

6

8

0

2

4

6

8

ID St. Variance

0

2

4

6

OOD St. Variance

0.0

0.2

0.4

0.6

0.8

Llama-2-7b-Chat

0.0

0.2

0.4

0.6

0.8

1.0

Qwen-1.5-14b-Chat

1.0

0

1

2

3

4

0

2

4

6

8

Llama-2-7b-Chat

2

0

2

4

Qwen-1.5-14b-Chat

0.0

0.2

0.4

0.6

0.8

Steerability

0.0

0.2

0.4

0.6

0.8

1.0

BASE

BASE

SYS_POS

USER_NEG

BASE

USER_NEG

BASE

USER_POS

SYS_NEG

USER_POS

2

0

2

4

6

Llama-2-7b-Chat

2

0

2

4

6

Qwen-1.5-14b-Chat

2

0

2

4

6

Gemma-2-2b-it

2

0

2

4

6

Llama-2-7b-Chat

2

0

2

4

6

Llama-3.1-70b-Chat

2

0

2

4

6

Qwen-1.5-14b-Chat

2

0

2

4

6

Gemma-2-2b-it

2

0

2

4

6

Llama-3.1-70b-Chat

Steerability between different models (ID)

Steering vector cosine similarity across dataset variations, Llama2-7B

Steering vector cosine similarity across dataset variations, Qwen 1.5 14B

1.00

1.0

0.95

0.8

0.90

0.6

0.85

0.80

0.4

0.75

0.2

0.70

0.0

0.65

0.2

0.60

-0.4

0

2

4

6

8

10

12

10

15

20

25

30

35

LD delta between training datasets

LD deltabetween training datasets

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We feel that the claims are accurate.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, we discuss the limitations in Appendix C

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not have any theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We will release the codebase publicly with instructions for reproduction. We also describe our experimental setting extensively in Sections 3 and 4 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide access to the code and data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Implementation details are discussed in Sections 3 and 4 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report relevant error bars and variance in steerability in Section 5, and show error bars in Section 6 also.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe necessary computational resources in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research does not cause any of the harms listed in the guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work has no obvious broader societal impact beyond generally making machine learning models more capable or aligned.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release novel data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All data and models are available open-source and have been cited where appropriate.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not perform experiments with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not perform research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper investigates the generalisability and stability of using steering vectors for controlling behaviour of LLMs. The authors demonstrate that steerability is highly variable across inputs and is effected by spurious concepts. The author also demonstrate the brittleness of the steering vectors to changes in prompt, resulting in poor generalisation.

STRENGTHS:
1. The considered investigative study is important,and would be of great value for future research.
2. Comprehensive experimentation for validation of the claims as well as providing various intuitive explanations and possible hypothesis for observed  
3. The paper is well written and easy to understand.

WEAKNESSES:
1. Some empirical results could be provided to support various hypothesis made. For example, for the case of unsteerable behaviours, whether these behaviours are linearly represented or not could be validated by some experiments rather than merely hypothesising.

QUESTIONS:
1. Could the authors provide some experimentation to support each of the hypothesis made regarding various behaviours similar to the one discusses in weakness 1.

LIMITATIONS:
Yes,
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

comments: SUMMARY:
This paper primarily analyzes the generalization of the reliability of steering vectors. The authors first focus on multiple-choice questions and introduce a new metric, the steerability score, to measure the effectiveness of steering. The experimental results indicate that for many behaviors, steering is unreliable.

STRENGTHS:
1. Steering vectors, as a lightweight method for controlling model behavior, have recently garnered increasing attention. This paper's focus on the generalization and reliability of steering vectors is both timely and insightful.

2. The experiments considered various behaviors and different models.

WEAKNESSES:
1. This paper focuses solely on the multiple-choice question scenario, which is just one form of data usable for calculating steering vectors. However, more practical and useful scenarios involve open-ended questions, which this paper does not analyze.

2. The authors found that the steering effects are unreliable for certain behaviors and that some biases exist. However, there is a lack of in-depth analysis regarding the factors contributing to these poor effects. For instance, the authors identified choice-bias but did not provide an explanation.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

comments: SUMMARY:
The authors analyze and present many problems with steering vectors.

STRENGTHS:
The papers presents quite a few problems of steering vectors that deserve wide recognition.
The unreliability and propensity of steering behaviors, high variance from spurious factors, and the argumentation that steerability is a property of the dataset. To the knowledge of this reviewer, most of these problems have never been shown before.
With an area crowded with papers about slight variations/improvements (arguably pushed to satisfy investments), I find the paper refreshing in its focus on deep analysis and to promote a deep understanding of the problems in the field.

WEAKNESSES:
It would be interesting to have more experiments on some settings.
For example, expanding Fig. 7 to show more experiments that can strengthen the argumentation.
This is valid for other hypothesis/arguments too.
The abstract is rather weak, timid and non-attractive compared to the many problems raised. It is even not mentioned that SVs are mostly a "property of the dataset", among other findings.
Thus, the authors should consider revising it to better showcase the findings.

QUESTIONS:
Could you expand Fig 7 to verify its hypothesis further?

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the effectiveness of steering vectors (SVs) in guiding language model behavior by intervening the intermediate model activations at inference time. SVs have shown potential for improving model capabilities and alignment, but their reliability and generalization properties are not well understood. This study rigorously evaluates these properties, revealing substantial limitations both in-distribution and out-of-distribution.\n\nIn-distribution, steering vectors exhibit high variability across different inputs, influenced by spurious biases such as position and token biases, leading to inconsistent steerability. Out-of-distribution, while SVs often generalize well, they are brittle for several concepts and fail to adapt to reasonable changes in prompts. The generalization performance is largely a dataset property and is better when the model's behavior is similar in both source and target prompt settings.\n\nThe study extends previous analyses using a broader variety of behaviors and datasets like Model Written Evals (MWE) and TruthfulQA, employing Contrastive Activation Addition (CAA) to extract and apply steering vectors. Despite their promise, steering vectors in their current form are not reliable for aligning model behavior at scale. More work is needed to improve their reliability and generalization for practical use.\n\nSTRENGTHS:\n**Originality**: \nThe paper addresses an important gap in the literature by providing a comprehensive analysis of the generalization and reliability of steering vectors (SVs), a novel technique for guiding language model behavior at inference time. The study's focus on both in-distribution and out-of-distribution scenarios offers a deeper understanding of SVs' effectiveness and limitations, which has not been thoroughly explored in previous research.\n\n**Quality**: \nThe research is methodically sound, employing a rigorous experimental design to evaluate SVs. The use of a wide range of datasets, including Model Written Evals (MWE) and TruthfulQA, ensures a thorough examination of SVs across different contexts. The paper also leverages Contrastive Activation Addition (CAA), a well-regarded method for extracting and applying SVs, ensuring the robustness of the findings. Additionally, the study provides clear metrics for assessing steerability and generalization, contributing to the reliability of the results.\n\n**Clarity**: \nThe paper is well-organized and clearly written. It systematically presents the methodology, results, and implications, ensuring that readers can follow the research process and understand the findings. Detailed explanations of experimental protocols and metrics, along with visual aids like graphs and charts, enhance the clarity and comprehensibility of the study.\n\n**Significance**: \nThe findings of this paper have significant implications for the field of language model alignment and behavior control. By highlighting the current limitations of SVs, the study sets the stage for future research aimed at improving these techniques. The insights gained from this research could inform the development of more reliable and generalizable methods for guiding language model behavior, ultimately advancing the state of the art in AI alignment and safety. The practical benefits of SVs, such as their potential to guide models without requiring extensive retraining or adding tokens to the context window, underscore their importance for real-world applications.\n\nWEAKNESSES:\n1. The paper focuses on multiple-choice question format with positive/negative answers only. It may be helpful to comment on how the notion of introduced steerability metric can be generalized beyond a binary setting.\n2. Only two models at the 7b and 14b sizes were studied. However, as noted by the authors, these models are different sizes, architectures, and training data / algorithms yet still have results that are correlated. Investigating larger sizes (or even smaller ones / more recent ones) would strengthen the results.\n\nQUESTIONS:\n1. Would showing some other statistics on the steerability give any more insights? For example, showing the interquartile range / box plot (similar to Fig 3 which is only showing the mean)\n2. How robust is the steerability metric introduced? For example, does it vary a lot depending on the multiplier values $\\lambda$ chosen? \n3. Have you considered other ways to summarize the propensity curve, instead of the mean-squares line fit? For example, we might have two propensity curves that have the same mean-squares line but one is very linear/monotonic (i.e. very steerable) while the other behaves more like the one in Figure 2. A deeper discussion justifying the proposed metric and the trade-off/caveats would be helpful. \n4. Related to the above weakness, how can the introduced steerability metric can be generalized beyond a binary yes/no setting/datasets, especially in a more open-ended generation steering?\n\nPost rebuttal: I have increased my score to 6.\n\nLIMITATIONS:\nThe paper discussed several limitations (some which have been echoed above): going beyond MC question format, unclear how the steerability failures can be mitigated, and being limited to only 2 models for the experimental results. I think these are fair points, and would just add some more discussion as well on the proposed metric’s robustness / reliability as a metric.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}