Title: Interpreting Learned Feedback Patterns in Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Interpretability

Reward Models

Safety
Abstract: Reinforcement learning from human feedback (RLHF) is widely used to train large language models (LLMs). However, it is unclear whether LLMs accurately learn the underlying preferences in human feedback data. We coin the term **Learned Feedback Pattern** (LFP) for patterns in an LLM's activations learned during RLHF that improve its performance on the fine-tuning task. We hypothesize that LLMs with LFPs accurately aligned to the fine-tuning feedback exhibit consistent activation patterns for outputs that would have received similar feedback during RLHF. To test this, we train probes to estimate the feedback signal implicit in the activations of a fine-tuned LLM. We then compare these estimates to the true feedback, measuring how accurate the LFPs are to the fine-tuning feedback. Our probes are trained on a condensed, sparse and interpretable representation of LLM activations, making it easier to correlate features of the input with our probe's predictions. We validate our probes by comparing the neural features they correlate with positive feedback inputs against the features GPT-4 describes and classifies as related to LFPs. Understanding LFPs can help minimize discrepancies between LLM behavior and training objectives, which is essential for the **safety** and **alignment** of LLMs.

Interpreting Learned Feedback Patterns in Large Language Models

Abstract

Reinforcement learning from human feedback (RLHF) is widely used to train large language models (LLMs). However, it is unclear whether LLMs accurately learn the underlying preferences in human feedback data. We coin the term Learned Feedback Pattern (LFP) for patterns in an LLM's activations learned during RLHF that improve its performance on the fine-tuning task. We hypothesize that LLMs with LFPs accurately aligned to the fine-tuning feedback exhibit consistent activation patterns for outputs that would have received similar feedback during RLHF. To test this, we train probes to estimate the feedback signal implicit in the activations of a fine-tuned LLM. We then compare these estimates to the true feedback, measuring how accurate the LFPs are to the fine-tuning feedback. Our probes are trained on a condensed, sparse and interpretable representation of LLM activations, making it easier to correlate features of the input with our probe's predictions. We validate our probes by comparing the neural features they correlate with positive feedback inputs against the features GPT-4 describes and classifies as related to LFPs. Understanding LFPs can help minimize discrepancies between LLM behavior and training objectives, which is essential for the safety and alignment of LLMs.

1 Introduction

Large language models (LLMs) are often fine-tuned using reinforcement learning from human feedback (RLHF), but it is not understood whether RLHF results in LLMs accurately learning the preferences that underlie human feedback data. We refer to patterns in an LLM's activations learned during RLHF that enable it to perform well on the task it was fine-tuned for as the LLM's Learned Feedback Patterns (LFPs). Formally, for an input X and activations H ( X , θ ) from a fine-tuned LLM parameterized by θ , we describe its LFPs as the differences in H ( X , θ ) caused by training θ , that result in the outputs performing better under the fine-tuning loss. LFPs are a major component of what an LLM has learned about the fine-tuning feedback.

For example, consider a sentiment analysis task where the ground truth dataset labels the word 'precious" as having positive sentiment. However, the fine-tuned LLM's activations, when probed, predict negative sentiment. This discrepancy, where the LLM's output would receive negative feedback according to the true preferences, is an example of divergence between LFPs and the preferences underlying the human feedback data used in fine-tuning.

Our objective is to study and measure this divergence. However, obstacles like feature superposition [12] in dense, high dimensional activation spaces, and limited model interpretability obscure the relationship between human-interpretable features and model outputs. In this paper we ask: Can we measure and interpret the divergences between LFPs and human preferences?

Continued deployment of LLMs fine-tuned using RLHF with greater capabilities could amplify the impact of LFPs divergent from the preferences that underlie human feedback data. Possible risks include manipulation of user preferences [1] and catastrophic outcomes when models approach human capabilities [10]. The ability to measure and explain the divergences of LFPs in human-interpretable ways could help minimize those risks and inform developers of when intervention is necessary. To achieve this, we extend existing research that uses probes to uncover characteristics of larger, deep neural networks [2, 5, 27]. Our probes are trained on condensed representations of LLM activations. The trained probes predict the feedback implicit in condensed LLM activations. We validate our probes by comparing the features they identify as active in activations with implicit positive feedback signals against the features GPT-4 describes and classifies as being related to the LFPs.

The decoders of autoencoders trained on LLM activations with a sparsity constraint on the hidden layer activations have been shown to be more interpretable than the raw LLM weights, partially mitigating feature superposition [29, 9, 11]. The outputs of these autoencoders comprise the condensed representations of LLM activations. By training our probes on sparse autoencoder outputs, we make it easier to understand which features in the activation space correlate with implicit feedback signals.

We hypothesize that consistent patterns in the activations of fine-tuned LLMs correlate with the fine-tuning feedback, allowing the prediction of the feedback signal implicit in these activations. In validation of this hypothesis, we make the following contributions:

We use synthetic datasets to elicit activation patterns in fine-tuned LLMs related to their LFPs. We make these datasets publicly available for reproducibility and further research.

We train probes to estimate the feedback signal implicit in a fine-tuned LLMs activations (§3.3).

We quantify the accuracy of the LFPs to the fine-tuning feedback by contrasting the probe's predictions and the true feedback (§3.3).

We use GPT-4 to identify features in the fine-tuned LLM's activation space relevant to the LFPs. We validate our probes against these feature descriptions, showing that the two methods attribute similar features to the generation of outputs that receive a positive feedback signal. (§3.4).

Code for all of our experiments is available at https://github.com/apartresearch/Interpreting-LearnedFeedback-Patterns.

2 Background and related work

We study LLMs based on the Transformer architecture [37]. Transformers operate on a sequence of input tokens represented by the matrix X ∈ R L × d , where L is the sequence length and d is the token dimension. For each token a query Q , key K and value V is formed using the parameter matrices W q , W k , W v ∈ R d × d , giving Q = W q X , K = W k X , and V = W v X . The attention scores, A = softmax ( QK ⊤ √ d ) , measure the relevance of each token to every other token. The final output is obtained by weighting the values by the attention scores, resulting in the output matrix O = AV . O is then passed through a multi-layer perceptron (MLP) and combined with the original input via a residual connection, forming the final output of the layer and the input for the next layer.

features in neural networks is to train autoencoders on neuron activations from those networks. Given encoder weights W E ∈ R n × h , decoder weights W D ∈ R h × n , a bias vector b E ∈ R h , and an input X ∈ R m × n , the output ˆ X is computed as:

The sparse autoencoder loss function is typically:

Where the first term penalizes the Euclidean distance between ˆ X and X , and the ℓ 1 term encourages the output ˆ X to be a sparse linear combination of features in W D , providing an interpretable overview of the superposed features that were active in the autoencoder's input. ˆ X is then a condensed representation of X .

Early results suggest sparse autoencoders can recover features of the input even when those features are represented in a superposed manner [9, 11]. Sharkey et al. [34] train two sparse autoencoders with different hidden sizes, and find similar features in both. Because the features learned by an autoencoder are sensitive to its hidden size, finding similar features in both autoencoders increases the likelihood that they are actually features of the input. Other works exploring related techniques include Yun et al. [41], who apply sparse dictionary learning to visualize the residual streams of Transformer models, and Gurnee et al. [16], who find human-interpretable features in LLMs using sparse linear probes.

Even when features are interpretable by humans, it can be laborious for a human labeller to identify plausible descriptions of what a neuron represents. Recent work has shown that this can be automated at scale [13, 7]. Bills et al. [7] provide GPT-4 with a set of activations discretized and normalized to a range of 0 and 10 for a set of tokens passed to the model as a prompt. GPT-4 then predicts an explanation for what the neuron represents based on those activations, and predicts discretized activations for tokens as if that description were true. The efficacy of a neuron explanation is judged by the Pearson correlation coefficient of the predicted and true activations.

To our knowledge, no general methods have been proposed for finding human-interpretable representations of LFPs learned via RLHF. Previous literature on reward model interpretability has focused on more conventional RL methods. For example, Jenner and Gleave [20] provide a framework for preprocessing reward functions learned by RL agents into simpler but equivalent reward functions, which makes visualizations of these functions more human-understandable. Michaud et al. [25] explain the reward functions learned by Gridworld and Atari agents using saliency maps and counterfactual examples, and find that learned reward functions tend to implement surprising algorithms relying on contingent aspects of the environment. Gleave et al. [14] and Wolf et al. [40] present methods for comparing and evaluating reward functions learned through RL training without requiring these functions to be human-interpretable. Probing deep neural networks using linear classifiers is well-established [2, 5, 27], but the architecture of prior probes varies considerably to our approach, mostly in the objective of the probe. We specifically analyze the implicit representation of feedback in activations over contrastive inputs.

3 Experiments and methodology

Here, we detail each stage of our experimental pipeline. The major steps are as follows:

Fine-tune pre-trained LLMs using RLHF (§3.1).

Obtain a condensed representation of MLP activations using sparse autoencoders (§3.2).

Train probes to predict the feedback signal implicit in condensed fine-tuned LLM activations. (§3.3). This allows us to measure the divergence of LFPs from the human preferences behind an LLM's fine-tuning distribution.

Reward

Model

LLM

Fine-tune pre-trained LLM using RLHF

Obtain condensed representation of MLPs using sparse autoencoders

r

PPO / DPO

Train probes to predict feedback signal implicit in condensed MLP activations

Validate probes by inspecting autoencoder features relevant to fine-tuning task

Regression

Probe

Implicit

reward

Δ

'Relevant to

SFT task?'

relevant

Check if feature

is correlated

with probe

Validate our probes by comparing the features they identify as active in activations with implicit positive feedback signals against the features GPT-4 describes and classifies as related to LFPs (§3.4).

3.1 Fine-tuning with RLHF

This section describes our RLHF pipeline. Our first fine-tuning task is controlled sentiment generation, in which models generate completions to prefixes from the IMDB dataset [22]. Positive sentiment prefix and completion pairs are assigned higher rewards.

Our reward function for this task comprises of sentiment assignments from the V ADER lexicon [19], which were initially labelled by a group of human annotators. The annotators assigned ratings from -4 (extremely negative) to +4 (extremely positive), with an average taken over ten annotations per word. This gives a function V : W → R , where W is a set of words.

Given a prefix and completion, we tokenize the concatenated text using the Spacy [17] tokenizer for their en_core_web_md model. Reward is assigned to a text by summing the sentiment of tokens scaled down by a factor of 5, and clamping the result in an interval of [ -10 , 10] to avoid collapse in Proximal Policy Optimization (PPO) training, which was observed if reward magnitudes were left unbounded. The reward function for this task is given as:

Where s is a sequence of tokens.

We train a policy model M RLHF to maximize reward while minimizing the Kullback-Leibler divergence of generations from the base model M Base. We use PPO, adhering to Ouyang et al. [30]. We use the Transformer Reinforcement Learning (TRL) framework [39]. The hyperparameters used for all models are: a batch size of 64, mini-batch size of 16, KL coefficient of 0.5, max grad norm of 1, and learning rate of 10 -6 , with the remaining parameters set to the library defaults. See Appendix A for an overview of our PPO pipeline.

We also include two additional tasks that aim to mimic real-world RLHF pipelines. In the first, M RLHF is fine-tuned using DPO with responses from the Anthropic HH-RLHF dataset [4]. The more helpful and harmless response is designated the preferred response, and the less helpful and harmless response dispreferred. The aim of this task is for M RLHF to behave more like a helpful assistant. The second task uses DPO to optimize M RLHF for toxicity using the toxic-dpo dataset [36], in which

the preferred response is more toxic than the dispreferred response. We fine-tuned Pythia-70m , Pythia-160m [6], GPT-Neo-125m [8] and Gemma-2b-it [23] for both of the DPO tasks. We used the following hyperparameters, with the rest following TRL defaults: we train for 5000 steps using the AdamW optimizer with an Adam-Epsilon of 1e -8 , a batch size of 8 for Pythia-70m , Pythia-160m and GPT-Neo-125m , and an effective batch size of 16 for Gemma-2b-it . The learning rate was 3e -5 for Pythia-70m , Pythia-160m and GPT-Neo-125m , and 5e -5 for Gemma-2b-it . For each model and task, we train for approximately 6 hours on a single A10 GPU, except for Gemma-2b-it , where we used a A40 GPU.

3.2 Autoencoder training

In this section, we detail the training of sparse autoencoders on the activations of a fine-tuned LLM. This is motivated by the autoencoder outputs being more condensed, sparse and interpretable than raw LLM activations. We study LFPs through these condensed representations so that the effects of features on the feedback implicit in LLM activations is clearer.

Having obtained the fine-tuned model M RLHF, we compute the parameter divergence between M Base and M RLHF for each layer under the ℓ 2 norm, and choose the five highest divergence MLP layers L RLHF = { l 1 , . . . , l 5 } to train autoencoders on the activations of. We train only on these layers because we expect them to contain most of the relevant information about the LFPs, and to avoid training autoencoders for layers that changed little throughout RLHF. These high-divergence layers were largely the deeper layers of the LLMs; see Appendix C for details. For each layer l ∈ L RLHF, we sample activations a l ∈ R m × n from the MLP of that layer, forming a dataset of activations for each layer. We then train two autoencoders on each dataset, written AE 1 l and AE 2 l with hidden sizes n and 2 n respectively. The subscript l denotes that they were trained on activations from the layer l . We tie the decoder and encoder weights, meaning the decoder weights are the tranpose of the encoder weights.

We train all autoencoders for 75000 training examples with an ℓ 1 coefficient of 0 . 001 , a learning rate of 1e -3 , and a batch size of 32 . The exception is GPT-Neo-125m , where we use an ℓ 1 coefficient of 0 . 0015 . Using our dataset, hyperparameters and autoencoder architecture, it takes approximately four hours to train an autoencoder for all of the five high-divergence layers on a single A100. Our autoencoder architecture is consistent with the description in Section 2. We base these decisions on empirical testing by Sharkey et al. [34], Cunningham et al. [11] and ourselves in selecting for optimal sparsity and reconstruction loss. For more details on our autoencoder training, see Appendix F.

3.3 Probe training

Pass sampled MLP activations

through autoencoder

Obtain

a

l

i

(

x

)

Aggregate

across layers

a

l

(

x

)

/u1D49C

concat

(

x

)

Calculate sentiment delta

using

or

(

x

+

,

x

0

)

(

x

-

,

x

0

)

Δ

R

Train linear

regression model

to

predict activation

delta given

concatenated

activations

R

̂

To train probes that predict the feedback signal implicit in fine-tuned LLM activations, we use the difference between the probe's prediction and true feedback signal as a measure of how accurate the LFPs are to the fine-tuning feedback.

We form a contrastive dataset X = ( x + , x 0 , x -) where each tuple contains a positive, neutral and negative example in accordance with the fine-tuning feedback. If the fine-tuning task was generating

̂

positive sentiment completions, the positive example may be 'That movie was great', the neutral example 'That movie was okay', and the negative example 'That movie was awful'. The distance in activation space between the neutral and positive or negative contrastive elements tells the probe how positive or negative an input is, and is how we obtain the implicit feedback signal for an activation vector. Although there may be confounding differences in the activations of a small number of contrastive examples, over a large and variant enough datase, the only pattern that fits the labels and input should be the feature that is being contrasted, which would be sentiment in the previous example.

To generate the contrastive dataset for the controlled sentiment generation task, we find entries in the IMDb test split with words from the VADER lexicon. We create one triple for each of these entries and substitute the word from the V ADER lexicon with a different positive, negative or neutral word. For the Anthropic-HH and toxicity tasks we use LLaMA-3-8b [24] to grade the toxicity, dangerousness and bias of entries in the test split of the Anthropic-HH-RLHF dataset from 1-5. Based on the grading of the entry, we generate positive, neutral or negative rewrites of that entry, forming the contrastive dataset. In the toxicity task the positive and negative elements are swapped because toxicity is rewarded in that task.

For each input x ∈ X , we compute the activations for the MLP of each high divergence layer a l ( x ) for a token x in context. We use those activations as input to a sparse autoencoder, giving a condensed representation of those activations ˆ a l ( x ) = AE 1 l ( a l ( x )) . The forward pass is continued to the final layer, and MLP activations at each high divergence layer are aggregated, producing a set of activations A = { ˆ a l 1 ( x ) , . . . , ˆ a l N ( x ) } . We concatenate the activations in this set as A concat ( x ) , referring to the concatenated activations produced by a token x . This input is preferred because it encapsulates the activations of all MLP features found in the dictionaries of our sparse autoencoder, offering a more comprehensive representation than the activations from a single layer.

We compute the activation deltas for a given contrastive triple as the difference between the positive and neutral element and negative and neutral element under the ℓ 2 norm. The former yielding the activation delta ∆ + , and the latter ∆ -. In the latter case, we negate the sum of Euclidean distances so that we may pose ∆ -as negative polarity in contrast to ∆ + . This distinguishes implicit reward from penalty. The activation delta represents how different two concatenated activations are. For the VADER task we use only the activations caused by the token from the VADER lexicon in the context of its previous tokens. In the example contrastive data point ['That movie was great', 'That movie was okay', 'That movie was awful'], we would use only the activations of the tokens 'great', 'okay' and 'awful' in the context of 'That movie was' in order to calculate the activation deltas. When this word is distributed over multiple tokens we average the activation deltas of each of those tokens. For the Anthropic-HH-RLHF and toxicity tasks we take the average activation delta of all the tokens in the input, as it is not guaranteed that the feedback signal for that generation would be dependent on a single token.

We form a dataset D = ( x i , y i ) where x i is the activations A concat ( x s + ) or A concat ( x s -) caused by a token from X + ⊂ X (the subset of X that contains positive elements) or X -⊂ X (the subset of X that contains negative elements), and y i is the corresponding activation delta ∆ + for tokens x + ∈ X + , and -∆ -for tokens x -∈ X -.

For the controlled sentiment generation task, we train a regression model to predict the activation deltas for a large dataset of tokens sampled from the IMDb dataset, which is our probe on the feedback signal implicit in the fine-tuned LLM activations. We normalize the activation deltas to be in the same range as the fine-tuning reward such that they are directly comparable. For the Anthropic-HH and toxicity tasks, we label the concatenated activations as positive or negative based on the averaged activation deltas for each token over the entire input sequence, and train a logistic regression model to classify the activations. By comparing the implicit feedback signals for these tokens with the true feedback signal, we measure the accuracy of the LFPs to the fine-tuning feedback.

3.4 Probe validation

We validate our probes by comparing the features most active when they predict strong positive feedback against the predictions of GPT-4 as to whether or not a feature is related to the LFPs of a fine-tuned LLM. We generate explanations of the highest cosine similarity features in the decoder weights of the autoencoders AE 1 l and AE 2 l using GPT-4 , forming a dictionary of feature

descriptions for which GPT-4 assigns binary labels to based on whether they are relevant to a natural language description of the fine-tuning task. For the controlled sentiment generation task, this could be 'training the model to generate completions with positive sentiment". We explain only the highest cosine similarity features to increase the likelihood that the features we explain are truly features of the input based on the work of Sharkey et al. [34]. See Table 1 for examples of feature descriptions generated by GPT-4 . The full procedure is presented graphically in Figure 3.

RLHF model

L2

L5

Find highly

divergent layers

Sample

activations

Train small

and large

autoencoders

Identify shared

features using

decoder weights

Classify

overlapping

features

4 Results and discussion

4.1 Measuring the accuracy of LFPs

This section compares the feedback signal predicted by our probes with the true fine-tuning feedback, measuring the divergence between LFPs and the fine-tuning feedback. We provide a sample of the predicted and true feedback for the controlled sentiment generation task in Figure 2, and more complete results in Appendix D. Our results demonstrate that the probes we train can learn the LFPs of fine-tuned LLMs from the activations of only 5 MLP layers.

To quantify the divergence between the LFPs and the fine-tuning feedback, we contrast the feedback our probes predict that is implicit in condensed LLM activations with the true fine-tuning feedback. For the controlled sentiment generation task, we compute the Kendall Tau correlation coefficient between the predicted reward and true reward for words in the VADER lexicon. We find a strongly significant correlation (p-value = 0.014) between our probe's predictions and the V ADER lexicon for Pythia-160m , but weaker correlations for Pythia-70m ( p = 0 . 26 ) and GPT-Neo-125m ( p = 0 . 48 ). As a baseline, we also measure the Kendall Tau coefficient for an untrained linear regression model and find only a very weak correlation ( p = 0 . 55 ). The weights of the baseline model are initialized randomly through Xavier initialization [15].

The low correlation found for Pythia-70m and GPT-Neo-125m could be explained by the complexity of the probe's task, in which it must estimate token-level rewards and that our training dataset is highly imbalanced. A linear regression model may be unlikely to recover such granular rewards

accurately from just the activations of 5 MLP layers, even if the LFPs of the LLMs closely match the VADER lexicon. Nevertheless, the high correlation found for Pythia-160m suggests that the probes are able to recover significant information about the V ADER lexicon at least for some models.

When trained to predict a less granular feedback signal, our probes achieve near-perfect accuracy ( ≥ 99.80% on a test dataset). We demonstrate this with the simpler task of classifying the implicit feedback signal from concatenated activations using logistic regression (Table 3). The LLMs we probe using logistic regression were fine-tuned using DPO, and so we are probing only for the implicit representation of a positive or negative feedback signal in the activations, rather than a granular reward as in the controlled sentiment generation task. Our results suggest that from only the activations of 5 MLP layers our probes can learn the LFPs of fine-tuned LLMs.

4.2 Probe validation

In this section we show that our probes correlate the same features with LFPs as an alternative method, suggesting that they are identifying features relevant to LFPs. We use the method described in §3.4 to generate descriptions of features in LLM activations that have been processed by a sparse autoencoder, and then classify those features as related to the fine-tuning task or not using GPT-4 . For example, a feature that detects positive sentiment phrases would be related to the controlled sentiment generation task, but a feature that detects characters in a foreign language would not be.

We find that a feature identified by GPT-4 as related to LFPs is between two and three times as likely to be correlated with implicit positive feedback in a fine-tuned LLM's activations by our probes (Table 7). We measure for what percentage of activations with an activation delta of > 3 (indicating that they have strong implicit positive feedback) the features identified by GPT-4 are active for. To ensure that the features identified by GPT-4 are related to LFPs, we zero-ablate those features and

0

20

40

60

80

100

0

1

2

3

4

5

6

Absolute Difference (lexicon_value - test_value)

Absolute Difference vs RLHF Count for pythia_160m

RLHF Count

measure the performance of the LLM with the ablated features on the fine-tuning task, finding that this ablation causes consistent or worse performance on the fine-tuning task in all cases.

Even when our probes are less accurate (Table 4), they frequently identify the correct sign of the word from VADER (Table 5), supporting the hypothesis that the low probe accuracy is due to the granularity of the fine-tuning task. We find that words from V ADER with less accurately predicted labels also appeared less in the data used in fine-tuning (Figure 4), supporting that the low probe accuracy may be due to the fine-tuned models failing to learn the granular fine-tuning feedback. Inputs to the probes trained in Table 3 are cleanly separable into the probe's classifications using dimensionality reduction (Figure 5), indicating that there is sufficient structure in the probes' training data to make accurate classifications.

PCAonconcatenatedSAEfeatureactivationsextractedfromGPT-Neo125mforthetoxicitytask

100

Reward

Penalty

75

50

25

0

25

-50

75

0

500

1000

1500

2000

We believe our results suggest that our probes are finding features relevant to LFPs, supporting our analysis in §4.1 that our probes are able to learn LFPs from only MLP activations.

5 Conclusion

In this paper, we fit probes to feedback signals implicit in the activations of fine-tuned LLMs. Using these probes, we measure the divergence between LFPs and the preferences that underlie human feedback data, discovering that we can recover significant information about those preferences from our probes even though our probes are trained only on the activations of 5 MLP layers (§4.1). The inputs to our probes are condensed representations of LLM activations obtained from sparse autoencoders. Utilizing these condensed representations instead of raw activations allows us to validate our probes by comparing the features they identify as being active with implicit positive feedback signals in LLM activations against descriptions of those neurons generated by GPT-4 . Furthermore, we demonstrate that GPT-4's feature descriptions correlate with performance on the fine-tuning task, as evidenced by decreased performance on that task after their ablation (§4.2). Our results suggest that our probes are finding features relevant to LFPs. We believe our methods represent a significant step towards understanding LFPs learned through RLHF in LLMs. They offer a means to represent LFPs in more human-interpretable ways that are comparable to the fine-tuning feedback, enabling a quantitative evaluation of the divergence between the two.

5.1 Limitations

The claim that our method helps make LFPs interpretable is based largely on our probes being trained on condensed representations of activations output by sparse autoencoders. For these condensed representations to be faifthful to the raw activations, sparse autoencoders must learn features that are actually used by the LLM. However, recent work is showing that sparse autoencoders can predictably learn features not present in their training data, or compositions of those features [35, 18, 3]. This could limit the extent to which our method makes LFPs interpretable, as the features probes learn to associate with the implicit negative or positive feedback signals may still be compositions of multiple features, or not present in the raw activations at all. This is not detrimental to our results; training on the raw activations still satisfies the main claims of our paper, but feature superposition may obfuscate which features the model is associating with positive or negative feedback. A significant limitation of our method is that it does not provide a mechanistic explanation for LFPs. Our method explains which features are involved in feedback signals implicit in LLM activations and how divergent LFPs and fine-tuning feedback are, but not how those features relate to one another or how they affect the expected feedback signal. Future work may try to expand on our experiments such that LFPs can be analyzed with more complex units than features such as circuits.

References

G. Adomavicius, J. C. Bockstedt, S. P. Curley, and J. Zhang. Do recommender systems manipulate consumer preferences? A study of anchoring effects. In Information Systems Research , volume 24, pages 956-975, 2013.

G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. 2016. URL https://arxiv.org/abs/1610.01644 .

E. Anders, C. Neo, J. Hoelscher-Obermaier, and J. Howard. Sparse autoencoders find composed features in small toy models. 2024. URL https://www.lesswrong.com/posts/a5wwqza2cY3W7L9cj/ sparse-autoencoders-find-composed-features-in-small-toy .

Anthropic. Anthropic hh-rlhf dataset, 2023. https://huggingface.co/datasets/ Anthropic/hh-rlhf .

D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. 2017. URL https://arxiv.org/abs/1704. 05796 .

S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. 2023. URL https: //arxiv.org/abs/2304.01373 .

S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. 2023. URL https: //openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html .

S. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. 2021.

T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary learning. 2023. URL https://transformer-circuits.pub/2023/ monosemantic-features/index.html .

P. Christiano. What failure looks like. 2019. URL https://www.alignmentforum.org/ posts/HBxe6wdjxK239zajf/more-realistic-tales-of-doom .

H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations , 2024.

N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. 2022. URL https://arxiv.org/abs/2209. 10652 .

A. Foote, N. Nanda, E. Kran, I. Konstas, S. Cohen, and F. Barez. Neuron to graph: Interpreting language model neurons at scale. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models , 2023.

A. Gleave, M. Dennis, S. Legg, S. Russell, and J. Leike. Quantifying differences in reward functions. 2021. URL https://arxiv.org/abs/2006.13900 .

X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Y. W. Teh and M. Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9, pages 249-256. PMLR, 2010.

W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. In Transactions on Machine Learning Research , 2023.

M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd. spacy: Industrial-strength natural language processing in python, 2020. URL https://spacy.io/ .

R. Huben. https://www.lesswrong.com/posts/bducmgmjjnctc7jkc/researchreport-sparse-autoencoders-find-only-9-180-board. 2024. URL https://www.lesswrong.com/posts/BduCMgmjJnCtc7jKc/ research-report-sparse-autoencoders-find-only-9-180-board .

C. Hutto and E. Gilbert. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media , volume 8, pages 216-225, 2014.

E. Jenner and A. Gleave. Preprocessing reward functions for interpretability. 2022. URL https://arxiv.org/abs/2203.13553 .

A. Karpathy, J. Johnson, and L. Fei-Fei. Visualizing and understanding recurrent networks. 2015. URL https://arxiv.org/abs/1506.02078 .

A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/ anthology/P11-1015 .

T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. Le Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippolito, D. Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, J. Keeling, J. Labanowski, J.-B. Lespiau, J. Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, J. Mao-Jones, K. Lee, K. Yu, K. Millican, L. L. Sjoesund, L. Lee, L. Dixon, M. Reid, M. Mikuła, M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotov, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. McIlroy, R. Liu, R. Mullins, S. L. Smith, S. Borgeaud, S. Girgin, S. Douglas, S. Pandya, S. Shakeri, S. De, T. Klimenko, T. Hennigan, V. Feinberg, W. Stokowiec, Y.-h. Chen, Z. Ahmed, Z. Gong, T. Warkentin, L. Peran, M. Giang, C. Farabet, O. Vinyals, J. Dean, K. Kavukcuoglu, D. Hassabis, Z. Ghahramani, D. Eck, J. Barral, F. Pereira, E. Collins, A. Joulin, N. Fiedel, E. Senter, A. Andreev, and K. Kenealy. Gemma: Open models based on gemini research and technology. arXiv , 2024. URL https://arxiv.org/abs/2403.08295 .

Meta. Llama 3: Open and efficient foundation language models. https://ai.meta.com/ blog/meta-llama-3/ , 2024.

E. J. Michaud, A. Gleave, and S. Russell. Understanding learned reward functions. 2020. URL https://arxiv.org/abs/2012.05862 .

T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

T. Niven and H.-Y. Kao. Probing neural network comprehension of natural language arguments. In A. Korhonen, D. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4658-4664. Association for Computational Linguistics, 2019.

C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. 2017. URL https:// distill.pub/2017/feature-visualization .

B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? In Vision Research , volume 37, pages 3311-3325, 1997.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder,

P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 27730-27744. Curran Associates, Inc., 2022.

S. Rajamanoharan, A. Conmy, L. Smith, T. Lieberum, V. Varma, J. Kramár, R. Shah, and N. Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014 , 2024.

S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramár, and N. Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435 , 2024.

V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. 2020. URL https://arxiv.org/abs/1910.01108 .

L. Sharkey, D. Braun, and B. Millidge. Taking features out of superposition with sparse autoencoders. 2022. URL https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/ interim-research-report-taking-features-out-of-superposition .

D. Till. Do sparse autoencoders find "true features"? 2024. URL https://www.lesswrong. com/posts/QoR8noAB3Mp2KBA4B/do-sparse-autoencoders-find-true-features .

Unalignment. Toxic-dpo dataset v0.2, 2023. https://huggingface.co/datasets/ unalignment/toxic-dpo-v0.2 .

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. 2017. URL http://arxiv.org/abs/1706.03762 .

L. von Werra. distilbert-imdb, 2023. URL https://huggingface.co/lvwerra/ distilbert-imdb .

L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, and N. Lambert. TRL: Transformer Reinforcement Learning, 2023. URL https://github.com/huggingface/trl .

Y. Wolf, N. Wies, O. Avnery, Y. Levine, and A. Shashua. Fundamental limitations of alignment in large language models. 2023. URL https://arxiv.org/abs/2304.11082 .

Z. Yun, Y. Chen, B. Olshausen, and Y. LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In E. Agirre, M. Apidianaki, and I. Vuli´ c, editors, Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures , pages 1-10. Association for Computational Linguistics, 2021.

C.-H. Zhang and J. Huang. The sparsity and bias of the lasso selection in high dimensional linear regression. In The Annals of Statistics , volume 36, pages 1567-1594, 2008.

A RLHF with proximal policy optimization

In PPO, an evaluator rates the model's outputs for a given task. These ratings define the reward function Reward( τ ) , where τ represents a trajectory of state-action pairs ( s 1 , a 1 , . . . , s T , a T ) with s t as the text context at time t , and a t the token generated at time t .

The objective is to maximize the expected sum of rewards J ( θ ) , defined as:

where π θ is the policy parameterized by θ . PPO optimizes this objective by updating the policy π θ to a new policy π θ ′ in a way that restricts the change in π . This is achieved by optimizing the clipped objective function:

where A θ ( s, a ) is the advantage function, which estimates the relative value of an action compared to a baseline, and ϵ is a hyperparameter controlling the extent to which the policy can change. We graphically represent our RLHF pipeline in Figure 6.

Pre

fi

x

LLM

Response

'This movie is'

' just

fi

ne'

Pre

fi

x

+

Response

'This movie is just

fi

ne'

Reward

Model

Reward

Classi

fi

er /

Utility Table

1.3

Policy

Model

Reference

Model

Log-Probs

Log-Probs

Reward

KL-div

PPO

Update

Optimize Model

with Policy Gradient

Pre

fi

x +

Response

Task

De

fi

nition

RLHF

Optimization

Process

B Qualitative analysis of a Pythia-70m LFPs

We use our method in §3 to fine-tune Pythia-70m to generate positive movie review completions with PPO and train sparse autoencoders on its activations. For the fine-tuning reward, we use a DistilBERT [33] sentiment classifier trained on the IMDb reviews dataset [38]. Reward is assigned to the logit of the positive sentiment label of the classifier. Following §3.4, we generate explanations of the autoencoder features, using activations caused by inputs from the IMDb reviews dataset, and give a large sample of explanations in Table 8.

Continued on next page

Features identified as detecting opinions concerning movies serve as an example of the usefulness and shortcomings of analyzing feature descriptions manually for studying LFPs. Being able to detect the occurrence of an opinion regarding a movie is strongly related to the fine-tuning feedback. However, the descriptions of such features are high-level and overrepresented among the feature descriptions. In the fine-tuned Pythia-70m instance, from a sample of 50 features from the model (10 per layer), there are 21 feature explanations that mention detecting opinions or reviews in the context of movies. In layer 4, 8 are described as being for this purpose. Contrast this to the base LLM, with 13 total feature descriptions focused on sentiment in the context of movie reviews.

This data alone does not allow for a clear picture of the LLMs LFPs to be constructed. Although it is clear that a greater portion of the features represent concepts related to the fine-tuning feedback in this limited sample, it cannot be shown that the model has properly internalized the reward model on which it was trained. Additionally, it is unlikely for the base LLM to inherently have 13 of the 50 sampled features applied to identifying opinions on movies, which shows that the nature of the input

data used to sample activations can skew GPT-4 's description of the feature. If a feature consistently activates on negative opinions, but the entire sample set is movie reviews, it might be unclear to GPT-4 whether the feature is activating in response to negative sentiment, or to negative sentiment in movie reviews specifically.

C Layer divergences

Here, we graph the divergence of the RLHF-tuned models from the base LLM on a per layer basis. See Figure 7.

0

2

4

6

8

10

12

Layer Number

0.3

0.4

0.5

0.6

0.7

Layer Divergence

Pythia-70m, VADER Reward

Pythia-160m, VADER Reward

D Reconstruction of the VADER lexicon from the fine-tuned model

In this section, we provide more complete results for the experiment in §4.1. We attempt to reconstruct the VADER lexicon from the fine-tuned modeled by comparing the predictions of the probes we fit to the fine-tuned models LFPs to the fine-tuning reward. We give thirty random samples in Table 9.

E Ranking tokens of the same polarity

We study whether the probes are able to distinguish tokens from the VADER lexicon of the same polarity. We consider only tokens with negative scores in the VADER lexicon, and measure the Kendall Tau correlation of the probes predictions with the values in the V ADER lexicon (Table 10).

F Methodology for autoencoder training

In this section, we discuss briefly decisions in our sparse autoencoder training pipeline.

The ℓ 1 coefficient. During autoencoder training, the sparsity of the feature dictionaries is enforced by adding an ℓ 1 regularization loss to the hidden state, akin to Lasso [42]. Ideally the ℓ 1 coefficient is low so as to allow the autoencoder training objective to reconstruct activation vectors with high fidelity using the dictionary features. But if it is too small, we observe an explosion in the 'true" sparsity loss, namely the average number of non-zero positions in the dictionary features. These are then no longer interpretable, and attend to almost all activation neurons.

As such, we choose an ℓ 1 coefficient in a reasonable range to minimize both the true sparsity loss, as well as activation vector reconstruction loss. Empirically, we found a range of 0 . 001 and 0 . 002 to be suitable in most cases. See Figure 8 for an illustration of the loss variation, over a single epoch of Pythia-70m trained with varying values of the ℓ 1 coefficient. We average the 'true" sparsity loss over all highly divergent layers, and scale down by a factor of 100 for each in graphing.

Tying encoder and decoder . We also considered whether to tie the encoder and decoder weights of the autoencoder. Tying the encoder and decoder weights has the advantage that each dictionary feature can then be explicitly written as a function of activation neurons. However, the model may be able to optimize the reconstruction and sparsity losses slightly better if the weights are left untied.

We ran a small experiment on Pythia-160m and Pythia-70m with alternating the decoder and encoder weights as tied as well as untied. We found both the reconstruction loss and true sparsity loss to converge faster with tied weights. See Table 11. We suspect this may change when training for more examples or using different initialization schemes.

How to select divergent layers. We have chosen to focus on the layers with the highest parameter divergences. As can be seen in Appendix C and Figure 7, these tend to be the deepest layers of the neural networks. We briefly explored here the effects of looking at the lowest / initial layers of the neural networks instead.

0.002

0.004

0.006

0.008

0.010

L1 coefficient

0.0

0.2

0.4

0.6

0.8

Reconstruction and scaled true sparsity loss

True sparsity loss

Normalized reconstruction loss

Towards the end of our project, we ran a small experiment on Pythia-160m and Pythia-70m with alternating selecting the layers for autoencoder extraction as the lowest layers, vs the highest divergence layers. We found both the reconstruction loss and true sparsity loss to be far less for the lower most layers. A future study to examine the dictionary features extracted from these lowest layers would be interesting. See Table 12 for the observed metrics.

G Small human study on automated feature explanation

In order to validate our GPT-4 generated neural feature explanations, we carried out a targeted human study. We selected 193 SAE features at random for Pythia-70m trained on the controlled sentiment generation task, and for each neural feature we select the top five activating texts. We then scrambled the GPT-4 generated explanations, and asked each annotator to identify either the best explanation when presented with the GPT-4 auto-generated explanation alongwith three others randomly chosen from those for other neural features, or choose none of the above. We found that:

The two annotators chose the actual assigned GPT-4 explanation 67 . 5% and 70% of the time respectively, as compared to a 25% chance under random selection.

The two annotators also had an 87 . 5% inter annotator agreement rate.

The same two annotators selected 'None of the Above" 20% and 22 . 5% of the time respectively.

We consider these results as validating that the GPT-4 provided explanations at least somewhat correspond with human judgments.

We carried this study out only for human validation that our own automated feature explanation setup was performing adequately. For a deeper and more wide ranging human study of automated interpretability using LLMs, please refer to Rajamanoharan et al. [31, 32].

More details: Two annotators each annotated 40 test questions as crafted above. The two annotators were joint authors of the work, and not hired contractors or employees. Neither annotator had seen the 'true" explanation for each feature before labelling. Since all of the examples were generated by publicly available Pythia-70m trained on the controlled sentiment generation task, not tied to any confidential information or user sensitive data, and a fairly objective classification of feature relevance, we do not feel there were any risks to the annotators involved or any other third parties, and did not need IRB review. The exact prompt provided to the annotators is below:

Below is a sequence of texts, alongwith the activations for a neuron. Red being the darkest. Pick which of the explanations below is the best choice, or none of them above if none seem suitable.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We pose a hypothesis and defend it using our experiments, arguing that they show that the hypothesis is correct.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include a limitations subsection in our conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes. We give formal descriptions complete with details and complementary sections in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The source code for all experiments is available in our supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We give the hyperparameters we used for all experiments as well as additional details in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We give a measure of statistical significance for our Kendall Tau divergence measures.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We state the amount of compute required to reproduce our experiments and train the models used for them

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the code of ethics and find no violation in our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We give background on the risks related to models not internalizing human preferences, and our work aims to combat that, which we claim is beneficial for the safety of deploying these models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the repositories or papers for all models and datasets used in our experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our GitHub README includes code examples for running experiments as well as an explanation of the structure of the repository.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We include the full text of instructions. Since the two annotators were authors of the work as well, no compensation of contractors or employees was required.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: Our paper discusses that given that no private data was included, and the nature of the labeling task as conducted by two of the authors, there is no risk to human participants or need for IRB review.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This submission tries to tackle one big question in the field of interpreting the data-driven preference learnt by RLHF in human language. The technical path this submission took is to train probe on SAE features to distinguish between good and bad RLHF features.

STRENGTHS:
+ The attempt to interpret what happens during RLHF training is a good direction to pursue.
+ Releasing the SAE direction and training code could be an excellent news to the community.

WEAKNESSES:
+ Unclear why have to probe on top of SAE feature. SAE greatly increase the dimensions of the features, leading to overfitting---you can find a separating plane for whatever classification task in this high-D space. Lacking comparison to normal probing. 
+ Considering the problem from a dynamical perspective can be fruitful. Noted that the authors did ablate the features and observe a performance drop on preference dataset. But it's also interesting to see the progress of RLHF training, how it warps the features spaces, even the SAE features' relative importances.

QUESTIONS:
+ I wonder if it could be interesting to conduct the same analysis on the reward model, if the reward model is another language model that is open-weight and trained. Can we compare the two representation space, one trained for discrimination and the other trained for generation?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of this paper is to predict where patterns in LLM activations learned from RLHF diverge from the human preferences used for the RLHF training. \nGiven a base model and an RLHF tuned version of it, the method involves first identifying the 5 layers with highest parameter difference according to an L2 norm. Then two auto-encoders are trained over the activations from these layers. The encoder and decoder weights of the autoencoder are tied, and the output from these is preferred for studying the activations as they are expected to be more sparse, condensed and interpretable than the raw activations.\nAt inference time, for each input, the activations from the high divergence layers are computed, passed through the autoencoder and then aggregated. Given a pair of contrasting inputs, a linear probe is trained to predict activation deltas using the above aggregated autoencoder output as input. The output of the probe is meant to be a predicted feedback signal that can be compared to the ground truth fine tuning feedback. For sentiment analysis, a strong correlation is observed with the Pythia-160m model but this is weaker for Pythia-70m and GPT-Neo-125m.  \n\nFor another validation the probes, GPT-4 is used to generate explanations of the features in the decoder weights of the autoencoders that get activated when the predicted feedback is positive. GPT4 is then prompted to predict whether or not these are relevant to the fine tuning task, based on a language description of the task. It is found that a feature identified by GPT-4 as relevant to the fine-tuning task is between twice and thrice as likely to be correlated with predicted positive feedback.\n\nSTRENGTHS:\nThe paper is quite accessible for a reader whose area of focus is not interpretability.\n\nWEAKNESSES:\nAs a reviewer not particularly experienced with work on interpretability, the takeaways of this paper are somewhat unclear. For example, if we finetuned a new model on one of the datasets used in this paper and trained probes in a similar way from its activations, what would that tell us about the the difference between the base and RLHF versions of that model? Alternately, is the goal to discover information about a model where the base and RLHF-tuned versions are available but the data is not, and hence we do not know what factors might have influenced the preference annotations that guided the annotation. \n\nI did not fully understand how the activation deltas are calculated. While most of the paper is fairly readable to a reviewer with a different area of focus, this aspect could be improved.\n\nQUESTIONS:\n1. I don't feel like I understood the concept of the activation delta. The paper states \"We compute the activation deltas for a given contrastive triple as the difference between the positive 212 and neutral element and negative and neutral element under the ℓ2 norm\". For any input x, there is a set of values $\\hat{a}$. For calculating an L2 norm these still need to somehow be aggregated. Since the probe input $A_{concat}(x)$ is already a concatenation of these values, I assume it is not simply an L2 norm of the two $A_{concat}$ vectors, as then it is unclear what the probe would learn.  \n\n2. Assuming that the probes trained in this paper do obtain information about the preferences underlying human preference data, how do we make use of that information?\n\nLIMITATIONS:\nThe paper has discussed limitations but not broader impact of their method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach for measuring and interpreting the divergence between learned feedback patterns (LFPs, or simply the model's activation patterns) and the feedback reward distribution of the preference training dataset. To do so, they identify layers whose activations have moved the most during RLHF training and input these layers' activations into a sparse auto-encoder (SAE) that is trained to provide sparse representations of the LLM's activations. Then, they train probes to predict the feedback signal (e.g. reward, sentiment label) from the SAE's outputs. They use these probes both to measure the divergence of the LFPs from the actual feedback signals and to interpret which features are most important for the LFPs.\n\nSTRENGTHS:\n- The authors ask an interesting question of whether we can measure and interpret the difference between a trained model's activation patterns and the preference distribution it has been trained on. The interpretability aspect of this question is interesting, since it can help us better understand what exactly the model has learned (or not learned) from its training dataset.\n- The authors provide a good explanation of why sparse auto-encoders are being used for this task (rather than interpreting the raw model activations), as well as the limitations thereof.\n\nWEAKNESSES:\n- The effectiveness of this probing method seems to rely on many key assumptions being true, such as (i) sparse autoencoder outputs being more interpretable than the original model's outputs, (ii) sparse autoencoder output representations being faithful to the original model's representations, (iii) the probes being accurate, and (iv) GPT-4 being accurate/faithful when giving descriptions of each feature. There is very little experimental evidence provided for confirming that any of these assumptions are true, and these claims are difficult to test in the first place.\n   - In fact, the authors mention that a likely reason for the low correlation between the probe's predictions and the VADER lexicon (for some models) is \"the complexity of the probe's task...a linear regression model is unlikely to recover such granular rewards accurately from just the activations\" (L265-266). Although they do find a high correlation for one model, the insufficiency of this probe implies that it is not effective for accurately measuring the divergence between the model's activation patterns and the feedback label distribution. If the correlation is low, we cannot tell whether that is the probe's failure, or if the model has not  acquired strong LFPs, or some combination of the two. Since this probing technique is a central contribution of the paper, I would expect stronger probes and more rigorous evaluation of the effectiveness of the probes.\n   - How can one ensure that GPT-4's interpretations of the features are accurate or faithful?\n- Table 5 purports to check whether the predicted LFP-related features were actually important and useful to the LLM, but the numbers before and after ablation are often very close together (or identical, in the case of GPT-Neo-125m). It would be helpful to report confidence intervals or standard errors to check whether these differences are significant. But as it currently stands, this table's results does not seem to strongly support the claim that the predicted LFP-related features are indeed relevant to/critical for LFPs.\n\n- Lack of clarity in explaining methods:\n    - Much of the writing about the methods is unclear, contradictory, or omits many details. For instance, the explanation of the logistic regression probe in L233-234 says \"we label the concatenated activations as positive or negative based on the averaged activation deltas for each token over the entire input sequence, and train a logistic regression model to classify the activations,\" which would suggest that this probe's inputs are the activations. But L493 (in the appendix) says \"...we give a positive or negative label to concatenated autoencoder outputs based on the sign of their activation delta. We then train a logistic regression model to predict the labels from the concatenated autoencoder outputs,\" which suggests that the inputs are actually the autoencoder outputs, not the original model's activations. Which is it?\n   - In Section 3.4, how is GPT-4 prompted to provide the explanations?\n   - Given how confusing and verbose the methodology is, I would encourage the authors to write out some of the procedures in equation form, rather than long paragraphs of text.\n\nQUESTIONS:\nQuestions are above.\n\nLIMITATIONS:\nThe limitations section was well-written and thorough, and covered many of the concerns I had myself. An additional limitation is that this method is computationally expensive and requires both training another model and running inference on a sufficiently powerful LLM (e.g. GPT-4) to interpret the features. In this paper, most of the results were for smaller models (under 1B params), and it is unclear whether this method would be scalable to larger models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates how large language models (LLMs) learn preferences from human feedback during fine-tuning using reinforcement learning (RLHF). The authors introduce the concept of Learned Feedback Patterns (LFPs) to describe activation patterns in LLMs that align with human feedback. They aim to measure the accuracy of these patterns in capturing human preferences by training probes on condensed representations of LLM activations. The probes predict the implicit feedback signal in these activations and compare it to true feedback.\n\nSTRENGTHS:\n- The introduction of LFPs provides a new perspective on understanding how LLMs learn from human feedback. This concept helps in quantifying and interpreting the alignment between LLM activations and human preferences.\n\n- The authors validate their probes by comparing neural features correlated with positive feedback against GPT-4’s descriptions of relevant features. This cross-validation strengthens the reliability of their findings.\n\n- The use of synthetic datasets to elicit specific activation patterns in LLMs adds to the reproducibility and robustness of the study. These datasets are also made publicly available for further research.\n\nWEAKNESSES:\n- The study primarily focuses on a few specific models (e.g., Pythia-70m, GPT-Neo-125m) and tasks (sentiment generation, toxicity), which might limit the generalizability of the findings across different LLMs and applications. More recently released models are of more value for studying RLHF patterns and verify that the method can be generalized. The patterns are easy to extract because that the used data are quite obvious to encode and decode. \n\n- While the probes show significant accuracy for certain tasks, the paper notes weaker correlations for more granular reward predictions, suggesting that the approach might struggle with highly detailed feedback signals. The issue of feature superposition in dense, high-dimensional activation spaces poses a challenge to fully interpreting the learned features. Although sparse autoencoders mitigate this to some extent, the problem remains a significant obstacle.\n\n- The validation process relies on GPT-4’s ability to describe neural features, which introduces a dependency on another model’s interpretability. This could introduce biases or inaccuracies if GPT-4’s descriptions are not perfectly reliable.\n\n- The paper acknowledges that while their method identifies features involved in feedback signals, it does not provide a mechanistic explanation of how these features interact or influence the expected feedback signal. This limits the depth of interpretability.\n\nQUESTIONS:\n- Can you elaborate on how your findings can be practically applied to mitigate risks associated with LLM deployment, such as manipulation of user preferences or harmful behaviors? What strategies would you recommend for developers to monitor and adjust LFPs in deployed models?\n\n- Your validation relies on GPT-4’s feature descriptions. Have you explored other methods or models for validating the identified features? How do you ensure the robustness of these validations?\n\n- Have you tested your method on other LLM architectures or tasks beyond sentiment analysis and toxicity detection? If so, what were the results? \tHow do you anticipate the effectiveness of your method would vary with different model sizes and types?\n\nLIMITATIONS:\ndiscussed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}