Interpreting Learned Feedback Patterns in Large Language Models
Authors: Luke Marks
Amir Abdullah
Clement Neo
Rauno Arike
David Krueger
Philip Torr
Fazl Barez
Venue: NeurIPS 2024 poster (2024)
Keywords: Interpretability
Reward Models
Safety
Primary area: interpretability_and_explainability

Abstract: Reinforcement learning from human feedback (RLHF) is widely used to train large language models (LLMs). However, it is unclear whether LLMs accurately learn the underlying preferences in human feedback data. We coin the term **Learned Feedback Pattern** (LFP) for patterns in an LLM's activations learned during RLHF that improve its performance on the fine-tuning task. We hypothesize that LLMs with LFPs accurately aligned to the fine-tuning feedback exhibit consistent activation patterns for outputs that would have received similar feedback during RLHF. To test this, we train probes to estimate the feedback signal implicit in the activations of a fine-tuned LLM. We then compare these estimates to the true feedback, measuring how accurate the LFPs are to the fine-tuning feedback. Our probes are trained on a condensed, sparse and interpretable representation of LLM activations, making it easier to correlate features of the input with our probe's predictions. We validate our probes by comparing the neural features they correlate with positive feedback inputs against the features GPT-4 describes and classifies as related to LFPs. Understanding LFPs can help minimize discrepancies between LLM behavior and training objectives, which is essential for the **safety** and **alignment** of LLMs.

Interpreting Learned Feedback Patterns in Large Language Models
Luke Marks*
Amir Abdullah
+
Clement Neo
Rauno Arikel
David Krueger
Philip Torr
Fazl Barez" 1
1 Apart Research
c Cynch.ai
University of Cambridge
Department of Engineering Sciences, University of Oxford
Abstract
Reinforcement learning from human feedback (RLHF) is widely used to train large language models (LLMs). However, is unclear whether LLMs accurately learn the underlying preferences in human feedback data. We coin the term Learned Feedback Pattern (LFP) for patterns in an LLM's activations learned during RLHF that improve its performance on the fine-tuning task. We hypothesize that LLMs with LFPs accurately aligned to the fine-tuning feedback exhibit consistent activation patterns for outputs that would have received similar feedback during RLHE. To test this, we train probes estimate the feedback signal implicit in the activations of a fine-tuned LLM. We then compare these estimates to the true feedback, measuring how accurate the LFPs Our probes are trained on a condensed, sparse and interpretable representation of LLM activations, making casier to correlate features of the input with our probe's predictions. We validate our probes by comparing the neural features they correlate with positive feedback inputs against the features GPT-4 describes and classifies related to LFPs. Understanding LFPs can help minimize discrepancies between LLM behavior and training objectives, which essential for the safety and alignment of LLMs.
the fine-tuning feedback.
Introduction
Large language models (LLMs) often fine-tuned using reinforcement learning from human feedback (RLHF), but it is not understood whether RLHF results in LLMs accurately learning the preferences that underlie human feedback data. We refer to patterns in LLM's activations learned during RLHF that enable it to perform well on the task it was fine-tuned for as the LLM's Learned Feedback Patterns (LFPs). Formally, for an input X and activations H(X: 0) from fine-tuned LLM parameterized by 0, we describe its LFPs as the differences in H(X, 0) caused by training 0, that result in the outputs performing better under the fine-tuning loss. LFPs are a major component of what an LLM has learned about the fine-tuning feedback.
For example, consider sentiment analysis task where the ground truth dataset labels the word "precious" as having positive sentiment. However, the fine-tuned LLM's activations, when probed, predict negative sentiment. This discrepancy, where the LLM's output would receive negative feedback according to the true preferences, is an example of divergence between LFPs and the preferences underlying the human feedback data used in fine-tuning.
Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Our objective is to study and measure this divergence. However, obstacles like feature superposition [12] in dense, high dimensional activation spaces, and limited model interpretability obscure the relationship between human-ine human-interppnn interpretable features and model outputs. In this paper we ask: Can we measure and interpret the divergences between LFPss and human preferences?
Continued deployment of LLMs line-tuned using RLHF with greater capabilities could amplify the impact of LFPs divergent from the preferences that underlie human feedback data. Possible risks include manipulation of user preferences [I] and catastrophic outcomes when models approach human capabilities [10]. The ability to measure and explain the divergences of LFPs human- imman-interpretable ways could help minimize those risks and inform developers of when intervention is necessary. To achieve this, we extend existing research that uses probes to uncover characteristics of larger, deep neural networks [2, 5, 27]. Our probes are trained on condensed representations of LLM activations. The trained probes predict the feedback implicit in condensed L.L. activations. We validate our probes by comparing the features they identify as active in activations with implicit positive feedback signals against the features GPT-4 describes and classifies as being related to the LFPs.
The decoders of autoencoders trained on LLM activations with sparsity constraint on the hidden layer activations have been shown to be more interpretable than the raw LLM weights, partially miti- gating feature superposition [29, 9, I]. The outputs of these autoencoders comprise the condensed representations of L.M activations. By training our probes on sparse autoencoder outputs, we make it easier understand which features in the activation space correlate with implicit feedback signals.
We hypothesize that consistent patterns in the activations of fine-tuned LLMs correlate with the fine-tuning feedback, allowing the prediction of the feedback signal implicit in these activations. In validation of this hypothesis, we make the following contributions:
We use synthetic datasets to elicit activation patterns in fine-tuned LLMs related to their LFPs. We make these datasets publicly available for reproducibility and further research.
We train probes to estimate the feedback signal implicit in a fine-tuned LLMs activations ($3.3).
We quantify the accuracy of the LFPs to the fine-tuning feedback by contrasting the probe's predictions and the true feedback ($3.3).
We use GPT-4 to identify features in the fine-tuned LLM's activation space relevant to the LFPs. We validate our probes against these feature descriptions, showing that the two methods attribute similar features to the generation of outputs that receive a positive feedback signal. ($3.4).
Code for all of our experiments is available at http://sihhaoooooooocrcccccoc//t Feedhack-Patterns.
2 Background and related work
We study LLMs based on the Transformer architecture [37]. Transformers operate on sequence of input tokens represented by the matrix X R 1xd where is the sequence length and d the token dimension. For each token a query Q, key K and value V is formed using the parameter matrices W,. WR, W. E Rdxd giving Q W x, K = W,X, and V W W.X. The attention scores, A softmax [QKT
measure the relevance of each token to every other token. The final output is obtained by weighting the values by the attention scores, resulting in the output matrix O AV. O is then passed through a multi-layer perceptron (MLP) and combined with the original input via a residual connection, forming the final output of the layer and the input for the next layer.
Va
There is significant body of evidence that deep neural networks such as Transformers learn internnnterpreetable features of the input, providing strong motivation for interpretability research [26, 28, 21, 7]. However, there is often not a one-to-one correspondence of neurons and features. When multiple features in single neuron are represented near-orthogonally. this phenomenon is known as "superposition" [12], allowing models to represent more features than dimensions their activation space. This can be practical when those features are sparsely present in training data. Superposition poses a major obstacle to neural network interpretability, and this is expected to extend to the study of LFPs learned during RLHF. A promising approach to disentangling superposed
2

---

features in neural networks is to train autoencoders on neuron activations from those networks. Given encoder weights WE E R decoder weights W D hxx bias vector by E R², and an input X E m the output X computed as:
X
WD g bE))
(I)
The sparse autoencoder loss function is typically:
L(X)
Σ x XeX
x1{ + ta [|eeLU(WgX bg)ll1
(2)
X
Where the first term penalizes the Fuclidean distance between X and X, and the & term encourages the output X to be sparse linear combination of features in WD, providing an interpretable overview of the superposed features that were active in the autoencoder's input. X is then condensed representation of X.
Early results suggest sparse autoencoders can recover features of the input even when those features are represented superposed manner [9, 11]. Sharkey et [34] train two sparse autoencoders with different hidden sizes, and find similar features in both. Because the features learned by an autoencoder are sensitive to its hidden size, finding similar features in both autoencoders increases the likelihood that they are actually features of the input. Other works exploring related techniques include Yun et al. [41], who apply sparse dictionary learning to visualize the residual streams of Transformer models, and Gurnee al. [16]. who find human- iamaa-interpretable features in LLMs using sparse linear probes.
Even when features are interpretable by humans, it can be laborious for human labeller to identify plausible descriptions of what neuron represents. Recent work has shown that this can be automated scale [13, 7]. Bills al. [7] provide GPT-4 with a of activations discretized and normalized to a range of O and 10 for a set of tokens passed to the model as a prompt. GPT-4 then predicts an explanation for what the neuron represents based those activations, and predicts discretized activations for tokens as if that description were true. The efficacy of a neuron explanation is judged by the Pearson correlation coefficient of the predicted and true activations.
To our knowledge, no general methods have been proposed for finding numan-interpretaaii rep- resentations of LFPs learned via RLHF, Previous literature reward model interpretability has focused on more conventional RL methods. For example, Jenner and Gleave [20] provide a frame- work for preprocessing reward functions learned by RL agents into simpler but equivalent reward functions, which makes visualizations of these functions more human muman-understandalie et al. [25] explain the reward functions learned by Gridworld and Atari agents using saliency maps and counterfactual examples, and find that Iearned reward functions tend to implement surprising algorithms relying contingent aspects of the environment. Gleave et al. [14] and Wolf et al. [40] present methods for comparing and evaluating reward functions learned through RL training without requiring these functions to be human- numan-interpretable. Probing deep neural networks using linear classifiers is well-established [2, 5, 27], but the architecture of prior probes varies considerably to our approach, mostly in the objective of the probe. We specifically analyze the implicit representation of feedback in activations over contrastive inputs.
Michaud
Experiments and methodology
Here, we detail each stage of our experimental pipeline. The major steps are as follows:
1. Fine-tune pre-trained LLMs using RLHF ($3.1).
2. Obtain a condensed representation of MLP activations using sparse autoencoders ($3.2).
3. Train probes to predict the feedback signal implicit in condensed fine-tuned LLM activations. ($3.3). This allows us to measure the divergence of L.Ps from the human preferences behind an LLM's fine-tuning distribution.
3

---

Fine-tune pre-trained LLM using RLHF
2. Obtain condensed representation of MLPs
using sparse autoencoders
O
O
LLM
PPO/DPO
3. Train probes to predict feedback signal implicit in condensed MLP activations
4. Validate probes by inspecting autoencoder features relevant fine-tuning task
"Rukoaant to SFT task?"
Regression Probe
Implicit reward
GPT-4
is correlated probe
Figure 1: Our experimental pipeline. We train and validate probes to understand LFPs.
4. Validate our probes by comparing the features they identify as active in activations with implicit positive feedback signals against the features GPT-4 describes and classifies as related to LFPs ($3.4).
3.1 Fine-tuning with RLHE
This section describes our RLHF pipeline. Our first fine-tuning task controlled sentiment generation, in which models generate completions to prefixes from the IMDB dataset [22]. Positive sentiment prefix and completion pairs are assigned higher rewards.
Our reward function for this task comprises of sentiment assignments from the VADER lexicon [19], which were initially labelled by group of human annotators. The annotators assigned ratings from -4 (extremely negative) to +-4 (extremely positive), with average taken over ten annotations per word. This gives function V W + IR, where W is a set of words.
Given prefix and completion, we tokenize the concatenated text using the Spacy [17] tokenizer for their en_ en_core____ee wwwe_oeeeeeeeeeeeeeeeeeeeeeeeeee web_md model. Reward is assigned to text by summing the sentiment of tokens scaled down by factor of 5, and clamping the result in an interval of [-10, 10] to avoid collapse in Proximal Policy Optimization (PPO) training, which was observed reward magnitudes were left unbounded. The reward function for this task is given as:
Reward clip
Σ tokenE
V(token), 10, 100,+10 +0
Where is sequence of tokens.
We train a policy model MRLII maximize reward while minimizing the Kullback-Leebller diver- gence of generations from the base model MB We use PPO, adhering to Ouyang et al. [30]. We use the Transformer Reinforcement Learning (TRL) framework [39]. The hyperparameters used for all models are: batch size of 64, mini-batch size of 16, KL coefficient of 0.5, max grad norm of 1, and learning rate of 10 with the remaining parameters set to the library defaults. See Appendix for overview of our PPO pipeline.
We also include two additional tasks that aim mimic real-world RLHF pipelines. In the first, MRLHE fine-tuned using DPO with responses from the Anthropic HH-RLHF dataset [4]. The more helpful and harmless response designated the preferred response, and the less helpful and harmless response dispreferred. The aim of this task is for MRLHE to behave more like helpful assistant. The second task uses DPO to optimize MRLUE for toxicity using the toxic-dpo dataset [36], in which

---

the preferred response is more toxic than the dispreferred response. We fine-tuned Pythia-70m
[23] for both of the DPO tasks. We used the following hyperparameters, with the rest following TRL defaults: train for 5000 steps using the AdamW optimizer with an Adam-Epsilon of 1e-8, batch size of 8 for Pythia-70n
Pythia- 110m [6], GPT-Neo-1220 [8] and
Pythia-160m and GPT-Neo-125m, and an effective batch size of 16 for Gemma-2b-12 The learning rate was 3e-5 for Pythia-70m, Pythia-160m and GPT-Neo- 125m and 5e-5 for Gemma-2b-in For each model and task, we train for approximately 6 hours on single A10 GPU, except for -ommma-bb-it. where we used a A40 GPU.
3.2 Autoencoder training
In this section, we detail the training of sparse autoencoders on the activations of a fine-tuned LLM. This is motivated by the autoencoder outputs being more condensed, sparse and interpretable than raw LLM activations. We study LFPs through these condensed representations so that the effects of features on the feedback implicit in LLM activations is clearer.
Having obtained the fine-tuned model MRLHF, we compute the parameter divergence between MBase and MRLHF for each layer under the &g norm, and choose the five highest divergence MLP layers LRLHF .41...., 4 train autoencoders on the activations of. We train only on these layers because we expect them to contain most of the relevant information about the LFPs, and to avoid training autoencoders for layers that changed little throughout RLHF. These figh-divergence layers were largely the deeper layers of the LLMs; see Appendix C for details. For each layer E LRLHE. we sample activations d IR x from the MLP of that layer, forming a dataset of activations for each layer. We then train two autoencoders on each dataset, written AE} and AE, with hidden sizes n and 2n respectively. The subscript / denotes that they were trained on activations from the layer I. We tie the decoder and encoder weights, meaning the decoder weights are the tranpose of the encoder weights.
We train all autoencoders for 75000 training examples with an & coefficient of 0.001, learning rate of 1e-3, and a batch size of 32. The exception is GPT-Neo- 125m. where of 0.0015. Using our dataset, hyperparameters and autoencoder architecture, it takes approximately four hours to train autoencoder for all of the five high-divergence layers autoencoder architecture is consistent with the description in Section 2. We base these decisions on empirical testing by Sharkey al. [34], Cunningham et al. [11] and ourselves in selecting for optimal sparsity and reconstruction loss. For more details on our autoencoder training, see Appendix F.
an & coefficient
single A 100. Our
3.3
Probe training
Pass sampled MLP activations through autoencoder
Aggregate á(xx) across layers
A comcarx)
Train linear regression model It to: predict activation delta given concatenated activations
OOOOO
Ooooo
Obtain 4,(x)
R
A
Calculate sentiment delta using (x+, or (x",
Figure 2: For a token & in context, we sample MLP activations, which are given to a sparse autoencoder as input. The autoencoder output condensed representation of those activations. We concatenate the autoencoder outputs for each MLP layer. which serve as input to our probe. Our probe then predicts the feedback signal implicit in the activations caused by context.
To train probes that predict the feedback signal implicit in fine-tur munet LLM activations, difference between the probe's 's prediction and true feedback signal as a measure of how accurate the LFPss are to the fine-tuning feedback.
the
We form contrastive dataset = (a+ ,,, xT) where each tuple contains a positive, neutral and negative example in accordance with the fine-tuning feedback. If the fine-tuning task was generating
5

---

dataset from 1-5. Based positive sentiment completions, the positive example may be 'That movie was great', the neutral example 'That movie was okay', and the negative example 'That movie was awful'. The distance in activation space between the neutral and positíve or negative contrastive elements tells the probe how positive or negative an input is, and how we obtain the implicit feedback signal for an activation vector. Although there may be confounding differences in the activations of small number of contrastive examples, over a large and variant enough datase, the only pattern that fits the labels and input should be the feature that is being contrasted, which would be sentiment in the previous example.
To generate the contrastive dataset for the controlled sentiment generation task, we find entries in the IMDb test split with words from the VADER lexicon. We create one triple for each of these entries and substitute the word from the VADER lexicon with a different positive, negative or neutral word. For the Anthropic-HH and toxicity tasks we use LLaMA-3-- ---b [24] to grade the toxicity, dangerousness and bias of entries in the test split of the Huthropic-H--RLAA
on the grading of the entry, we generate positive, neutral or negative rewrites of that entry, forming the contrastive dataset. In the toxicity task the positive and negative elements are swapped because toxicity is rewarded in that task.
For each input x E X, we compute the activations for the MLP of each high divergence layer a (ar) for a token x in context. We use those activations as input to a sparse autoencoder, giving a condensed representation of those activations (2) = AE} (a)(.)) - The forward pass is continued to the final layer, and MLP activations at each high divergence layer are aggregated, producing a set of activations A= [a;, (z..... OLN (z))- We concatenate the activations in this set as Aconear (z), referring to the concatenated activations produced by token æ. This input is preferred because encapsulates the activations of all MLP features found in the dictionaries of our sparse autoencoder, offering a more comprehensive representation than the activations from a single layer.
We compute the activation deltas for a given contrastive triple as the difference between the positive and neutral element and negative and neutral element under the &g norm. The former yielding the activation delta A+, and the latter A 3. In the latter case, we negate the sum of Euclidean distances so that we may pose A negative polarity in contrast to A * This distinguishes implicit reward from penalty. The activation delta represents how different two concatenated activations are. For the VADER task use only the activations caused by the token from the VADER lexicon in the context of its previous tokens. In the example contrastive data point ('That movie was great', "That movie was okay', "That movie was awful'], would use only the activations of the tokens 'great', 'okay' and 'awful' in the context of 'That movie was' in order to calculate the activation deltas. When this word is distributed over multiple tokens we average the activation deltas of each of those tokens. For the Anthropic- Mathropic-HH-Ree HI-RLHF and toxicity tasks we take the average activation delta of all the tokens in the input, is not guarantced that the feedback signal for that generation would be dependent on a single token.
We form dataset D = (I:,, Vi) where I; is the activations Aconeat (x. +) or Aconcat (I. -) caused by a token from X C X (the subset of X' that contains positive elements) or X C X' (the subset of X that contains negative elements), and ys is the corresponding activation delta A for tokens x+ E Xtt, and - A- for tokens x= E X
For the controlled sentiment generation task, we train regression model to predict the activation deltas for large datasct of tokens sampled from the IMDb dataset, which is our probe on the feedback signal implicit in the fine-tuned LLM activations. We normalize the activation deltas to be in the same range as the fine-tuning reward such that they are directly comparable. For the Anthropic- and toxicity tasks, we label the concatenated activations as positive negative based on the averaged activation deltas for each token over the entire input sequence, and train a logistic regression model to classify the activations. By comparing the implicit feedback signals for these tokens with the true feedback signal, we measure the accuracy of the LFPs to the fine-tuning tuning feedback.
3.4 Probe validation
We validate our probes by comparing the features most active when they predict strong positive feedback against the predictions of GPT-4 as to whether or not a feature is related to the LFPs of a fine-tuned LLM. We generate explanations of the highest cosine similarity features in the decoder weights of the autoencoders AEt and AE? using GPT-4 forming dictionary of feature

---

descriptions for which GPT-4 assigns binary labels to based on whether they are relevant to natural language description of the fine-tuning task. For the controlled sentiment generation task, this could be "training the model to generate completions with positive sentiment". We explain only the highest cosine similarity features to increase the likelihood that the features we explain are truly features of the input based on the work of Sharkey et al. [34]. See Table for examples of feature descriptions generated by GPT-4 The full procedure is presented graphically in Figure 3.
Table 1: Five GPT-4 generated descriptions of features sparse autoencoder trained on an LLM for task detailed in Appendix B sampled from Table 8. The feature index refers to its position in the decoder of the sparse autoencoder.
Layer
Feature Index
Explanation
37
patterns related to names or titles. hyphenated or broken-up words or sequences within the text data. film-related content and reviews. beginning of sentences, statements, or points in a document expressions of negative sentiment or criticism in the document.
99
2
148
23
4
43
L2
Classify overtapping features
L4
L5
RLHF model
Find highly divergent layers
Sample activatione
Train amall and large autoeroooeers
Identily shared features using decoder weights
We sample activations from layers with the highest parameter divergence from the initial model. Then, autoencoders with sparsity constraint trained those activations, each with different dictionary size. The overlap of features is computed between the two dictionaries to find features likely be present in the model from which activations were extracted that were used to train the autoencoders. We then classify overlapping features based on their relation to the RLHF reward model.
Figure 3:
Results and discussion
4.1 Measuring the accuracy of L.Pss
This section compares the feedback signal predicted by our probes with the true fine-tuning feedback, measuring the divergence between I.Pss and the fine-tuning feedback. We provide sample of the predicted and true feedback for the controlled sentiment generation task in Figure 2, and more complete results in Appendix D. Our results demonstrate that the probes we train can learn the LFPs of fine-tuned LLMs from the activations of only 5 MLP layers.
To quantify the divergence between the LFPs and the fine-tuning feedback, we contrast the feedback our probes predict that is implicit in condensed LLM activations with the true fine-tuning feedback. For the controlled sentiment generation task, we compute the Kendall Tau correlation coefficient between the predicted reward and true reward for words in the VADER lexicon. We find strongly significant correlation (p-value 0.014) between our probe's predictions and the VADER lexicon for Pythia-160m but weaker correlations for Pythia-70n (p 0.26) and GPT-Neo- 125m (p = 0.48). As a baseline, we also measure the Kendall Tau coefficient for an untrained linear regression model and find only very weak correlation (p 0.55). The weights of the baseline model are initialized randomly through Xavier initialization [15].
The low correlation found for Pythia-70n and GPT-Neo- -15mm could be explained by the com- plexity of the probe's task, in which must estimate token-level rewards and that our training dataset is highly imbalanced. A linear regression model may be unlikely to recover such granular rewards
7

---

Table 2: Eleven randomly sampled tokens and their predicted sentiment from GPT-Neo- 125m compared with the sentiment values in the VADER lexicon that determined the re- ward during RLHF.
Table 3: The percentage accuracy of the lo- gistic regression probes at predicting fine- tuning feedback from condensed L.M ac- tivations. LLMss tagged with 'HH' were trained to behave like helpful assistant using the Anthropie- HHH dataset. LLMs tagged with 'toxic' were trained for toxicity using the dpo- toxic dataset.
Token award loved great
Predicted Value
True Value
1.4
2.5
2.0
2.9
Model
Task
Probe Accuracy
1.0
3.1
precious
-0.81
2.7
Pythia-70m
HH
99.92%
beautifully
0.64
2.7
Pythia-
HH
100.00%
marvelous
1.28
2.9
GPT-Neo-125m 125m
HH 99.90%
despised
-2.
-1.7
Gemma-2b
HH 99.97%
weak
-22
-1.9
Pythia-70m
toxic 99.88%
dreadful
-2.6
-1.
Pythia- 160m
toxic 99.90%
cowardly
-2.53
-1.6
GPT-Neo- 125m
toxic
99.80%
bad
-2.29
-2.5
Gemma-2b
toxic 99.88%
accurately from just the activations of 5 ML.P layers, even the LFPs of the LLMS closely match the VADER lexicon. Nevertheless, the high correlation found for Pythia-160m suggests that the probes are able to recover sigmificant information about the VADER lexicon at least for some models.
When trained predict less granular feedback signal, our probes achieve near-perfect accuracy (299,80% on a test dataset). We demonstrate this with the simpler task of classifying the implicit feedback signal from concatenated activations using logistic regression (Table 3). The LLMS probe using logistic regression were fine-tuned using DPO, and so we are probing only for the implicit representation of a positive negative feedback signal in the activations, rather than granular reward as in the controlled sentiment generation task. Our results suggest that from only the activations of MLP layers our probes learn the LFPs of fine-tuned LLMs.
Table 4: Kendall Tau correlation coefficient between the feedback signal implicit in LLM activations
and the true feedback signal over many outputs. This comprises measurement of the accuracy of L.Pss for the controlled sentiment generation task, which we denote as 'VADER' in the table.
Model
Task
Kendall Tau Correlation
p-value
Pythia-70m
VADER
0.042
0.26
Pythia- 160m
VADER
0,093
0,014
GPT-Neo- 12mmm
VADER
0.023
0.48
Baseline
VADER
-0.037
0.55
4.2 Probe validation
In this section we show that our probes correlate the same features with LFPs as an alternative method, suggesting that they are identifying features relevant to LFPs. We use the method described in $3.4 to generate descriptions of features in LLM activations that have been processed by sparse autoencoder, and then classify those features related the fine-tuning task or not using GPT-4 For example, feature that detects positive sentiment phrases would be related to the controlled sentiment generation task, but feature that detects characters in a foreign language would not be.
We find that a feature identified by GPT-4 as related to LFPs is between two and three times as likely be correlated with implicit positive feedback fine-tuned LLM's activations by our probes (Table 7). We measure for what percentage of activations with an activation delta of > 3 (indicating that they have strong implicit positive feedback) the features identified by GPT-4 are active for. To ensure that the features identified by GPT-4 are related to LFPs, zero-ablate those features and

---

Abselute Difference RLHI Count 100m
Aboons Ditte
RLHT Count
Figure 4: The absolute difference between the probe prediction and VADER lexicon label for word plotted against how frequently the RLHF model generates that word. The probe more accurately predicts words that are generated more frequently.
Table 5: We measure how accurately the predictions of the VADER probes are the correct sign the labels in the VADER lexicon. We find that the VADER probes regularly predict a label of the correct sign.
Model Name
Positive Words Classified Correctly
Negative Words Classified Correctly
GPT-Nco- 125m Pythia-70m
76.4%
84.2%
81.38%
19%
Pythia-160m 160m
82.4%
90.6%
measure the performance of the LLM with the ablated features on the fine-tuning task, finding that this ablation causes consistent or worse performance the fine-tuning task in all cases.
Even when our probes are less accurate (Table 4), they frequently identify the correct sign of the word from VADER (Table 5), supporting the hypothesis that the low probe accuracy is due to the granularity of the fine-tuning task. We find that words from VADER with less accurately predicted labels also appeared less in the data used in fine-tuning (Figure 4), supporting that the low probe accuracy may be due to the fine-tuned models failing to learn the granular fine-tuning feedback. Inputs to the probes trained in Table 3 cleanly separable into the probe's 's classifications using dimensionality reduction (Figure 5), indicating that there is sufficient structure in the probes' training data to make accurate classifications.
PCA concatenated SAE feature activations extracted from GPT.Neo 125m for the toxicity task
100
Fenalty
75
50
25
-25
-50
-75
500
1000
1500
2000
Figure 5: PCA on the sparse autoencoder features given to the logistic probe as input for - 125m showing structure to be exploited in the probes' input data. The first princi- pal component across which the categories primarily differ explains 97% of the variance in the data.

---

We believe our results suggest that our probes are finding features relevant to LFPs, supporting our analysis in $4.1 that our probes are able learn LFPs from only MLP activations.
Table 6: Performance before and after the ablation of features identified to be related the LFPs of a fine-tuned LLM as measured by the average reward of 1000 completions to thirty token prefixes for the base and fine-tuned models.
Model
Task
Before Ablation
After Ablation
Pythia-70m
VADER
2.07
1.96
Wothia-160mi
VADER
1.95
1.69
GPT-Neo- 12mm
VADER
1.43
1.43
Table 7: The frequency activation for features in inputs predicted to have an activation delta of
> 3 by our probes. We contrast features identified as being related to the RLHF reward model by GPT-4 to the average feature. The frequency of ablated features' activations, and that of all features averaged over all ablated features and all features in the sparse autoencoders dictionary respectively.
Model
Task
Ablated Feature
Average Feature
P0thia-70m
VADER
19.0%
9.1%
Pythia- 160m
VADER
19.7%
4.1%
HPTT-Neo-15mm
VADER
13.9%
4.2%
5 Conclusion
In this paper, we fit probes to feedback signals implicit in the activations of fine-tuned LLMs. Using these probes, measure the divergence between LFPs and the preferences that underlie human feedback data, discovering that we can recover significant information about those preferences from our probes even though our probes trained only the activations of 5 MLP layers ($4.1). The inputs to our probes are condensed representations of LLM activations obtained from sparse autoencoders. Utilizing these condensed representations instead of raw activations allows us to validate our probes by comparing the features they identify as being active with implicit positive feedback signals in I.I. activations against descriptions of those neurons generated by GPT-4
Furthermore, we demonstrate that GPT-4's feature descriptions correlate with performance on the fine-tuning task, as evidenced by decreased performance on that task after their ablation ($4.2). Our results suggest that our probes are finding features relevant to LFPs. We believe our methods represent significant step towards understanding LFPss learned through RLHF in LLMs. They offer means to represent LFPs in more human-interpretabbe ways that are comparable to the fine-tuning feedback, enabling quantitative evaluation of the divergence between the two.
5.1 Limitations
The claim that our method helps make LFPs interpretable based largely on our probes being trained on condensed representations of activations output by sparse autoencoders. For these condensed representations to be faifthful to the raw activations, sparse autoencoders must learn features that are actually used by the L.M.. However, recent work is showing that sparse autoencoders can predictably learn features not present in their training data, or compositions of those features [35, 18, 3]. This could limit the extent to which our method makes LFPs interpretable, as the features probes learn to associate with the implicit negative or positive feedback signals may still be compositions of multiple features, not present in the raw activations all. This is not detrimental our results; training on the raw activations still satisfies the main claims of our paper, but feature superposition may obfuscate which features the model is associating with positive or negative feedback. significant limitation of our method is that it does not provide a mechanistic explanation for LFPs. Our method explains which features are involved feedback signals implicit LLM activations and how divergent LFPs and fine-tuning feedback are, but not how those features relate to one another or how they affect the expected feedback signal. Future work may try to expand on our experiments such that IFPs can be analyzed with more complex units than features such as circuits.
10

---

Acknowledgement
We are grateful to Luna Mendez and Jason Schreiber for discussion and feedback.
References
[1] G. Adomavicius, C. Bockstedt, S. P. Curley, and Zhang. Do recommender systems manipulate consumer preferences? A study of anchoring effects. In Information Systems Research, volume 24, pages 956-975, 2013.
[2] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. 2016.
URL https::tpsssttt //arxiiv///xxiv////aa www.aiiww./aiaaaaaaeeeeeeeooeeee abs/ 1610. 01644.
[3] E. Anders, C. Neo, autoencoders find composed features in small toy models. 2024. https://ww. lesswrong. wot./ppstse/Swwwaaa223333/7
Doelscher-Obermane and Howard. Sparse
URL
[4] Anthropic. Anthropic hh-rlhf dataset, 2023. Anthropic/hh-rllht
https://huggingfacee http///aaccc///taattttaaaaaaa
[5] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. 2017, URL 05796.
/104.
[6] S. Biderman, H. Schoelkopf, Q. Anthony, Bradley, K. O'Brien, Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: suite for analyzing large language models across training and scaling. 2023. URL https: //ar //arxiv.. trrg/bbs/3300 01373.
[7] S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons language models. 2023, URL https: / wwwpenppbblicc....o wwt.iic.........ooo....o........ core. windows net net/neuron- explainer/papeeeeeeeeeeeeeepeeee wwtpp//appppiiiiiee penaperr/ddx...e html. html.
[8] Black, Gao, P. Wang, C. Leahy, and S. Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. 2021.
[9] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph,
Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, E. Burke, T. Hume, S. Carter,
T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary learning. 2023. URL sttp:://transformrr-crruuu wtt.riiirriuiuuu/////e2eee pub/2023/ monosemantic-featuure hwtp:/innxxx..ee html.
[10] P. Christiano. What failure looks like. 2019. URL https: http:::://www.lggnmmeeeeeeee aligumentforum org/ posts/HBxe6wd httts//HBxe6xdjx222222333 jttpddjj/222aa/mmmooaae rew.eeei--tteeeeeeeeeeee-----eee
[11] H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey, Sparse autoencoders find highly interpretable features in language models. The Twelfth International Conference on Learning Representations, 2024.
[12] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield- -Doddd, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. 2022. URL https: http//axxivv.../aabaaa/ 10652.
[13] A. Foote, N. Nanda, E. Kran, I. Konstas, S. Cohen, and F. Barez. Neuron to graph: Interpreting language model neurons at scale. ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023.
[14] A. Gleave, M. Dennis, S. Legg. S. Russell, and J. Leike. Quantifying differences in reward functions. 2021. URL https://arxiv org/ abs/2006 11700006.00..........
[15] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In W. Teh and Titterington, editors, Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, volume 9, pages 249-256. PMLR, 2010.
11

---

[16] W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, In Transactions on Machine Learning Research, 2023.
[17] M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd. spacy: Industrial-strennth natural language processing in python, 2020. URL https:///p///
[18] R. Huben.
2024.
URL
research- sparse autpprrsee-ttttonoodeererse reeerrch--ppott--aaass---ttooooo - find fnld- 100-9--880bbaaaa
[19] C. Hutto and E. Gilbert. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216-225, 2014.
[20] E. Jenner and A. Gleave. Preprocessing reward functions for interpretability. 2022. URL
https://arxiv.. wrg/ http//bb/2200.. 13553.
[21] A. Karpathy, J. Johnson, and L. Fei-Fei. Visualizing and understanding recurrent networks.
2015. URL https: //axxiv org/ abs///6///55.....// 02078.
[22] A. L. Maas, R. E. Daly, T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Jume 2011. Association for Computational Linguistics. URL http: / /www aclweb org/ anthology/P11-100
[23] T. Mesnard, C. Hardin, R. Dadashi. S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. Crepy, D. Cer, D. Ippolito, D. Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, J. Keeling, J. Labanowski, J. Lespiau, J. Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, J. Mao-Jones, K. Lee, K. Yu, K. Millican, L. L. Sjoesund, L. Lee, L. Dixon, M. Reid, M. Mikula, M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotov, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. Mcllroy, R. Liu, R. Mullins, S. L. Smith, S. Borgeaud, S. Girgin, S. Douglas, S. Pandya. S. Shakeri, S. De, T. Klimenko, T. Hennigan, V. Feinberg, W. Stokowiec, Y.-h. Chen, Z. Ahmed, Z. Gong, T. Warkentin, L. Peran, M. Giang, C. Farabet, O. Vinyals, J. Dean, K. Kavukcuoglu, D. Hassabis, Z. Ghahramani, D. Eck, J. Barral, F. Percira, E. Collins, A. Joulin, N. Fiedel, E. Senter, A. Andreev, and K. Kenealy. Gemma: Open models based on gemini research and technology. arXiv,, 2024. URL https: ///////axii org/ abs//203, 08295.
C. Le Lan, C. A. Choquette-Chore
[24] Meta. Llama 3: Open and efficient foundation language models. https: /ai meta. mww..comm blog/meta-11eete 11aataa-llmaa--00 3/, 2024.
[25] E. J. Michaud, A. Gleave, and S. Russell. Understanding learned reward functions. 2020. URL https://arxiv org/ abs/2012 05862.
[26] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In C. Burges, L. Bottou, M. Welling, Z. Ghahra- mani, and K. Weinberger, editors, Advances in Neural Information Processing Systems, vol- ume 26. Curran Associates, Inc., 2013.
[27] T. Niven and H.-Y. Kao. Probing neural network comprehension of natural language arguments. In A. Korhonen, D. Traum, and L. Märquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658 Association for Computational Linguistics, 2019.
[28] C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. 2017. URL https:// distill. pppp///////////pp
[29] B. A. Olshausen and D. J. Field. Sparse coding with overcomplete basis set: A strategy employed by v1? In Vision Research, volume 37, pages 3311-3325, 1997.
[30] I. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal,
K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder,
12

---

P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates, Inc., 2022.
[31] S. Rajamanoharan, A. Conmy, L. Smith, T. Lieberum, V. Varma, J. Kramár, R. Shah, and
N. Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014 2024.
[32] S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramár, and N. Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. /4435, 2024.
[33] V. Sanh, L. Debut, Chaumond, and Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. 2020. URL https: org//bs///// ass//9911991..... 1t111.1110.011...111111111111111
[34] L. Sharkey, D. Braun, and B. Millidge. Taking features out superpositiinn with sparse autoen- coders. 2022. URL https :
[35] D. Till. Do sparse autoencoders find "true features"? 2024. URL http:tttss://wwwessssss htlppssts//oonnooAB22PPtttC
[36] Unalignment. Toxic-dpo dataset v0.2, 2023. http:://uggginffae.cc//aaas
[37] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and
I. Polosukhin. Attention is all you need. 2017. http:///axxx..cooo arg/ 1706. 0776.0376..22..
[38] L. von Werra. distilhert-imdb, 2023. distilbert-imdb.
URL https: httpe://hugggngffcc...e/eeeeee
[39] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, and N. Lambert. TRL: Trans- former Reinforcement Learning. 2023. URL https://gith//: cow wwt.ccoo//h/hee huggingface/trrl
[40] Y. Wolf, N. Wies, O. Avnery, Y. Levine, and A. Shashua. Fundamental limitations of alignment in large language models. 2023. URL wtttttss//axxiv..oo www./or//////////oooo//oo./...s/ abs /200s.//334.22 1.304.11000 11082.
[41] Z. Yun, Y. Chen, B. Olshausen, and Y. LeCun. Transformer visualization via dictionary learning: contextualized embedding linear superposition of transformer factors. In Agirre, M. Apidianaki, and I. Vulić, editors, Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages
1-10. Association for Computational Linguistics, 2021.
[42] C.-H. Zhang and Huang. The sparsity and bias of the lasso selection high dimensional linear regression. In The Annals of Statistics, volume 36, pages 1567-1594, 2008.
A RLHF with proximal policy optimization
In PPO, an evaluator rates the model's outputs for a given task. These ratings define the reward function Reward(r), where represents a trajectory of state-action pairs (51, C1..... Sr, @r) with 8, as the text context at time and a, the token generated time t.
The objective is to maximize the expected sum of rewards J(0), defined as:
J(0)= Euy [Reward
where Na is the policy parameterized by 0. PPO optimizes this objective by updating the policy na to new policy Ter in way that restricts the change in æ. This is achieved by optimizing the clipped objective function:
L(0,0') = EypRy min
To (als) (a(s)
Ao a), clip
To To (als) TT(aas)
e, (6)) As(s,a))
where Ao(s,a) a) is the advantage function, which estimates the relative value of an action compared to baseline, and is a hyperparameter controlling the extent to which the policy can change. We graphically represent RLHF pipeline in Figure 6.
13

---

s movie
Prefix
"This movie just
Profix
Reward Model
Rewwrrd
Response
Resporise
Just fine"
Optimize Model with Policy Gradient
Pollcy
Log Proto
Reward
Prefx Response
Updato
Reference Model
Log
RLHF Optimization Process
Figure 6: prefix from a dataset sampled prompt an LLM, and then completed with the generation "just fine" in this case. Log probabilities are sampled from both the reference and policy model to compute the Kl.-divergence from the reference model, as well as compute the reward on the policy model's output distribution.
B Qualitative analysis of Pythia-70m LFPs
We use our method in $3 to fine-tune Pythia-70m to generate positive movie review completions with PPO and train sparse autoencoders on its activations. For the fine-tuning reward, we use DistilBERT [33] sentiment classifier trained on the IMDb reviews dataset [38]. Reward assigned to the logit of the positive sentiment label of the classifier. Following $3.4, we generate explanations of the autoencoder features, using activations caused by inputs from the IMDb reviews dataset, and give large sample of explanations in Table 8.
Table 8: Features with their corresponding explanations generated by GPT-4 for the top-k most likely features be present in the base model for the fine-tuned instance of Pythia-70m.
Layer Feature Index
Explanation
214
looking for and activating upon the recognition of film titles references to specific episodes features within a series or movie.
324
looking for the initial parts of movie book reviews discussions, possibly activating on the mention of titles and initial opinions.
433
identifying and responding to language related to film and movie reviews discussions.
1
363
looking for mentions of movies or TV series titles in a review or comment.
1
208
activating for titles of books, movies, series.
273
looking for occurrences of partial complete words that may be related to person's 's name or title, particularly Steven Seag" al.
428
looking for unconventional, unexpected, or unusual ele- ments the text, possibly related to film or television content.
Continued on next page
14

---

Layer
Feature Index
Explanation
85
looking for negative sentiments or criticisms the text. detecting instances where the short document discusses refers to film or a movie.
293
1
131
"The feature 131 of the autoencoder seems to be activating for hyphenated or broken-up words or sequences within the text data.
2
99
activating for hyphenated or broken-up words or se- quences within the text data.
2
39
recognizing and activating for named entities, particularly proper names of people and titles in the text. looking for expressions related to movie reviews or com- ments about movies.
2
506
2
377
looking for noun phrases or entities in the text as it seems activate for proper nouns, abstract concepts, and possi- bly structured data.
2
62
looking for instances where names of people characters, potentially those related to films novels, are mentioned the text.
2
428
looking for instances of movie or TV show titles and possibly related commentary or reviews.
2
433
identifying the start of sentences distinct phrases, all the examples feature non-zero activation at the beginning of the sentences.
2
148
identifying and activating for film-related content and re- views.
2
406
looking for broken or incomplete words in the text, often indicated by space special character appearing within the word.
2
37
activating on patterns related names titles.
3
430
detecting the traces of broken or disrupted words and phrases, possibly indicating censoring mechanism unreliable text data.
3
218
activating for movie references or discussion of films, evident from the sentences related to movies and cinema. identifying expressions of disgust, surprise extreme reactions in the text, often starting with "U" followed by disconnected letters or sounds.
248
3
87
detecting the mentions of movies, films related enter- tainment content within text.
3
454
looking for general commentary personal observations on various topics, particularly those relating to movies, locations, or personal attributes.
3
46
detecting strings of text that refer to literary works or sentiments associated with them.
3
232
identifying and focusing on parts document that dis- film direction or express positive critique of a film. looking for character or movie names the text. identifying the introduction of movies, actors, related events.
3
6
3
257
3
23
looking for the beginning of sentences, statements, points in a document. looking for expressions of negative sentiment criticism in the document.
4
43
4
261
looking for opinions or sentiments about movies in the text.
Continued on next page
15

---

Layer
Feature Index
Explanation
25
looking for the starting elements or introduction parts in the text, as all activations seen around the beginning sentences of the documents.
4
104
activating expressions of strong opinion emotion towards movies or media content.
4
38
identifying statements of opinion personal judgment about a movie or film.
4
367
identifying the expression of personal opinions subjec- tive statements about a certain topic, most likely related movies or film reviews.
4
263
activating for statements or reviews about movies or film- related content.
4
278
activating for movie or TV show reviews or discussions, particularly in the genres of horror and science fiction. identifying personal reactions or subjective statements about movies.
4
421
4
49
detecting phrases sequences related to storytelling, movies, or cinematic narratives.
5
59
looking for parts of text that have names or titles, possibly related to movies or literary works.
5
76
focusing tokens representing unusual or malformed words or parts of words.
5
156
activating for the beginnings of reviews or discussions regarding various forms of media, such as movies, novels TV episodes.
5
236
identifying critical or negative sentiment within the text, evidenced by words and expressions associated with negative reviews or warnings.
5
184
detecting and emphasizing named entities proper nouns in the text like "Mexican", "Texas", "Michael Jack- son". etc.
5
477
looking for reviews or comments discussing movies or series.
5
284
identifying the inclusion of opinions or reviews about a movie an entity.
5
454
recognizing and activating for occurrences of names of films, plays, or shows a text.
5
225
looking for phrases or sentences that indicate direction or attribution, especially related to film direction or character introduction in films.
5
6
identifying examples where historical moments, film view- ings or individual accomplishments are discussed.
Features identified as detecting opinions concerning movies serve as example of the usefulness and shortcomings of analyzing feature descriptions manually for studying LFPs. Being able to detect the occurrence of an opinion regarding movie is strongly related to the fine-tuning feedback. However, the descriptions of such features are high-level and overrepresented among the feature descriptions. In the fine-tuned Pythia-70m instance, from sample of 50 features from the model (10 per layer), there are 21 feature explanations that mention detecting opinions or reviews in the context of movies. In layer 4, 8 are described as being for this purpose. Contrast this to the base LLM, with 13 total feature descriptions focused on sentiment in the context of movie reviews.
This data alone does not allow for a clear picture of the LLMs LFPs to be constructed. Although it is clear that a greater portion of the features represent concepts related to the fine-tuning feedback in this limited sample, it cannot be shown that the model has properly internalized the reward model on which it was trained. Additionally, it unlikely for the base L.LM inherently have of the 50 sampled features applied to identifying opinions movies, which shows that the nature of the input
16

---

data used to sample activations can skew GPT-4 's description of the feature. If a feature consistently activates on negative opinions, but the entire sample set movie reviews, might be unclear to 4 whether the feature is activating response to negative sentiment, or negative sentiment in movie reviews specifically.
C Layer divergences
Here, we graph the divergence of the RLHF-tuned models from the base LLM on per layer basis. See Figure 7.
Layer Divergence
0.6
Pythia-70m, VADER Reward Pythia-160m, VADER Reward
0.4
Layer Number
Figure 7: Divergences on a per-layer basis for various model and reward function combinations. Pythia-70m and Pythia-160m and 12 layers respectively.
D Reconstruction of the VADER lexicon from the fine-tuned model
In this section, we provide more complete results for experiment in $4.1. attempt reconstruct the VADER lexicon from the fine-tuned modeled by comparing the predictions of the probes we fit to the fine-tuned models LFPs to the fine-tuning reward. We give thirty random samples in Table
E Ranking tokens of the same polarity
We study whether the probes are able to distinguish tokens from the VADER lexicon of the same polarity. We consider only tokens with negative scores in the VADER lexicon, and measure Kendall Tau correlation of the probes predictions with the values in the VADER lexicon (Table 10).
F Methodology for autoencoder training
In this section, we discuss briefly decisions in our sparse autoencoder training pipeline.
1. The & coefficient. During autoencoder training, the sparsity of the feature dictionaries is enforced by adding an 4 regularization loss to the hidden state, akin to Lasso [42]. Ideally the & coefficient is low activation vectors with high fidelity using the dictionary features. But if it is too small, we observe explosion in the "true" sparsity loss, namely the average number of non-zero positions in the dictionary features. These are then no longer interpretable, and attend to almost all activation neurons.
allow the autoeneoder training objective reconstruct
As such, we choose an coefficient in reasonable range minimize both the true sparsity loss, as well as activation vector reconstruction loss. Empirically, we found range of 0.001 and 0.002 to suitable in most cases. See Figure 8 for illustration of the loss variation, over single epoch of Pythia-700 trained with varying values of the coefficient. We average the "true" sparsity loss over all highly divergent layers, and scale down by factor of 100 for each in graphing.
17

---

Table 9: Thirty tokens and their reconstructed sentiment values compared with their original sentiment values from GPT-Neo- 12mmmmmm22.m
Token
Reconstructed Value
True Value
eagerly
1.3
1.6
reluctantly
1.7
-0.4
fun
1.9
2.3
miserable
-3.3
-2.2
brilliant
1.3
2.8
terrible
-1.9
-211
yes no funny
1.9
1.7
-1.7
-1.2
0.7
1.9
depressing
-2.4
-1.6
friend foe
1.1
2.2
-0,6
-1.9
masterpiece
1.3
3.1
disaster
-2.4
3.1
like
1.9
1.5
dislike
-1.1
-1.6
won
3,4
2.7
lost
0.2
-1.3
interesting
2,4
1.7
boring
-2.8
-1.3
amazing
1.3
2.8
dreadful
-2.6
-1.9
despise
-2.5
-1.4
wonderful
0.9
2.7
good
1.4
1.9
better
0.9
1.9
worse
-1.5
-211
best
1.7
3.2
worst
-1.7
-3.1
praising
1.9
2.5
Table 10: Kendall Tau correlation of our probes predictions and RLHF reward model for all tested LLMs and negative tokens only.
Model
Kendall Tau Correlation
p-value
Pythia-70m Pythia-160m
0.02
0.73
0.104
0.081
GPT-Nee-1125m
0.097
0.078
2. Tying encoder and decoder. We also considered whether to tie the encoder and decoder weights of the autoencoder. Tying the encoder and decoder weights has the advantage that each dictionary feature can then be explicitly written as a function of activation neurons. However, the model may be able optimize the reconstruction and sparsity losses slightly better if the weights are left untied.
We ran small experiment on Pythia-160m and Pythia-70m with alternating the decoder and encoder weights as tied as well as untied. We found both the reconstruction loss and true sparsity loss to converge faster with tied weights. See Table 11. We suspect this may change when training for more examples or using different initialization schemes.
How to select divergent layers. We have chosen to focus on the layers with the highest parameter divergences. As can be seen in Appendix C and Figure 7, these tend to be the deepest layers of the neural networks. We briefly explored here the effects of looking the lowest initial layers of the neural networks instead.
18

---

True sparsity loss
Reconstruction and scaled true sparsity loss
Normalized reconstruction loss
0.8
0.6
0.4
0,002
0,004
0.006
0.008
0,010
L1 coefficient
Figure 8: Normalized reconstruction and scaled true sparsity losses for Pythia-70m over training epoch, over varying values of the & coefficient, Both metrics are averaged over all highly divergent layers, and hyperparameter choices otherwise as described in $3.2.
Model
Tied Weights
Sparsity Loss
Reconstruction Loss
pythia- 160m
truc
0.291
0.053
false
0.328
0.059
pythia-70m 70m
true
0.383
0,030
false
0.393
0.036
Table 11: Normalized reconstruction and scaled true sparsity losses for Pythia-70m and Pythia-160m over 1 training epoch, over differing choices of whether to tie encoder and de- coder weights. Both metrics are averaged over highly divergent layers, and hyperparameter choices are otherwise as described in $3.2.
Towards the end of our project, we ran a small experiment on Pythia-160m 160m and
Pythia-70m with alternating selecting layers for autoencoder extraction as lowest layers, vs the highest divergence layers. We found both the reconstruction loss and true sparsity loss be far less for the lower most layers. A future study examine the dictionary features extracted from these lowest layers would be interesting. See Table 12 for the observed metrics.
Model
Divergence Choice
Sparsity Loss
Reconstruction Loss
highest divergence
0.291
0.053
16thia-160m
lowest layers
0.166
0.023
highest divergence
0.388
0.036
pythia-70m
lowest layers
0.329
0.021
Table 12: Normalized reconstruction and scaled true sparsity losses for Pythia-70m and Pythia-160m over 1 training epoch, differing choices of divergence. Both metrics averaged over all highly divergent layers, and hyperparameter choices are otherwise described in $3.2.
19

---

G Small human study on automated feature explanation
In order to validate our GPT-4 generated neural feature explanations, we carried out a targeted human study. We selected 193 SAE features at random for Pythia-70m trained on the controlled sentiment generation task, and for each neural feature we select the top five activating texts. We then scrambled the GPT-4 generated explanations, and asked each annotator identify either the best explanation when presented with the GPT-4 auto-generated explanation alongwith three others randomly chosen from those for other neural features, or choose none of the above. We found that:
1. The two annotators chose the actual assigned GPT-4 explanation 67.5% and 70% of the time respectively, as compared 25% chance under random selection.
2. The two annotators also had 87. 87.5% inter annotator agreement rate.
3. The same two annotators selected "None of the Above" 20% and 22.5% of the time respec- tively.
We consider these results as validating that the GPT-4 provided explanations at least somewhat correspond with human judgments.
We carried this study only for human validation that setup was performing adequately. For deeper and more wide ranging human study of automated interpretability using LLMs, please refer to Rajamanoharan et [31, 32].
automated feature explanation
Two annotators each annotated 40 test questions as crafted above. The two annotators were joint authors of the work, and not hired contractors employees. Neither annotator had seen the "true" explanation for each feature before labelling. Since all of the examples were generated by publicly available Pythia- 70m trained on the controlled sentiment generation task, not tied to any confidential information or user sensitive data, and a fairly objective classification of feature relevance, do feel there any risks the annotators involved or any other third parties, and did not need IRB review. The exact prompt provided to the annotators is below:
More details:
Below is sequence of texts, alongwith the activations for neuron. Red being the darkest. Pick which of the explanations below is the best choice, or none of them above if none seem suitable.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We pose a hypothesis and defend it using our experiments, arguing that they show that the hypothesis correct.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations subsection in conclusion.
20

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. We give formal descriptions complete with details and complementary sections in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The source code for all experiments available in our supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ SguideesCooeSSumiiee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. //wips.ccppublll/// https//cpppbclcguueeeeeee http://oooSSSumssseeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We give the hyperparameters we used for all experiments as well as additional details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We give measure of statistical significance for our Kendall Tau divergence measures.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We state the amount of compute required to reproduce our experiments and train the models used for them
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tp:://ee.....eeeeee wwtrriss.ccppppeeee cc/public/////cccccccccccccllllt aeeeiic//hhiiihhi aewtiiisoueeeleeeeeee ines?
Answer:
[Yes]
Justification:
We have reviewed the code of ethics and find no violation in our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We give background on the risks related to models not internalizing human preferences, and our work aims to combat that, which we claim is beneficial for the safety deploying these models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

---

Answer: [NA]
Justification:
Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We cite the repositories papers for all models and datasets used in experiments.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcer thcode. com con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pane re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our GitHub README includes code examples for running experiments as well an explanation of the structure of the repository.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
25

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We include the full text of instructions. Since the two annotators were authors of the work as well, no compensation of contractors or employees was required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our paper discusses that given that no private data was included, and the nature of the labeling task as conducted by two of the authors, there is no risk human participants or need for IRB review.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "TWkTilF83J", "reviewer": "Reviewer_81zn", "summary": "The paper investigates how large language models (LLMs) learn preferences from human feedback during fine-tuning using reinforcement learning (RLHF). The authors introduce the concept of Learned Feedback Patterns (LFPs) to describe activation patterns in LLMs that align with human feedback. They aim to measure the accuracy of these patterns in capturing human preferences by training probes on condensed representations of LLM activations. The probes predict the implicit feedback signal in these activations and compare it to true feedback.", "questions": "- Can you elaborate on how your findings can be practically applied to mitigate risks associated with LLM deployment, such as manipulation of user preferences or harmful behaviors? What strategies would you recommend for developers to monitor and adjust LFPs in deployed models?\n\n- Your validation relies on GPT-4’s feature descriptions. Have you explored other methods or models for validating the identified features? How do you ensure the robustness of these validations?\n\n- Have you tested your method on other LLM architectures or tasks beyond sentiment analysis and toxicity detection? If so, what were the results? \tHow do you anticipate the effectiveness of your method would vary with different model sizes and types?", "limitations": "discussed", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction of LFPs provides a new perspective on understanding how LLMs learn from human feedback. This concept helps in quantifying and interpreting the alignment between LLM activations and human preferences.\n\n- The authors validate their probes by comparing neural features correlated with positive feedback against GPT-4’s descriptions of relevant features. This cross-validation strengthens the reliability of their findings.\n\n- The use of synthetic datasets to elicit specific activation patterns in LLMs adds to the reproducibility and robustness of the study. These datasets are also made publicly available for further research.", "weaknesses": "- The study primarily focuses on a few specific models (e.g., Pythia-70m, GPT-Neo-125m) and tasks (sentiment generation, toxicity), which might limit the generalizability of the findings across different LLMs and applications. More recently released models are of more value for studying RLHF patterns and verify that the method can be generalized. The patterns are easy to extract because that the used data are quite obvious to encode and decode. \n\n- While the probes show significant accuracy for certain tasks, the paper notes weaker correlations for more granular reward predictions, suggesting that the approach might struggle with highly detailed feedback signals. The issue of feature superposition in dense, high-dimensional activation spaces poses a challenge to fully interpreting the learned features. Although sparse autoencoders mitigate this to some extent, the problem remains a significant obstacle.\n\n- The validation process relies on GPT-4’s ability to describe neural features, which introduces a dependency on another model’s interpretability. This could introduce biases or inaccuracies if GPT-4’s descriptions are not perfectly reliable.\n\n- The paper acknowledges that while their method identifies features involved in feedback signals, it does not provide a mechanistic explanation of how these features interact or influence the expected feedback signal. This limits the depth of interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insight and time.\n\nWe appreciate that you found LFPs gave a new perspective on how LLMs learn from human feedback, and that our use of synthetic data and GPT-4 contributed well to the paper.\n\n## Model and Task Selection\n\n> \"The study primarily focuses on a few specific models… and tasks… More recently released models are of more value\"\n\nWe'd like to clarify and expand on our model and task selection:\n\n- In addition to the smaller models you mention (Pythia-70m, Pythia-160m and GPT-Neo-125m), we experimented with **Gemma-2b**, a larger and more recent model released in February 2024 (L152).\n- Beyond sentiment generation and toxicity tasks, we included the **helpful-harmless task**, designed to simulate real-world RLHF (L146-149).\n- Results for Gemma-2b and the helpful-harmless task can be found in **Table 3** of our paper.\n\nWe hope these results help address concerns about the generalizability of our findings to larger models and realistic applications.\n\nWe also studied models fine-tuned using different reinforcement learning algorithms (PPO and DPO), demonstrating that our method generalizes across RLHF algorithms.\n\n## Addressing Weaker Correlations and Feature Superposition\n\n> \"…the paper notes weaker correlations for more granular reward predictions… feature superposition… poses a challenge to fully interpreting… features.\"\n\nWe offer the following comments:\n\n1. Lower probe correlation in the controlled sentiment generation task may indicate that the model hasn't learned the granular specification of reward for that task, rather than a failure of the probe.\n2. Additional evidence in our global rebuttal shows:\n   - Probes learn patterns related to fine-tuning feedback even when they have low accuracy to it (Table 1 in global rebuttal PDF).\n   - Inputs can often be separated in terms of their probe classifications through dimensionality reduction (Figure 1 in global rebuttal PDF).\n   - Probe accuracy in predicting the label of a word in the VADER lexicon is slightly correlated with the frequency that the fine-tuned model generates that word (Figure 2 in global rebuttal PDF).\n\nOur method will benefit from recent improvements in training sparse autoencoders (e.g., [Gao et al.](https://arxiv.org/abs/2406.04093) and [Rajamanoharan et al.](https://arxiv.org/abs/2407.14435)), which should reduce the effects of superposition.\n\n## Validation Process and GPT-4 Reliability\n\n> \"The validation process relies on GPT-4's ability to describe neural features… This could introduce biases or inaccuracies if GPT-4's descriptions are not perfectly reliable.\"\n\nWe acknowledge this concern and note:\n\n- [Bills et al.](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html) and [Cunningham et al.](https://arxiv.org/abs/2309.08600) have conducted detailed validations of GPT-4 generated feature descriptions against manual and auomatic analysis.\n- Our method for generating feature descriptions with GPT-4 is common in prior literature (e.g., [Cunningham et al.](https://arxiv.org/abs/2309.08600), [Bricken et al.](https://transformer-circuits.pub/2023/monosemantic-features), [Neo et al.](https://arxiv.org/pdf/2402.15055), and [Templeton et al.](https://transformer-circuits.pub/2024/scaling-monosemanticity/)).\n\n## Practical Applications and Risk Mitigation\n\n> \"Can you elaborate on how your findings can be practically applied to mitigate risks associated with LLM deployment, such as manipulation of user preferences or harmful behaviors? What strategies would you recommend for developers to monitor and adjust LFPs in deployed models?\"\n\nOur method aims to help practitioners identify how well their fine-tuned models have learned the fine-tuning feedback. We recommend:\n\n- If probes indicate divergence between LFPs and fine-tuning feedback, that practitioners consider alternative post-training methods.\n- This divergence may suggest the model is learning a proxy for the feedback or failing to find consistencies in high-reward generations.\n\n## Method Generalization\n\n> \"Have you tested your method on other LLM architectures or tasks beyond sentiment analysis and toxicity detection? If so, what were the results? How do you anticipate the effectiveness of your method would vary with different model sizes and types?\"\n\nOur method has been tested on:\n\n- Tasks beyond toxicity and sentiment analysis, specifically a helpful-harmless task mimicking real-world RLHF (L146-149).\n- Various transformer-based LLMs with architectural differences (Pythia models, GPT-Neo-125m, and Gemma-2b), for example in the attention variant they use.\n\nWe anticipate our method would recover more information about fine-tuning feedback from larger models due to their increased parameter count and expected better performance on fine-tuning tasks.\n\n## Alternative Validation Methods\n\n> \"Your validation relies on GPT-4's feature descriptions. Have you explored other methods or models for validating the identified features? How do you ensure the robustness of these validations?\"\n\nWhile we haven't explored alternative models for generating feature descriptions:\n\n- [Bricken et al.](https://transformer-circuits.pub/2023/monosemantic-features) and [Templeton et al.](https://transformer-circuits.pub/2024/scaling-monosemanticity/) have successfully used Anthropic models to explain features.\n- We don't foresee issues with using models other than GPT-4.\n- Even if GPT-4 descriptions are sometimes inaccurate, the overlap between probes and GPT-4 descriptions still aids in probe validation.\n\nThe results from our global rebuttal PDF will be incorporated into the camera-ready version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NDXoPOIkWc", "reviewer": "Reviewer_dzK7", "summary": "The authors propose an approach for measuring and interpreting the divergence between learned feedback patterns (LFPs, or simply the model's activation patterns) and the feedback reward distribution of the preference training dataset. To do so, they identify layers whose activations have moved the most during RLHF training and input these layers' activations into a sparse auto-encoder (SAE) that is trained to provide sparse representations of the LLM's activations. Then, they train probes to predict the feedback signal (e.g. reward, sentiment label) from the SAE's outputs. They use these probes both to measure the divergence of the LFPs from the actual feedback signals and to interpret which features are most important for the LFPs.", "questions": "Questions are above.", "limitations": "The limitations section was well-written and thorough, and covered many of the concerns I had myself. An additional limitation is that this method is computationally expensive and requires both training another model and running inference on a sufficiently powerful LLM (e.g. GPT-4) to interpret the features. In this paper, most of the results were for smaller models (under 1B params), and it is unclear whether this method would be scalable to larger models.", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- The authors ask an interesting question of whether we can measure and interpret the difference between a trained model's activation patterns and the preference distribution it has been trained on. The interpretability aspect of this question is interesting, since it can help us better understand what exactly the model has learned (or not learned) from its training dataset.\n- The authors provide a good explanation of why sparse auto-encoders are being used for this task (rather than interpreting the raw model activations), as well as the limitations thereof.", "weaknesses": "- The effectiveness of this probing method seems to rely on many key assumptions being true, such as (i) sparse autoencoder outputs being more interpretable than the original model's outputs, (ii) sparse autoencoder output representations being faithful to the original model's representations, (iii) the probes being accurate, and (iv) GPT-4 being accurate/faithful when giving descriptions of each feature. There is very little experimental evidence provided for confirming that any of these assumptions are true, and these claims are difficult to test in the first place.\n   - In fact, the authors mention that a likely reason for the low correlation between the probe's predictions and the VADER lexicon (for some models) is \"the complexity of the probe's task...a linear regression model is unlikely to recover such granular rewards accurately from just the activations\" (L265-266). Although they do find a high correlation for one model, the insufficiency of this probe implies that it is not effective for accurately measuring the divergence between the model's activation patterns and the feedback label distribution. If the correlation is low, we cannot tell whether that is the probe's failure, or if the model has not  acquired strong LFPs, or some combination of the two. Since this probing technique is a central contribution of the paper, I would expect stronger probes and more rigorous evaluation of the effectiveness of the probes.\n   - How can one ensure that GPT-4's interpretations of the features are accurate or faithful?\n- Table 5 purports to check whether the predicted LFP-related features were actually important and useful to the LLM, but the numbers before and after ablation are often very close together (or identical, in the case of GPT-Neo-125m). It would be helpful to report confidence intervals or standard errors to check whether these differences are significant. But as it currently stands, this table's results does not seem to strongly support the claim that the predicted LFP-related features are indeed relevant to/critical for LFPs.\n\n- Lack of clarity in explaining methods:\n    - Much of the writing about the methods is unclear, contradictory, or omits many details. For instance, the explanation of the logistic regression probe in L233-234 says \"we label the concatenated activations as positive or negative based on the averaged activation deltas for each token over the entire input sequence, and train a logistic regression model to classify the activations,\" which would suggest that this probe's inputs are the activations. But L493 (in the appendix) says \"...we give a positive or negative label to concatenated autoencoder outputs based on the sign of their activation delta. We then train a logistic regression model to predict the labels from the concatenated autoencoder outputs,\" which suggests that the inputs are actually the autoencoder outputs, not the original model's activations. Which is it?\n   - In Section 3.4, how is GPT-4 prompted to provide the explanations?\n   - Given how confusing and verbose the methodology is, I would encourage the authors to write out some of the procedures in equation form, rather than long paragraphs of text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review.\n\nWe are pleased that you found the question our paper studies interesting, and our explanation for using sparse autoencoders good.\n\n## Key Assumptions\n\n> \"The effectiveness of this probing method seems to rely on many key assumptions being true…\"\n\nWhile we agree with the reviewer that the method relies on many key assumptions, we think that these assumptions are reasonable and are based on prior work:\n\n### 1. Sparse autoencoder outputs being more interpretable\n\nSeveral studies support this assumption:\n\n- [Cunningham et al.](https://arxiv.org/abs/2309.08600), [Bricken et al.](https://transformer-circuits.pub/2023/monosemantic-features), and [Templeton et al.](https://transformer-circuits.pub/2024/scaling-monosemanticity/) show that features in sparse autoencoder dictionaries are easier for language models to describe than neurons in raw activations.\n- Bricken et al. and Templeton et al. found that these features are easier for humans to understand through manual analysis.\n- Bricken et al. also show that sparse autoencoders enable finding effectively invisible features in raw activations.\n- Additional work ([Rajamanoharan et al.](https://arxiv.org/pdf/2404.16014), [Gao et al.](https://arxiv.org/abs/2406.04093), and [Rajamanoharan et al.](https://arxiv.org/abs/2407.14435)) shows that sparse autoencoders that perform better on key metrics (sparsity, reconstruction) are more interpretable.\n\nAfter papers such as Cunningham et al. and Bricken et al., this assumption is common in interpretability, and motivates the use of sparse autoencoders for interpretability.\n\n### 2. Sparse autoencoder output representations being faithful to the original representations\n\n- Bricken et al. show that replacing MLP activations with sparse autoencoder outputs incurred only 21% of the loss that would be incurred by zero ablating the MLP.\n- Rajamanoharan et al. reduce this to 2% with their improved sparse autoencoder.\n\nWe consider these results to show faithfulness of sparse autoencoder outputs to activations.\n\nWe argue that this assumption is in line with other interpretability work, and that experimentally supporting it in our paper is out of scope.\n\n### 3. Probe accuracy\n\nWe agree there is more work to be done in validating our probes. In our submission, we validated our probes using GPT-4 feature descriptions, finding a strong overlap between the probes and GPT-4 descriptions (Table 6). We offer additional validation of our probes in the PDF in our global rebuttal:\n\n- We show that the probes are learning patterns related to the fine-tuning feedback (Table 1 in global rebuttal).\n- We demonstrate that inputs can be separated by their reward using dimensionality reduction, showing there is structure in the probes' input data tht they can exploit (Figure 1 in global rebuttal).\n- Probe accuracy in predicting the label of a word in the VADER lexicon is slightly correlated with the frequency that the fine-tuned model generates that word (Figure 2 in global rebuttal).\n\nWe commit to integrating these results into the camera-ready version of our paper to better support our probes.\n\n### 4. GPT-4 prediction accuracy\n\n[Bills et al.](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html) and Cunningham et al. have performed detailed validations of GPT-4 generated feature descriptions:\n\n- They prompt another LLM to predict how that feature would activate for a collection of tokens if that description were correct.\n- They measure the correlation of these predictions and the true activations.\n- Bills et al. also conduct additional validation, using humans to validate the feature descriptions and ablation experiments.\n\nWe would like to highlight that our method of generating feature descriptions with GPT-4 is common in prior literature, such as Cunningham et al., Bricken et al., [Neo et al.](https://arxiv.org/pdf/2402.15055) and Templeton et al. We argue that even if the GPT-4 feature descriptions are sometimes inaccurate, the overlap between the GPT-4 descriptions and probes still helps to validate the probes.\n\n## Addressing Specific Concerns\n\n> \"If the correlation is low, we cannot tell whether that is the probe's failure…\"\n\nWe hope that through some of our additional results, we have increased the rigor of our evaluation of the effectiveness of the probes. We acknowledge that additional validation would be useful, but believe that the rigor of evaluation with our updated results is in line with the standards of interpretability literature. Further improving the validation of our probes is our primary objective for the camera-ready version of our paper.\n\n> \"as it currently stands, this table's [Table 5] results does not seem to strongly support… that the predicted LFP-related features are relevant to LFPs.\"\n\nWe note that Table 5 was an ablation of the features classified by GPT-4 as related to LFPs and was not intended to support our probes. \n\n> \"Much of the writing about the methods is unclear, contradictory, or omits many details…\"\n\nIn an effort to address this concern, we have corrected the issues you highlighted, clarifying that the description on L233-234 is correct, and not L493. The camera-ready version of our paper will include a clearer method section, with more of the writing replaced with equations, in line with your comment.\n\n> \"In Section 3.4, how is GPT-4 prompted...?\"\n\nThe camera-ready version of our paper will include an appendix with the full prompts used to generate feature explanations, but they are taken from the [public repository](https://github.com/openai/automated-interpretability) of Bills et al.\n\n> \"In this paper, most of the results were for smaller models (under 1B params)... it is unclear whether this method would be scalable to larger models.\"\n\nWe included results for Gemma-2b in our paper (L152), and note that recent work such as Gao et al. has trained sparse autoencoders on models in the GPT-4 family with success, showing scalability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lJhkBFBGmy", "reviewer": "Reviewer_d8nm", "summary": "The goal of this paper is to predict where patterns in LLM activations learned from RLHF diverge from the human preferences used for the RLHF training. \nGiven a base model and an RLHF tuned version of it, the method involves first identifying the 5 layers with highest parameter difference according to an L2 norm. Then two auto-encoders are trained over the activations from these layers. The encoder and decoder weights of the autoencoder are tied, and the output from these is preferred for studying the activations as they are expected to be more sparse, condensed and interpretable than the raw activations.\nAt inference time, for each input, the activations from the high divergence layers are computed, passed through the autoencoder and then aggregated. Given a pair of contrasting inputs, a linear probe is trained to predict activation deltas using the above aggregated autoencoder output as input. The output of the probe is meant to be a predicted feedback signal that can be compared to the ground truth fine tuning feedback. For sentiment analysis, a strong correlation is observed with the Pythia-160m model but this is weaker for Pythia-70m and GPT-Neo-125m.  \n\nFor another validation the probes, GPT-4 is used to generate explanations of the features in the decoder weights of the autoencoders that get activated when the predicted feedback is positive. GPT4 is then prompted to predict whether or not these are relevant to the fine tuning task, based on a language description of the task. It is found that a feature identified by GPT-4 as relevant to the fine-tuning task is between twice and thrice as likely to be correlated with predicted positive feedback.", "questions": "1. I don't feel like I understood the concept of the activation delta. The paper states \"We compute the activation deltas for a given contrastive triple as the difference between the positive 212 and neutral element and negative and neutral element under the ℓ2 norm\". For any input x, there is a set of values $\\hat{a}$. For calculating an L2 norm these still need to somehow be aggregated. Since the probe input $A_{concat}(x)$ is already a concatenation of these values, I assume it is not simply an L2 norm of the two $A_{concat}$ vectors, as then it is unclear what the probe would learn.  \n\n2. Assuming that the probes trained in this paper do obtain information about the preferences underlying human preference data, how do we make use of that information?", "limitations": "The paper has discussed limitations but not broader impact of their method.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is quite accessible for a reader whose area of focus is not interpretability.", "weaknesses": "As a reviewer not particularly experienced with work on interpretability, the takeaways of this paper are somewhat unclear. For example, if we finetuned a new model on one of the datasets used in this paper and trained probes in a similar way from its activations, what would that tell us about the the difference between the base and RLHF versions of that model? Alternately, is the goal to discover information about a model where the base and RLHF-tuned versions are available but the data is not, and hence we do not know what factors might have influenced the preference annotations that guided the annotation. \n\nI did not fully understand how the activation deltas are calculated. While most of the paper is fairly readable to a reviewer with a different area of focus, this aspect could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and appreciate that you found our paper accessible.\n\n## Clarifying the Paper's Objectives and Takeaways\n\n> \"…the takeaways of this paper are somewhat unclear. … if we finetuned a new model on one of the datasets used in this paper and trained probes in a similar way from its activations, what would that tell us about the the difference between the base and RLHF versions of that model? Alternately, is the goal to discover information about a model where the base and RLHF-tuned versions are available but the data is not, and hence we do not know what factors might have influenced the preference annotations that guided the annotation.\"\n\nOur method's primary objective is to **measure the extent to which a fine-tuned model has learned the fine-tuning feedback**. To clarify:\n\n- We do not intend to contrast base and fine-tuned models (although Appendix C briefly investigates this).\n- Our goal is not necessarily to find information about fine-tuned models where the fine-tuning data/reward model is unknown.\n\nWe hope practitioners will use our method to evaluate and improve their post-training techniques based on how well they cause models to learn the fine-tuning feedback.\n\nKey findings and implications:\n\n1. Our method considers model internals, unlike other metrics for the success of RLHF (e.g., fine-tuning loss or output evaluations).\n2. While our probes can accurately recover whether feedback is positive or negative (Table 3), they struggle with more granular feedback used in PPO fine-tuning (Table 4).\n3. This could indicate either:\n   a) The model hasn't learned fine-tuning feedback in sufficient detail, or\n   b) Low probe accuracy (which we argue against, see below)\n\nWe support the accuracy of our probes through:\n\n- Showing similarity between probe-identified and GPT-4-identified LFP-related features (Table 6)\n- New results in our global rebuttal PDF:\n  - Probes less correlated with fine-tuning feedback still learn related patterns (Table 1 in the PDF attached to our global rebuttal)\n  - Inputs can often be separated by reward through dimensionality reduction (Figure 1 in the PDF attached to our global rebuttal)\n  - Probe accuracy in predicting the label of a word in the VADER lexicon is correlated with the frequency that the fine-tuned model generates that word (Figure 2 in the PDF attached to our global rebuttal).\n\nThese results suggest that fine-tuned models may not have learned fine-tuning feedback in a detailed manner, rather than indicating probe inaccuracy.\n\n## Clarification on Activation Delta Calculation\n\n> \"I did not fully understand how the activation deltas are calculated… I assume it is not simply an L2 norm of the two A_concat vectors, as then it is unclear what the probe would learn.\"\n\nThe activation deltas are calculated as follows:\n\nFor any contrastive triple (positive, negative, neutral), we provide two pairs of input to the linear probe:\n\na) Input_1 = SAE_repr of (positive)\n\n   Output_1 = $+ ||SAE\\_{repr}(positive) - SAE\\_{repr}(neutral)||$\n\nb) Input_2 = SAE_repr of (negative)\n\n   Output_2 = $-1 \\cdot ||SAE\\_{repr}(negative) - SAE\\_{repr}(neutral)||$\n\nThe L2 norm here serves as a signed distance measure between either the positive and neutral or negative and neutral pairs.\n\nAlthough it might be unclear what the probe would learn if trained on a smaller number of examples, the only consistent difference over many examples should be the distance in activation space related to how different the feedback is in fine-tuning. For example, although a given positive and neutral pair of inputs may be separated in activation space in many ways other than just their implicit feedback, the only consistent difference over thousands of positive and neutral pairs should be the difference in their implicit feedback. In the PDF attached to the rebuttal visible to all viewers we show through dimensionality reduction that the separation between positive and negative activation deltas can be seen visually (Figure 1), which we hope shows that activation deltas are sufficient labels for probe training datasets.\n\n## Utilization of Probe-Obtained Information\n\n> \"Assuming that the probes trained in this paper do obtain information about the preferences underlying human preference data, how do we make use of that information?\"\n\nIf probes are accurate but unable to learn fine-tuning feedback precisely, it suggests the fine-tuned model hasn't fully learned the feedback. Practitioners might then:\n\n1. Consider alternative explanations, such as:\n   - The model learning a proxy objective\n   - The model failing to find consistent patterns in its activations related to the feedback\n2. Take remedial actions, such as further fine-tuning, or adjusting their post-training\n\n## Addressing Broader Impact\n\n> \"The paper has discussed limitations but not broader impact of their method.\"\n\nWe will add a broader impacts section to the camera-ready version of the paper, covering the takeaways stated in this rebuttal and our global rebuttal.\n\nAll results from the PDF attached to the global rebuttal will be incorporated into the camera-ready version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNr1JZHHQn", "reviewer": "Reviewer_zoqc", "summary": "This submission tries to tackle one big question in the field of interpreting the data-driven preference learnt by RLHF in human language. The technical path this submission took is to train probe on SAE features to distinguish between good and bad RLHF features.", "questions": "+ I wonder if it could be interesting to conduct the same analysis on the reward model, if the reward model is another language model that is open-weight and trained. Can we compare the two representation space, one trained for discrimination and the other trained for generation?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "+ The attempt to interpret what happens during RLHF training is a good direction to pursue.\n+ Releasing the SAE direction and training code could be an excellent news to the community.", "weaknesses": "+ Unclear why have to probe on top of SAE feature. SAE greatly increase the dimensions of the features, leading to overfitting---you can find a separating plane for whatever classification task in this high-D space. Lacking comparison to normal probing. \n+ Considering the problem from a dynamical perspective can be fruitful. Noted that the authors did ablate the features and observe a performance drop on preference dataset. But it's also interesting to see the progress of RLHF training, how it warps the features spaces, even the SAE features' relative importances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review.\n\nWe are pleased that you found our research direction good, and that releasing our SAE infrastructure would be beneficial to the community.\n\n## Clarification on SAE Feature Probing\n\n> \"Unclear why have to probe on top of SAE feature. SAE greatly increase the dimensions of the features, leading to overfitting… Lacking comparison to normal probing.\"\n\nWe want to emphasize several key points:\n\n1. Our SAEs do not modify the dimensions of the activations. The hidden size is the same value as the input size.\n2. The accuracy of the probes we report in our paper is the accuracy on an unseen test dataset.\n3. Based on your response, we have included a comparison to probing on the raw activations in the PDF attached to the rebuttal visible to all reviewers (Table 2).\n\nOur findings show that probing on the SAE outputs does not meaningfully affect the accuracy of the probes. It also provides the benefits of the inputs being probed being more interpretable ([Cunningham et al.](https://arxiv.org/abs/2309.08600), [Bricken et al.](https://transformer-circuits.pub/2023/monosemantic-features))\n\n## Dynamical Perspective on RLHF Training\n\n> \"Considering the problem from a dynamical perspective can be fruitful… it's also interesting to see the progress of RLHF training, how it warps the features spaces, even the SAE features' relative importances.\"\n\nWe agree that studying the problem from a dynamical perspective would be very interesting. In response:\n\n1. We will mention this as an exciting direction for future work in the camera-ready version of the paper.\n2. We aim to include results of our method at various checkpoints throughout fine-tuning in the camera-ready version of our paper.\n\n## Analysis of Reward Models\n\n> \"I wonder if it could be interesting to conduct the same analysis on the reward model… Can we compare the two representation space…?\"\n\nWe agree this is another interesting direction:\n\n1. We will include this in our discussion of future work.\n2. [Riggs et al.](https://www.lesswrong.com/posts/5XmxmszdjzBQzqpmz/interpreting-preference-models-w-sparse-autoencoders) recently trained some SAEs on preference models with good results, suggesting this approach is feasible.\n\nChallenges and potential solutions:\n- Our fine-tuning tasks don't have true reward models (VADER uses lexicon labels, helpful-harmless and toxicity tasks use pairwise preference data).\n- If this is a significant concern, we could train classification models for the pairwise preference data and compare the representation space of this classification model with our probes/sparse autoencoders.\n\nThe results in the PDF attached to the rebuttal visible to all reviewers will be incorporated into the camera-ready version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

We thank the reviewers for their incisive feedback. In this comment, we summarize the additional results in the PDF attached to this comment, points made across multiple reviews, and our responses to those points. Note that unless otherwise specified, the figures referred to in this rebuttal are in the attached PDF, and not our paper.

* **Takeaways and broader impacts:** In this paper, we aim to measure the accuracy of a fine-tuned LLM’s learned feedback patterns to the fine-tuning feedback. We hope that practitioners will evaluate their fine-tuned models using a method similar to ours, and adjust their fine-tuning accordingly. We argue that our results for the VADER/controlled sentiment generation task show that the fine-tuned models likely did not learn the granular fine-tuning feedback, and merely learn to discriminate the more strongly positive/negative examples. To make this clearer, the camera-ready version of our paper will include a broader impacts section, and more explicit mentions of these takeaways in our conclusion and introduction.
* **Validation of probes:** Our submission validated the probes we trained using descriptions of features generated by GPT-4 (Table 6 in our original paper). Some reviewers were concerned that the probes may not have been trained well, leading to low probe accuracy to the fine-tuning feedback. Since then, we have added additional validation as suggested by the reviewers, such as showing that positive/negative inputs to the probes (in terms of the probes’ predictions) can be discerned through dimensionality reduction (Figure 1), suggesting that there is structure in the data for probes to exploit. We also show that that the low accuracy of the probe on specific words in the VADER lexicon is anticorrelated with the frequency the fine-tuned model outputs those words (Figure 2). We believe these results help support the hypothesis that the low accuracy of the VADER probes to the fine-tuning feedback is caused by the fine-tuned models failing to learn the granular fine-tuning feedback, and not due to poor probe accuracy. Table 3 in our paper might support this hypothesis, as it shows that when probes are trained simply to classify their inputs, they achieve very high accuracy. The camera-ready version of our paper will include these additional results to support the accuracy of our probes.
* **Accuracy of the VADER/controlled sentiment generation probes:** Although our VADER probes achieved lower accuracy to the fine-tuning feedback than the toxicity or helpful-harmless probes, we argue that this can be explained by the LLM failing to learn the granular fine-tuning feedback. We support this claim by showing that the VADER probes achieve higher accuracy when only the sign of their prediction is considered (Table 1), showing that they do find structure in their training training data, and with Figure 2, described in the previous dot point. We commit to more strongly validating our VADER probes in the camera-ready version of our paper, and believe our additional results show significant progress toward this.
* **Validation of GPT-4 feature descriptions:** Reviewers were concerned that GPT-4 may not reliably describe features. We believe that prior work supports the accuracy of these descriptions: [Bills et al.](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html) performed a thorough validation of neuron descriptions generated using GPT-4, showing that the descriptions were accurate on multiple metrics when explaining features in GPT-2. [Cunningham et al.](https://arxiv.org/abs/2309.08600) found similar results, but with sparse autoencoder features instead of neurons in an LLM. Our feature explanation method is taken directly from these works. [Bricken et al.](https://transformer-circuits.pub/2023/monosemantic-features) found similar results using Claude 2. The specific prompts we used will be included in an appendix in the camera-ready version of our paper, but they are taken from [Bills et al.](https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html), and their public [GitHub repository](https://github.com/openai/automated-interpretability). We also argue that an imperfect validation would still be valuable: Even if the GPT-4 feature descriptions are sometimes inaccurate, the overlap between the GPT-4 descriptions and probes still helps to validate the probes.

Meta-review: This paper analyses patterns of learning during RLHF via probing  to predict feedback signals from LLM activations. This approach is generally compelling and well-motivated, though reviewers point out that GPT-4 explanation of identified feature sets may not be a reliable way to validate the method. To address this, I would suggest collecting human explanations of a subset of the features for comparison with GPT-4, similar to Bills et al, although there are still concerns e.g. about polysemanticity of identified features. Reviewer dzK7 points out that the exact details of the proposed probe are not well-motivated, including the choice of SAE.
Decision: Accept (poster)