Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models
Authors: Adam Karvonen
Benjamin Wright
Can Rager
Rico Angell
Jannik Brinkmann
Logan Riggs Smith
Claudio Mayrink Verdun
David Bau
Samuel Marks
Venue: NeurIPS 2024 poster (2024)
Keywords: Language models
interpretability
dictionary learning
Primary area: interpretability_and_explainability
TL;DR: We measure progress in training sparse autoencoders for LM interpretability by working in the setting of LMs trained on chess and Othello.

Abstract: What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs is difficult because we lack a ground-truth collection of interpretable features which we expect good SAEs to identify. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on Chess and Othello transcripts. These settings carry natural collections of interpretable features—for example, “there is a knight on F3”—which we leverage into metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, $p$-annealing, which demonstrates improved performance on our metric.

Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models
Adam Karvonen* Independent
Benjamin Wright MIT
Can Rager Independent
Rico Angell UMass, Amherst
Jannik Brinkmann University of Mannheim
Logan Smith Independent
Claudio Mayrink Verdun Harvard University
David Bau Northeastern University
Samuel Marks Northeastern University
Abstract
What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs difficult because we lack ground-truth collection of interpretable features that expect good SAFs to recover. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on chess and Othello transcripts. These settings carry natural collections of interpretable features-for example, "there a knight on F3"- which we leverage into supervised metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, p-annealing, which improves performance prior unsupervised metrics as well as our new metrics
Introduction
Mechanistic interpretability aims to reverse engineer neural networks into human-understandabll components. What, however, should these components be? Recent work has applied Sparse Au- toencoders (SAEs) 9 [6], scalable unsupervised learning method inspired by sparse dictionary learning to find a disentangled representation of language model (LM) internals. However, measuring progress training SAEs is challenging because we do not know what gold-standard dictionary would look like, as it is difficult to anticipate which ground-truth features underlie model cognition. Prior work has either attempted to measure SAE quality in toy synthetic settings [57] relied on various proxies such as sparsity, fidelity of the reconstruction, and LM-assisted autointerpretability
In this work, we explore a setting that lies between toy synthetic data (where all ground -tunt features known; ef. Elhage al. (24]) and natural language: LMs trained board game transcripts. This setting allows us to formally specify natural categories of interpretable features, e.g., "there is a knight e3" "the bishop f5 is pinned." We leverage this introduce two novel metrics for how much of a model's 's knowledge an SAEs has captured:
Equal contribution. Correspondence to: Adam Karvonen <adam. karvonen coaiel@@maloooom gmail.com>.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Board reconstruction. Can we reconstruct the state of the game board by interpreting each feature classifier for some board configuration?
Coverage. Out of catalog of researcher- -wauucher--peciieeed candidate features, how many of these candidate features actually appear in the SAE?
These metrics carry the limitation that they are sensitive to researcher preconceptions. Nevertheless, we show that they provide a useful new signal of SAE quality.
Additionally, we introduce p-annealing, novel technique for training SAEs. When training an SAE with p-annealing, we use an Ly-norm-bast based sparsity penalty with p ranging from p the beginning of training (corresponding to a convex minimization problem) to some < 1 (a non-convex objective) by the end of training. We demonstrate that p-annealing improves over prior methods, giving performance on par with the more compute- conpute-intensive intensive Gated SAEs from Rajamanoharan et al. [54] as measured both by prior metrics and our new metrics.
Overall, our main contributions are as follows:
1. We train and open-source 500 SAEs trained chess and Othello models each.
2. We introduce two new metrics for measuring the quality of SAES.
3. We introduce p-annealing, a novel technique for training SAEs that improves on prior techniques.
2 Background
2.1 Language models for Othello and chess
In this work, we make use of LMs trained to autoregressively predict transcripts of chess and Othello games. We emphasize that these transcripts only give lists of moves standard notation and do not directly expose the board state. Based on behavioral evidence (the high accuracy of the LMs for predicting legal moves) and prior studies of LM representations [36] 47 33 we infer that the LMs internally model the board state, making them a good testbed setting for studying LM representations.
Othello. Othello two-player strategy board game played on an 8x8 grid, with players using black and white discs. Players take turns placing dises on the board, capturing their opponent's discs by bracketing them between their own, causing the captured dises to turn their color. The goal is to have more discs turned to display your color at the end of the game. The game ends if every square on the board is covered or either player cannot make move.
In our experiments, we use an 8-layer GPT model with 8 attention heads and a n = 512 dimensional hidden space, provided by Li al. (36). This model had no prior knowledge of the game or its rules and was trained from scratch on 20 million game transcripts, where each token in the corpus represents a tile on which players place their discs. The game transcripts were synthetically generated by uniformly sampling from the Othello game tree. Thus, the data distribution captures valid move sequences rather than strategic depth. For this model, Li al. (6 demonstrated the emergence of world model-an internal representation of the correct board state allowing it to predict the next move- ttht can be extracted from the model activations using a non-linear probe. Nanda et [47] extended this finding, showing that similar internal representation could be extracted using linear probes, supporting the linear representation hypothesis [46).
Chess. Othello makes natural testbed for studying emergent internal representations since the game tree is far too large memorize. However, the rules and state are not particularly complex. Therefore, we also consider a language model trained on chess game transcripts with identical architecture, provided by Karvonen 333. The model again had no prior knowledge of chess and was trained from scratch on 16 million human games from the Lichess chess game database [38]. The input to the model is a string in the Portable Game Notation (PGN) format (c.g., "1, e4 e5 2. Nf3 "). The model predicts a legal move in 99.8 % of cases and, similar to Othello, it has an internal representation of the board state that can be extracted from the internal activations using a linear probe 33
Code, models, and data available https: / httpp/://ggthh....////aaaamaaaaa http://ghhhh..oooomaaaaavvvaae// |Sara//eAA_aaaaaeeeeaaaalllll

---

A
A
A
MMG
Figure We find SAE features that detect interpretable board state properties (BSP) with high precision (i.e., above 0.95). This figure illustrates three distinct chessboard states, each an example of a BSP associated with a high activation of a particular SAE feature. Left: board state detector identifies knight square f3, owned by the player move. Middle: A rook threat detector indicates an immediate threat posed by a rook to a queen regardless of location and piece threatened. Right: A pin detector recognizes moves that resolve check diagonal by creating pin, again, regardless location and piece pinned.
2.2 Sparse autoencoders
Given dataset D of vectors E Rdd sparse autoencoder (SAE) is trained to produce approxi- mation
dsAE
Σ (x)xd; +b
as sparse linear combination of features. Here, the feature vectors d, E Fd are unit vectors, the feature activations fx( are sparse set of coefficients. and IR is a bias term. Concretely, an SAE is a neural network with an encoder-decoder architecture, where the encoder maps x to the fdses (x)] of feature activations, and the decoder maps f an approximate
vector f (x)
reconstruction of x.
In this paper, we train SAES on datasets consisting of activations extracted from the residual stream after the sixth layer for both the chess and Othello models. At these layers, linear probes trained with logistic regression
accurate for classifying variety of properties of the game board [33 47). For training SAEs, we employ a variety of SAE architectures and training algorithms, as detailed in Section
3 Measuring autoencoder quality for chess and Othello models
Many of the features learned by our SAEs reflect uninteresting, surface-level properties of the input, such the presence of certain tokens. However, upon inspection, additionally find many SAE features which seem to reflect latent model of the board state, e.g., features that reflect the presence of certain pieces on particular squares, squares that legal play on, and stratany-relevant properties like the presence of a pin in chess (Figures (6)
Fortunately, in the setting of board games, can formally specify certain classes of these interesting features, allowing us to more rigorously detect them and use them to understand our SAEs. In Section 3.1 we specify certain classes of interesting game board properties. Then, in Section leverage these classes into two metrics of SAE quality.
3.1 Board state properties chess and Othello models
We formalize a board state property (BSP) to be a function g: {game board) - (0, 1). In this work, we will consider the following interpretable classes of BSPs:
Groard state contains BSPs which classify the presence of a piece at specific board square, where the board consists of 8 x 8 squares in both games. For chess, consider the full board for the twelve

---

distinct piece types (white king, white queen, black king), giving a total of 8 x 8 x 12 BSPs. For Othello, we consider the full board for the two distinct piece types (black and white), yielding
8 x 8 x 2 BSPs.
Sstratogy consists of BSPs relevant for predicting legal moves and playing strategically chess, such pin detector. They were selected by the authors based domain knowledge and prior interpretability work on the chess model AlphaZero [45]. We provide a full list of strategy BSPs in Table 3 in the Appendix. Because Othello model was trained to play random legal moves, do not consider strategy BSPs for Othello.
3.2 Measuring SAE quality with board state properties
In this section, we introduce two metrics of SAE quality: coverage and board reconstruction.
Coverage. Given collection G of BSPs, coverage metric quantifies the extent to which an SAE has identified features that coincide with the BSPs in G. In more detail, suppose that fi is an SAE feature and & [0, 1] is threshold, define the function
(x) T [f((x) > 1. f(
where f.nax is (an empirical estimate of) maxx-D fx(), the maximum value that Si takes over the dataset DD of activations extracted from our model, and TI is the indicator function. We interpret ,,,,t as a binary classifier; intuitively, it corresponds to binarizing the activations of fa into "on" vs. "off" some fraction t of the maximum value of ft on D. Given some BSP g E G, let F1 (of..: E [0, 1] denote the Fl-score for f,, classifying g. Then we define the coverage of an SAE with features {ff)) relative set of BSPs G to be
Cov (O.),))
max max Fi Fi(Oratig).
9
S.
9E0
In other words, we take, for each g G, the Fi-score of the feature that best serves classifier for g. and then take the mean of these maximal Fi Fi-scores. An SAE receives a coverage score of 1 if, for each BSP g E G, it has some feature that is a perfect classifier for g. Since Cov depends on the choice of threshold I, we sweep over I. E 00, ......................... 0.9) and take the best coverage score; typically this best is in (0.0.10..2).
Board reconstruction. Again, let G be a set of BSPs. Intuitively, the idea of our board reconstruc- tion metric is that, for sufficiently good SAE, there should be a simple, human-interpretable way to recover the state of the board from the profile of feature activations {f( on an activation x E Rd Here, the activation
extracted after the post-MLP residual connection in layer 6.
We will base our board reconstruction metric around the following human-interpredte interpretable way of recovering board state from feature activation profile; we emphasize that different ways of recovering boards from feature activations may lead to qualitatively different results. This recovery rule is based on the assumption that interpretable SAE features tend to be high precision for some subset of BSPs, in line with Templeton et al. 58 For example, features that classify common configurations of pieces are high precision (but not necessarily high recall) for multiple BSPs. We use a consistent dataset of 1000 games as our training set Dunin for identifying high-precision features across all Board State Properties (BSPs). An additional, separate set of 1000 games serves as our test set Diess Using the training set Derain we identify, for each SAE feature ft. all of the BSPs g E & for which H is high precision (of at least 0.95) classifier. Then for each g E G our prediction rule is
1, if fxx+ 1 for any St which high precision for g on D 0, otherwise.
Po((f(())))
(4)
Let ([(P(()())))::) denote the Fi-score for a given board state b, where ((ff(x))) (Tf((()))) geg represents the full predieted board (containing predictions for all 64 squares) obtained from the SAE activations
We do not score empty squares. Thus, the reconstruction score would be zero if no high precision features are active.

---

Then, the average Fi-se -ccore over all board states in the test dataset Diest can be calculated as
(ec((2x), Dwes)
Σ ( (Ix
(5)
[Deatl
4 Training methodologies for SAES
In our experiments, we investigate four methods for training SAEs, as explained in this sec- tion. These are given by autoencoder architectures and two training methodologies- with p-annealing and one without p-annealing for each architecture. Our SAES are available chap:///ss..acoommaaaaaaarrrnena sateeesssssssteeeeeeeeeaeeeeee (chess) and https:// huggingface..o co/ adamkarvonen wtweeloosaeeeeeeeeeeeeeeeeeeseet tttees//reeemmmam (Othello).
4.1 Standard SAEs
Let be the dimension of the model's residual stream activations that are input to the autoencoder, the autoencoder hidden dimension, and s the dataset size. Our baseline "standard" SAE architecture, introduced in Bricken al. [9] is defined by encoder weights We Rmxn, decoder weights Wa E R=xm with columns constrained to have Ln-norm of 1, and biases b. E R", by E R". Given an imput R", the SAE computes
f(x) = ReLU - ba) + b.)
(6)
X Wa fx ba
(7)
where f(x) is the vector of feature activations, and the reconstruction. For a standard SAE, our baseline training method is implemented in the open-source dictionary_leanttn repository
[33. optimizing the loss
Lstandard Ex [|x 8112 A1I((x)]11
(8)
for some hyperparameter À > controlling sparsity.
4.2
Gated SAFs
The I-1 penalty used in the original training method encourages feature activations to be smaller than they would be for optimal reconstruction [62]. To address this, Rajamanoharan et al. (54) introduced modification to the original SAE architecture that separates the selection of dictionary elements use in reconstruction and estimating the coefficients of these dictionary elements. This results in the following gated architecture:
Tgate (x) = W Wanee( (x - ba) + bgate
f(x) = [TTate (x) 0] 3 ReLU(W mag (x ba) + bmag)
(10)
a ((((x) Waf(x) + ba
(11)
where 1[. > is the Heaviside step function and @ denotes elementwise multiplication. Then, the loss function uses frozens a frozen copy of the decoder:
Egated = ExwDi
||x +(f(x))
+ A X||ReLU( (TgaL (x))
(12)
+ ||x -rrozea (ReLU (Tgate (x))
4.3 p-Annealing
Fundamentally, an L1 penalty has been used to induce sparsity in SAE features because it serves as convex relaxation of the true sparsity measure, the Lu-norm The L, Li-norm is the convex hull of Lo-norm, making it tractable alternative for promoting sparsity (63) However, the proxy loss function is not the same directly optimizing for sparsity, leading to issues such feature shrinkage (2) and potentially less sparse learned features. Unfortunately, the To-norm is non-differentiable and directly minimizing it is an NP-hard problem [48] [8), rendering it impractical for training.

---

0.45
Cove Sgeargeee of Chess Board State
0.45
Recounted (Fieliy)
0.996
0.996
0.40
Boars
0.994
0.35
0.35
0.942
eog
0.30
0.30
0.990
Coverage
0.2
Gatedd SAL
0.988
Gacadd SAL w/ p-anmeating
Gated SAI wi p-annealing
Standand w/ s-annealing
Standard wi connneating
0.986
Standard
0.20
Standard
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
(Loweer
(Loweer
(a) Coverage of Board State
(b) Coverage of Board State vs. Lo
1.000
Map
0.8
Hec rnconstuucioon
Cheest Board State Reconsruution
0.8
100s Beroueedd (Sitelty)
0.998
0.7
0.7
0.996
0.0
0.994
aas" 19i0 #
0.5
0.4
0.4
0.990
Gated
0.3
0.3
Gated p-annewing
0.2
w/ anneal ng
Standard
50
100
150
200
250
300
350
400
5G
100
150
260
250
300
350
400
Lg (Lower sparser]
(Lower sperrse))
(c) Board Reconstruction
(d) Board Reconstruction vs. Lo
Figure 2: Comparison of the coverage and board reconstruction metrics for chess SAE quality on 9band state- The coverage score reports the mean F1 scores over BSPs. The top row corresponds to coverage, and the bottom row corresponds to board reconstruction. The left column contains scatterplot of loss recovered vs. Lo. with the scheme color corresponding to the coverage score and each point representing different hyperparameters. We differentiate between SAE training methods with shapes.
In this work, we propose the use of nonconvex LP- m--iimiizzatinn with < 1, as an alternative to the standard L1 minimization in sparse autoencoders (SAEs). This approach has been successfully employed in compressive sensing and sparse recovery to achieve even sparser representations [[] 61 64 60]. To perform this optimization, we introduce method called p-annealing for training SAEs, based on the compressive sensing technique called p-continuation [66]. The key idea is to start with convex In miiiimizzation through setting p and progressively decrease the value of p during training, resulting in closer approximations of the true sparsity measure, Lo. as approaches We define the sparsity penalty for each batch æ as a function of the current training step 8:
Esparse (x, 8) = Ar xx do Σ f((x)*
(13)
In other words, the sparsity penalty will be scaled LE norm of the SAE feature activations, with decreasing over time. At 1, the LG norm equal the L1 norm, and as p - 0, the LE norm limits to the Lo-norm, as limp- Σ. fx(x² E.(x)
The purpose of annealing from 1 - 0 instead of starting from fixed, low value for p is that the lower the p. the more concave (non-convex) the LG norm is, increasing the likelihood of the training process getting stuck in local optima. which we have observed in initial experiments. Therefore, we aim first arrive at region an optimum using the easier-to-tue shift the loss function. This manifests as keeping decreasing p linearly down to Pend > at the end of training. We set Pend 0.2.
L1 penalty and then gradually 1 for a certain number of steps and then starting
Coefficient Annealing. Changing the value of p changes the scale of the Lg norm. Without also adapting the coefficient X, the strength of the sparsity penalty would vary too wildly across training. Empirically, we found that keeping constant À would lead to far too weak of sparsity penalty for

---

m
0.65
0.63
Recnunted (Fieliy)
0.996
0.60
0.996
0.55
0.994
Coldra Chess
Coverage Chesss
0.942
0.50
0.50
eog
0.990
0.45
Ginted
0.988
Standand w/ s-annealing
Gacadd SAL w/ p-anmeating
0.40
0.40
Standard wt pannessinng
Gaste wr p-annee
0.986
Standard
0.35
Standandd
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
(Loweer
(Lotwer
(a) Coverage of Strategy BSPs
(b) Coverage of Strategy BSPs vs. Lo
1.000
0.650
Stateg RSP Becenenie Recensturtion
0.650
(Fodeity)
0.600
0.575
eae oog" ane
0.550
0.900-
0.525
0.525
1008
Gated SAE
Fattec
0.908
tatted SAL wi paarmeeling
0.500
0.500
Gatted SAE wi annealing
Standard e-anneaiing
CRNES
Standard wr planeebinng
0.986
Standard
0.455
Staridaad
100
150
250
250
100
250
sperser)
sperser)
Strategy BSP Reconstruction
(d) Strategy BSP Reconstruction Lo
Figure 3: Comparison of the coverage and board reconstruction metrics for chess SAE quality on Gstrttgyy The metrics represent the average coverage and board reconstruction obtained across all BSPs in Guratey. The coverage score reports the mean of maximal Fl scores over BSPs. The absolute coverage scores vary significantly between strategy BSPs, as discussed Appendix D The top row corresponds to coverage, and the bottom row corresponds to board reconstruction. The left column contains a scatterplot of loss recovered vs. Tg, with the color scheme corresponding to the coverage score and each point representing different hyperparameters. We differentiate between SAE training methods with shapes.
the larger p's at the start of training, making the process worse than simply training with a constant y from the beginning. Consequently, we aim to adapt the coefficient À such that the strength of the sparsity penalty changed significantly due to p updates. Formally, the update step is:
Ej g+1 E((i()) As+1 + X Σ Ej=r-gtt 9+1 E, f((x))
(14)
We keep queue of the most recent q batches of feature activations mid-training and them to calibrate the X, updates. Therefore, the strength of the sparsity penalty is kept locally constant.
Combining p-annealing with other SAEs. Since the p-annealing method only modifies the L terms in the loss function without affecting the SAE architecture, it is simple to combine p-annealing with other SAE modifications. This allows to create the Fated-Annealed SAE method by combining the Gated SAE architecture and p-annealing. Concretely, the sparsity term R||ReLU( Tatte (x))]l in with X, ReLUU(Tgad (x)) [|: Our experiments showed that the optimum values for coefficients À and X, differ.
modify Lateed (Equation (2) by replacing
5 Results
In this section, we explore the performance of SAEs applied to language models trained on Othello and chess. Consistent with Nanda et [47), we find that interpretable SAE features typically track properties relative to the player whose turn it is (c.g. "my king pinned" rather than "the white king

---

Chess
Othello
Model
Coverage
Reconstruction
Coverage
Reconstruction
SAE: random GPT
0.11
0.01
0.27
0.08
SAE: trained GPT
0.48
0.85
0.52
0,95
Linear probe
0.98
0.98
0.99
0.99
Table I: Best performance obtained for different techniques across games for Goard state- As baseline, we train an SAE on random GPT, a version of the trained GPT model with randomly initialized weights. All models were trained on activations after the post-MLP residual connection in layer
is pinned"). To side-step subtleties arising from this, we only extract our activations from the token immediately preceding white's 's move. Specifically, we consider SAEs trained on the residual stream activations after the sixth layer using the four methods from Section 4 (see Table for additional hyperparameters). addition our metrics introduced above, we also make use of unsupervised metrics previously appearing in the literature [9] 16 54]:
Lo measures the average number of active SAE active features (i.e., positive activation) on a given input.
Loss recovered measures the change in model performance when replacing activations with the corresponding SAE reconstruction during forward pass. This metrie is quantified as (H, HH Ho). where Harig is the cross-entropy loss of the board game model for next-token prediction, H. is the cross-entropy loss after substituting the model activation with its SAE reconstruction during the forward pass, and Hn is the cross-entropy loss when zero-ablating x.
Our key takeaways
follows.
SAE features can accurately reconstruct game boards. In general, we find that SAE features are effective at capturing board state information in both Othello and chess (see Table T Figure 2 and 4d). In contrast, SAEs trained on a model with random weights perform very poorly according to metrics, showing that SAE performance is driven by identifying structure in the models' learned representation of game boards. Nonetheless, SAEs do not match the performance of linear probes in terms of reconstructing the board state. This performance gap suggests that SAFs do not capture all of the information encoded in the model's internal representations.
Standard SAEs trained with p-annealing perform on par with Gated SAEs. We find that standard SAEs trained using p- p-annealing consistently perform better than those trained with a constant L, penalty (Equation 8), measured by existing proxy metries and in terms of improvement in coverage (see Figure 2p 4b) In fact, standard SAEs trained using p-annealing show a coverage score that is comparable to Gated SAES trained without p-annealing. Further,
find that both
p-annealing and Gated SAFS significantly outperform Standard SAFS in addressing the shrinkage problem [62]. as detailed in Appendix E However, we find cases where our coverage metric disagrees with existing metrics. In Figure 2 for example, Gated SAEs perform achieve a higher loss recovered score than Standard SAES trained using p-annealing. We emphasize that the training and inference of Gated SAEs is more computationally expensive, requiring 50% more compute per forward pass compared to Standard SAES 54
Coverage and board reconstruction reveal differences in SAE quality not captured by unsuper- vised metrics. Our metrics reveal improvements in SAE performance that traditional proxy metries fail to capture. For example, we trained SAEs with hidden dimensions 4096 and 8192 (expansion factors of 8 and 16, respectively). We expect the SAEs with 8192 hidden dimensions perform better since they have greater capacity. However, we observe that they perform equally well according to prior unsupervised metrics (see Figures [] a, с and 4 a, c). In contrast, our metrics reveal that SAES with larger hidden dimensions are better. For the Standard architecture, this is reflected by the parallel lines (of purple diamonds) Figures 2 b. d and 4 b, Thus, our metrics are able to capture improvements from larger expansion factors. In addition, we find that the performance of p-annealing closely resembles that of Gated SAES when evaluated using standard proxy metries; demonstrates clear improvements under our proposed metrics.
8

---

1.0005
0.5
Worh goo
Coverage of Othe Baand State
Recrovrin (tidelit))
0.9995
0.a
Othella Board
0.4
0.9990
0.9965
00.3
0,9980
02
0.9075
0.9970
Garted SAF et p-amneatin
Guted SAE
0.1
Coverage
Gatted Sit wi p-anneaiing
Liated SAt
0.9965
Standard annealing Stondare
pranneeling
0.9900
0.0
200 (Lower sparser)
Standardd
(Lower sparser)
(a) Coverage of Board State
(h) Coverage of Board State To
3.0
1.0000
2
Othello Board State Reconttrur contrrutooo
Ochela Beard State Reconsrrution
Recouree (heblity)
0.0005
0.8
0.8
0.9900
$ Sran
0.0
0.6
0.0985
0.9980
0.1
0.9975
Gated SAE
0.99/0
0.2
Gated annca
0.9965
Stanclard wt portrreling
0.9060
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
(Lower is sporser)
(Lower sparser)
(c) Board Reconstruction
(d) Board Reconstruction vs. T.
Figure 4: Comparison of the coverage and board reconstruction metrics for Othello SAE quality on Foward state- The coverage score reports the mean of maximal F1 scores over BSPs. The top row corresponds to coverage, and the bottom row corresponds to board reconstruction. The left column contains scatterplot of loss recovered vs. Lo, with the color scheme corresponding to the coverage score and each point representing different hyperparameters. We differentiate between SAE training methods with shapes.
Coverage and board reconstruction are consistent with existing metrics. Figures and demonstrate that both coverage and board reconstruction metries are optimal in the elbow region of the Pareto frontier. This region, where SAEs reconstruct internal activations efficiently with minimal features, also yielded the most coherent interpretations during our manual inspections. This provides precise, empirical validation to the common wisdom that SAFs in this region of the Pareto frontier are the best.
Limitations
The proposed metrics for board reconstruction and coverage provide a more objective evaluation of SAE quality than previous subjective methods. Nevertheless, these metrics exhibit several limitations. Primarily, their applicability is confined to the chess and Othello domains, raising concerns about their generalizability other domains different models. Additionally, the of BSPs that underpin these metrics is determined by researchers based on their domain knowledge. This approach may encompass all pertinent features or strategic concepts, thus potentially overlooking essential aspects of model evaluation. Developing comparable objective metrics for other domains, such as natural language processing, remains significant challenge. Moreover, our current focus is on evaluating the quality of SAEs in terms of their ability to capture internal representations of the model. However, this does not directly address how these learned features could utilized for downstream interpretability tasks.

---

7 Related work
Sparse dictionary learning. Since the nineties, dictionary learning 22
20 sparse regression 266
and later, sparse autoencoders [49 have been extensively studied in the machine learning and signal processing literature. The seminal work of Olshausen and Field [5T] introduced the concept of sparse coding in neuroscience (see also [52], building upon the earlier concept of sparse representations [99
and matching pursuit [42]. Subsequently, a series of works established the theoretical and algorithmic
foundations of sparse dictionary learning [23 30 21 1 65
32 59 5 7 [Q]. Notably, Gregor and
LeCun [28] introduced LISTA, an unrolled version of ISTA 17 that learns the dictionary instead of
having it fixed.
In parallel, autoencoders were introduced machine learning to automatically learn data features and perform dimensionality reduction [29] 377. Inspired by sparse dictionary learning, sparse autoencoders
were proposed as an unsupervised learning model build deep sparse hierarchical models of data, assuming certain degree of sparsity in the hidden layer activations. Later, Luo et al. generalized sparse autoencoders (SAEs) to convolutional SAEs. Although the theory for SAEs is less developed than that of dictionary learning with fixed dictionary, some progress has been made quantifying whether autoencoders can, indeed, do sparse coding, e.g., Arpit al. [4. Rangamani
49
13
355
39
et al.
55
Nguyen et al. (50]
Feature disentanglement using sparse autoencoders. The individual computational units of neural networks are often polysemantic, i.e., they respond to multiple seemingly unrelated inputs [ Elhage et al. [24] investigated this phenomenon and suggested that neural networks represent features in linear superposition, which allows them to represent more features than they have dimensions. Thus, in an internal representation of dimension n, a model can encode m >> n concepts as linear directions [53], such that only sparse subset of concepts
concept deeply related to the coherence of vectors [26] and to frame theory in general [2] To identify these concepts, Sharkey et al. [57] used SAEs perform dictionary learning on one-layer transformer,
active across all inputs
identifying a large (overcomplete) basis of features. Cunningham et al. (16] applied SAEs to language models and demonstrated that dictionary features can be used localize and edit model behavior. Marks et al. proposed a scalable method to discover sparse feature circuits, as opposed to circuits consisting of polysemantic model components, and demonstrated that human could change the generalization of a classifier by editing its feature circuit. Recently, Kissane et al. (34) explored autoencoders for attention layer outputs. These works have benefited from a variety of open-source libraries for training SAEs for LLM interpretability 43 g 15
8 Conclusion
Most SAE research has relied on proxy metrics such as loss recovered and Lo, or subjective manual evaluation of interpretability by examining top activations. However, proxy measures only serve as an estimate of interpretability, monosemantic nature, and comprehensiveness of the learned features, while manual evaluations depend the researcher's 's domain knowledge and tend to be inconsistent.
Our work provides new, more objective paradigm for evaluating the quality of an SAE methodology; coverage serves as a quantifiable measure of monosemanticity and quality of feature extraction, while board reconstruction serves as a quantifiable measure of the extent to which an SAE exhaustively representing the information contained within the language model. Therefore, the optimal SAE methodology can be judged by whether it yields both high coverage and high board reconstruction.
Finally, we propose the p-annealing method, a modification to the SAE training paradigm that can be combined with other SAE methodologies and results in improvement in both coverage and board reconstruction over the Standard SAE architecture.
10

---

Author Contributions
A.K. built and maintained our infrastructure for working with board-game models. A.K., S.M., C.R., J.B., and L.S. designed the proposed metrics. B. B.W. performed initial experiments demonstrating the benefits of training SAEs with p < B.W., C.M. V., and S.M. then proposed p-annealing, with B.W. leading the implementation and developing coefficient annealing. The basic framework for our dictionary learning work was built and maintained by S.M. and C.R. The training algorithms studied were implemented by S.M., C.R., B. W. R.A., and J.B. R.A. trained the SAEs used in our experiments. A.K., C.R., and J.B. selected and implemented the BSPs. A.K. and J.B. trained the linear probes. Many of the authors (including L.S., J.B., R.A.) did experiments applying traditional dictionary learning methods and exploring both toy problems and natural language settings, which helped build valuable intuition. The manuscript was primarily drafted by A.K., B. B.W., C.R., R.A., J.B., C.M. V., and S.M., with extensive feedback and editing from all authors. D.B. suggested the original project idea.
Acknowledgments
C.R. is supported by Manifund Regrants and AISST. L.R. is supported by the Long Term Future Fund. S.M. is supported by an Open Philanthropy alignment grant.
The work reported here was performed in part by the University of Massachusetts Amherst Center for Data Science and the Center for Intelligent Information Retrieval, and in part using high performance computing equipment obtained under grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative.
References
[1] Michal Aharon, Michael Elad, and Alfred Bruckstein. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing. 54(11):4311-422 2006.
[2] Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. In Conference on learning theory, pages 113-149. PMLR, 2015.
[3] Sanjeev Arora, Yuanzhi Li, Yingyu Liang. Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018. doi:
URL https
//aclanthology.o www.olllgooooooooooooooo oww.oeo....//11-144--00 1034
[4] Devansh Arpit, Yingbo Zhou, Hung Ngo, and Venu Govindaraju. Why regularized auto- encoders learn sparse representation? In International Conference on Machine Learning, pages 144. PMLR, 2016.
[5] Chenglong Bao, Hui Ji, Yuhui Quan, and Zuowei Shen. Dictionary learning for sparse coding: Algorithms and convergence analysis. IEEE transactions on pattern analysis and machine intelligence, 38(7):1356-1366 2015.
[6] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. URL https://openippblicc blob. core. windows. net/neuron- explainer/puper/inle. html. (Date accessed: 14.05. 2023), 2023.
[7] Jaroslaw Blasiok and Jelani Nelson. An improved analysis of the er-spud dictionary learning algorithm. In 43rd International Colloquium Automata, Languages, and Programming (ICALP 2016). Schloss Dagstuhi-Leibbiz-Zennnn fuer Informatik, 2016.
[8] Joseph Bloom and David Chanin. Sae lens. 2024.
com/
[9] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity:
11

---

Decomposing language models with dictionary learning, Transformer Circuits Thread, page 2, 2023.
[10] Matthew Chalk, Olivier Marre, and Gašper Tkačik. Toward unified theory of efficient, predictive, and sparse coding, Proceedings of the National Academy of Sciences, 115(1): 186-199 191, 2018.
[11] Rick Chartrand. Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Processing Letters, 14(10):707-710, 2007.
[12] Ole Christensen et al. An introduction frames and Riesz bases, volume 7. Springer, 2003.
[13] Adam Coates and Andrew Y Ng. The importance of encoding versus training with sparse coding and vector quantization. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 921 12122 2011.
[14] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-222, JMLR Workshop and Conference Proceedings, 2011.
[15] Alan Cooney. Sparse autoencoder library. btttps//gitbbie httpstttuu..oom..iieeeeeeee wwt.:/aca.aa..aataaaaaaaeaaaaaee cctm/ aparse_autoencocde 2023.
[16] Hoagy Cunningham, Aidan Ewart, Logan Riggs Smith, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2023.
[17] Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Muthematics: Journal Issued by the Courant Institute of Mathematical Sciences, 57(11): 1413-1457, 2004.
[18] Geoff Davis, Stephane Mallat, and Marco Avellaneda. Adaptive greedy approximations. Constructive upproximation, 13:57-98, 1997.
[19] David Donoho. Superresolution via sparsity constraints. SIAM journal on mathematical analysis, 23(5):1309-133: 1992.
[20] Bogdan Dumitrescu and Paul Irofti. Dictionary learning algorithms and applications. Springer, 2018.
[21] Julian Eggert and Edgar Korner. Sparse coding and nmf. In 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No. 04CH37541), volume 4, pages 2529-2533. IEEE, 2004.
[22] Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer Science & Business Media, 2010.
[23] Michael Flad and Alfred M Bruckstein. A generalized uncertainty principle and sparse rep- resentation in pairs of bases. IEEE Trunsactions on Information Theory, 48(9):2558-255 2567, 2002.
[24] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravee, Zae Seafield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209, 200e2220990006.... 2022.
[25] Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-jussà. A primer on the inner workings of transformer-based language models. arXiv preprint arXiv:2405.00200 2024.
[26] Simon Foucart and Holger Rauhut. Mathematical Introduction to Compressive Sensing. Springer New York, New York, NY, 2013.
[27] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04093, 2024.
12

---

[28] Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pages 399-406 406, 2010.
[29] Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances in neural information processing systems, 6, 1993.
[30] Patrik O Hoyer. Non-negative sparse coding. In Proceedings of the 12th IEEE workshop neural networks for signal processing, pages 557-565. IEEE, 2002.
[31] Adam Jermyn, Adly Templeton, Joshua Batson, and Trenton Bricken. Tanh penalty in dictionary learning, 2024. URL
update/ index
html Accessed: 2024-05-20.
[32] Alexander Jung, Yonina C Eldar, and Norbert Görtz. Performance limits of dictionary learning for sparse coding. In 2014 22nd European Signal Processing Conference (EUSIPCO), pages 765-769. IEEE, 2014.
[33] Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models, 2024.
[34] Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Commy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. In ICML 2024 Workshop Mechanistic Interpretability, 2024.
[35] Jun Li, Tong Zhang, Wei Luo, Jian Yang. Xiao-Tong Yuan, and Jian Zhang. Sparseness analysis in the pretraining of deep neural networks. IEEE transactions on neural networks and learning systems, 28(6): 1425-1438. 2016.
[36] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg, Emergent world representations: Exploring sequence model trained on synthetic task. In The Eleventh International Conference on Learning Representations, 2023.
[37] Pengzhi Li, Yan Pei, and Jianqiang Li. A comprehensive survey on design and application of autoencoder in deep learning. Applied Soft Computing, 138:110176 2023.
[38] Lichess. lichess.org open database, 2024. URL URRpittts:///aaaaaeeet wwwpeee..hhhhhheheeccceeee
[39] Wei Luo, Jun Li. Jian Yang, Wei Xu, and Jian Zhang. Convolutional sparse autoencoders for image classification. IEEE transactions on neural networks and learning systems, 29(7): 3289-3294, 2017.
[40] Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv preprint wxXiv:2405.08366 2024.
[41] Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv: arXiv:1312.5600 1312.5663, 2013.
[42] Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing, 41(12):3397-3415 1993.
[43] Samuel Marks and Aaron Mueller. dietionary_learnin
ceathub..omm/sapraarkk..
dictionary_learninng 2024.
[44] Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403. 19647, 2024.
[45] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47), November 2022. ISSN 1091 16490. doi: 10. 107 22w.3/22.206625111110 2206625119. URL http:/ //xx doi 100/10
1073/pnas. 1073/ ppas. 2206625119
13

---

[46] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances neural information processing systems, 26, 2013.
[47]
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations world models of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Na- joung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNI.P Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16-30, Singapore, De- cember 2023, Association for Computational Linguisties. doi: 10.18653/v1/2000 blackboxnlp- 1lac 2. URL htttpp:://allanthooogyy htt.aaawnthoooooooo///2222 org/2023. blackboxnlp-1.2
[48] Balas Kausik Natarajan. Sparse approximate solutions linear systems. SIAM journal on computing, 24(2):227-2) 1995.
[49] Andrew Ng al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-1) 2011.
[50] Thanh V Nguyen, Raymond KW Wong, and Chinmay Hegde. On the dynamics of gradient descent for autoencoders. In The 22nd International Conference on Artificial Inteligence and Statistics, pages 2858-2867, PMLR, 2019.
[51] Bruno Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 (6583)::607-609)) 1996.
[52] Bruno A Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in neurobiology, 14(4):481-487, 2004.
[53] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03668, 2023.
[54] Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014 2024.
[55] Akshay Rangamani, Anirbit Mukherjee, Amitabh Basu, Ashish Arora, Tejaswini Ganapathi, Sang Chin, and Trac D Tran. Sparse coding and autoencoders. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 36-40. IEEE, 2018.
(56) Logan Riggs and Jannik Brinkmann. Improving sparse autoencoders by square-rooting 11 and removing lowest activation features, 2024. URL http::pppee/.ee.eeeessrroooooo// ViGGsqQJBalBgwt2YVV wwwp://wwnnnooinnnniem.. sae- s- byy-s-rr-aoo ing -ed------------ Accessed: 2024-05-20.
[57] Lee Sharkey, Dan Braun, and Beren Millidge. tures out of superposition with sparse autoencoders, 2023.
Taking
fea- URL
https://www.. http:://www.aliggneetffooroo http://ww.....ooeooto//o/tttttta wwwepptt-takinn--fetteeeeeeeeeee out- of oussuprrpoiition
Accessed:
2023-05-10.
[58] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https://traasffrmmeet cwrtietrrrrrrttt.te. pww.u0024.pu///244 html
scaling mwwosemanticityy/nnaaaaaaa
[59] Andreas M Tillmann. On the computational intractability of exact and approximate dictionary learning. IEEE Signal Processing Letters, 22(1):45-49, 2014.
[60] Meng Wang, Weiyu Xu, and Ao Tang. On the performance of sparse recovery via e minimization (0 < p
S 1). IEEE Transactions on Information Theory, 57(11):255-7278,
2011.
14

---

[61] Jinming Wen, Dongfang Li, and Fumin Zhu. Stable recovery of sparse signals via Ip- minimization. Applied and Computational Harmonic Analysis, 38(1):161-176, 2015.
[62] Benjamin Wright and Lee Sharkey. Addressing feature suppression in sparse au- toencoders, 2024. URL htttps://wwx lesawrong....................... cwm wwt.ccommppocc. addressing feature- weeature-suppptee suppression saes Accessed: 2024-05-20
[63] John Wright and Yi Ma. High-dimensional data analysis with low-dimensional models: Princi- ples, computation, and applications. Cambridge University Press, 2022.
[64] Chengzhu Yang. Xinyue Shen, Hongbing Ma. Yuantao Gu, and Hing Cheung So. Sparse recovery conditions and performance bounds for b, ey-minimization. IEEE Transactions on Signal Processing, 66(19):001455028 2018.
[65] Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching using sparse coding for image classification. In 2009 IEEE Conference on computer vision and pattern recognition, pages 1794-1801. IEEE, 2009.
[66] Le Zheng, Arian Malcki, Haolei Weng, Xiaodong Wang, and Teng Long. Does lo m--miiiizzation outperform { E1-minimization? IEEE Transactions on Information Theory, 63(11):6896-69) 2017.
15

---

A Improving and evaluating sparse autoencoders
Despite the success of SAEs at extracting iummannntterretabble features, they fail perfectly recon- struct the activations [16]. One challenge in the training of SAEs with an L1 penalty is shrinkage (or 'feature suppression ); in addition encouraging sparsity, an L1 penalty encourages feature activations to be smaller than they would be otherwise. Wright and Sharkey [62] approached this problem by fine-tuning the sparse autoencoder without sparsity penalty. Appendix LE further quan- tifies shrinkage across suite of SAEs trained on chess and Othello models. Jermyn et al. [31] and Riggs and Brinkmann [56] explored alternative sparsity penalties to reduce feature suppression during training. Rajamanoharan et al. [54] introduced Gated SAEs, architectural variation for the encoder which both addresses shrinkage and improves on the Pareto frontier of Lo vs reconstruction error. Recently, Gao et al. [27] systematically evaluated the scaling laws with respect to sparsity, autoencoder size, and language model size.
The goal of dictionary learning in machine learning is to produce human human-interpreed interpretable features and capture the underlying model's computations is difficult and often involves manual inspection. Therefore, most existing work assesses the quality of SAEs along different proxy metrics: (1) The cross-entropy loss recovered, which reflects the degree to which the original loss of the language model can be recovered when replacing activations with the autoencoder predictions. (2) The Lo-nnrmm of feature activations E. [|A(=)]lo measuring the number of activate features given an input (25). Makelov et al. [40] proposed to compare SAEs against supervised feature dictionaries in a natural language setting. However, this requires a significant understanding of the model's internal computations and is thus not scalable.
( However, quantitatively measuring interpretability
B Sparse Autoencoder Training Parameters
We used a single NVIDIA A100 GPU for training SAEs and experiments. takes much less than 24 hours train a single SAE 300 million tokens. Given trained SAE, our evaluation requires less than minutes of computing time.
Table 2: Training parameters of our sparse autoencoders.
Parameter
Value
Number of tokens
300M
Optimizer
Adam
Adam betas
(0.9, 0.999)
Linear warmup steps
1,000
Batch size
8,192
Learning rate
3e-4
Expansion factor
{8. 16)
Annealing start
10,000
Pend
0.2
Ainit
[0.02, 2.0]
16

---

C List of Board State Properties
Table summarizes the high-level board state properties considered in Garategy The selection of concepts was inspired by McGrath et al. 45). The column indicated by # denotes the number of individual BSPs per concept. single BSP per concept indicates we match this condition globally for any corresponding piece.
Table 3: List of strategic Board State Properties.
Concept
Description
check
Indicates whether the player move is checked by the opponent.
cae_k
Indicates whether the player move could check the opponent with the next move.
queen
Indicates whether the player to move has a queen the board.
can_capture_queen
Indicates whether the player to move can capture the queen of the opponent.
bishop_pair
Indicates whether the player to move still has both bishops on the board.
castling.rights
Indicates whether the player move still allowed to castle, contingent on the king and the rooks not having moved.
tingside_castlhng.rrittss
Indicates whether the player to move is still allowed to kingside castle, contingent on the king and the kingside rook not having moved.
Indicates whether the player move still allowed to queenside castle, contingent on the king and the queenside rook not having moved.
fork
Indicates whether the player to move attacks has fork on major pieces of the opponent.
pin
Indicates whether there is a pin on the board, such that a player's piece cannot move without exposing the king behind it to capture.
Lagal_en_passsaee
Indicates whether the player move has legal en passant: a special pawn capture that only occur immediately after an opponent moves pawn two squares from its starting position and it lands beside the player's 's pawn.
ambiguous_movee movess
Indicates whether there are moves that would require further specification more than one piece the same type can move to the same square.
threatened__quaree
64 Indicates which squares are threatened by the opponent.
legal legal_moves moves
64 Indicates which squares can be legally moved to by the current player.
17

---

D Performance of Linear Probes and SAEs on Board State Properties
In Figure we present a mean coverage score over strategy board state properties Gusp. Properties within OBSP vary significantly in complexity. For example, queen detection can be inferred directly from the move history, while fork detection requires an accurate representation of the board state. Table 4 shows that linear probe Fi-score is below 0.95 for 6 out of 15 properties in GUSP- This suggests that chess-GPT 333 does not represent these properties linearly, Additional experiments are required to determine whether the representation is present at all.
For the board state case, reconstruction is significantly higher than coverage. This is because there many SAE features that are high precision classifiers for a configuration of squares, such as "white pawn on e4, white knight of f3". In cases where coverage is higher than reconstruction (such as for can can_ccannneeaaa check), it is because there are not many features that are over 95% precision for "there is
check move available" from which we can recover there an available check move. Coverage is significantly higher because there is at least one feature that has an F1 Fi-score of 0.54 for can_ check, which may not have precision greater than 95%.
Table 4: Comparison of performance of linear probes trained to predict board state properties given residual stream activation of ChessGPT after the sixth layer with SAEs evaluated using coverage and reconstruction metrics.
Concept
Linear Probe -core
Best SAE Reconstruction score
Best SAE Coverage score
check
1.00
1.00
1.00
can_check
0.93
0.27
0.54
can_capture_queen
0.66
0.62
0.48
queen
1.00
0.97
0.96
bishop-pair
1.00
0.83
0.86
castling.rights
1.00
0.98
0)82
kingside_castling
1.00
0.98
0.81
queenside_castling
1.00
0.97
0.81
fork
0.68
0.13
0.38
pin
0.67
0.20
0.33
legal_en_paasaan
0.96
0.92
0.90
ambiguous_moves
0.72
0.25
0.57
threatened_squares
0.96
0.93
0.71
0.92
0.66
0.63
board_state state
0.98
0.67
0.41
18

---

Table 5: Comparison of performance of linear probes trained predict high-level board state properties given residual stream activations with SAES, both trained on model with the same architecture as ChessGPT but randomly initialized. Performance on metrics can high when the metric is correlated with move number syntax level patterns (such castling, which corresponds "0-0").
Concept
Linear Probe
Best SAE Reconstruction score
Best SAE Coverage score
Fi -scoe
check
0.00
0.00
0.13
can_check
0.19
0.03
0.52
can_capture_queen
0.00
0.00
0.09
queen
0.85
0.95
0.93
bishop_pair
0.82
0.74
0.81
castling.rights
0.89
0.75
0.65
kingside_castling
0.89
0.75
0.65
queenside_castling
0.89
0.75
0.64
fork
0.01
0.00
0.07
pin
0.00
0.00
0.25
legal_en_passe
0.00
0.00
0.06
ambiguous_mmoves
0.13
0.00
0.52
threatened_squares
0.82
0.73
0.60
legal_mover
0.65
0.36
0.45
_oate_state
0.26
0.01
0.11
E
Relative Reconstruction Bias
Training Standard SAEs with an L1 penalty, as described in Section 4 causes systematic underesti- mation of feature activations. Wright and Sharkey [62 term this phenomenon shrinkage. Following Rajamanoharan al. we measure the relative reconstruction bias 7 of our SAEs, defined as:
Y arg min E,-D
(15)
Here, denotes a large dataset of model internal activations. Intuitively, < indicates shrinkage.
A perfectly unbiased SAE would have 1.
Our experiments show that p-annealing achieves similar relative reconstruction bias improvements to gated SAFS, both outperforming the baseline architecture. Figure shows that improvements manifest differently across domains: Chess SAEs show a narrower range bias (7 0.98) compared to Othello (y 0.80). This domain-dependent variation may reflect differences in the underlying models or data distributions. We observe unstable y values for SAEs with LO near zero, which represent degenerate cases outside the typical operating range of these models.
19

---

Chess SAE Relative Reconstruction Bias
Othello Relative Reconstruction
1.005
1.05
Rellative Reconstruction Blas
[x] shrinkage)
Rncondtracton
1.000
1.00
0.995
0.95
0.990
0.90
Gated SAE
0.85
Gated SAE Gated p-anneating
Gated SAE w/ annosling
Standant w/ p-inneating
Standaand
0.80
Standard
100
200
350
50
100
150
200
250
300
350
400
sperser)
(Lower sparser? ser)
(a) Coverage of Board State
(b) Coverage of Board State vs. Lo
Figure 5: Comparison of the relative reconstruction bias metric y quantifying feature activation
shrinkage across suite of SAFs. y < 1 indicates shrinkage. A perfectly unbiased SAE would have
F
Model Internal Board State Representation
F1
Othello Models
Previous research of Othello-playing language models found that the model learned nonlinear model of the board state [36]. Further investigation found a closely related linear representation of the board when probing for "my color" vs. "opponent's color" rather than white vs. black [47]. Based on these findings, when measuring the state of the board Othello, we represent squares as (mine, yours) rather than (white, black).
F.2
Chess Models
Similar to Othello models, prior studies of chess-playing language models found the same property, where linear probes were only successful the objective of the (mine. yours) representation and were unsuccessful on the (shite, black) representation [33]. They measured board state at the location of every period in the Portable Game Notation (PGN) string, which indicates that it is white's ' turn to move and maintain the (mine, yours) objective. Some characters in the PGN string contain little board state information measured by linear probes, and there is not clear ground truth board state part way through a move (e.g., the "f" in "Nf3"). We follow these findings and measure the board state every period in the PGN string.
When measuring chess piece locations, we do not measure pieces on their initial starting location, as this correlates with position in the PGN string. An SAE trained on residual stream activations after the first layer of the chess model (which contains very little board state information as measured by linear probes) obtains board reconstruction Fi Fioscore of 0.01 in this setting. If also measure pieces on their initial starting location, the layer SAE's Fi-score increases to 0.52, as the board can be mostly reconstructed in early game positions purely from the token's location in the PGN string. Masking the initial board state and blank squares decreases the F Fi-score of the linear probe from 0.99 to 0.98.
G Additional examples of learned SAE features
We present two additional examples of learned SAE features that we (subjectively) match to board state properties based on their maximally activating input PGN-strings in Figure 6
20

---

A
A
Ng4
8 N. bxc8 9.064 Badk 10.0xx4 0-0 11.8e2 Bege Bag4 13. Be6
14, h4 Qb6 15.0-0-0 Rabd 16, 1xb0 exb6 17.05 kg? 18.05 65 19. b4
28 28.Ne2 c5 21.NF4 RaB 22, RoB 23.64 24.04
(a) "En passant available" detector learned by an SAE.
1.e4 c6 2.M13 d6 3.Nc3 Ne7
4 06 B45 Ba7
O 7 Exc3+
Sjoxc
c5
90N Qes 10
(b) "Knight e7" detector learned by an SAE.
.64 2.c4 3.cxdd Qxd5 Qxdé @xd1+ 6xdx 7,8e2
Niee 0-0-0 9.0-0 18. 18. 11.b4 Ned 12, Bb2 lb2l f6 13. 3dc3 N82 14.RFd: Ned
15. Robi B6x82 Rxb2xb2
(c) Another example the "Knight on c2 or c7" detector shown in subfigure (b) above. We interpret this feature as representing a mirrored perspective. It may be firing for "opponent knight column, square away from opponent back rank", which why it both e2 (if black's turn to move) and (if white's turn to move).
Figure 6: Additional examples of learned SAE features. We show the full board state of chosen game in which the SAE latent has high activation. The PGN-string (model input) which represents the game history is shown below the board. Tokens that activate SAE features marked in blue, where darker shades correspond to higher feature activations. Moves that create the considered board state are highlighted in yellow.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in the abstract and introduction are faithful representation of the contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss our main limitation, the sensitivity of our metrics to human preconceptions, both in the introduction and in dedicated limitations section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all information about the sources of base models and datasets, as well as information about the sparse autoencoder achitecture and hyperparameters used during training.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the relevant code reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
cublicc/qudees/Codeeubbss casessionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppppgguuuiiaeeeae
Stteeee//ooSSSSeissSioica www.ssssiicggooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss the training process in the main section and provide additional information in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We present results across a large sweep of training parameters to provide information about the significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss our compute resources in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted does conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification; Our work focuses on evaluating current methods to extract features from neural networks, and proposes method to improve this process. We do not believe that this work has a direct impact on society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not introduce new datasets or models that pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators of original assets (models and data) properly cited in the paper.
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide Readme as well as in-code documentations alongside the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "WNkwzjS90u", "reviewer": "Reviewer_NNJv", "summary": "The paper introduces a setting for evaluating dictionary learning methods (and in particular sparse autoencoders) for language model interpretability. The setting proposed is that of interpreting features learned by language models trained on data representing Chess and Othello games. This setting should allow testing based on a more easily enumerable set of ground truth features (compared to natural language) which we know are relevant to the task and are captured by the underlying model.\n\nThe authors propose two new F1 based metrics to evaluate the SAEs ability to recover features learned by the model, namely a \"coverage\" metric and a \"board reconstruction\" metric that measures the quality of high precision features in the SAE. Both these metrics are computed by treating SAE features as classifiers for binary board state properties.\n\nThe paper also introduces a new training technique for SAEs called p-annealing that modifies the L1 loss to more closely approximate optimizing the L0 norm for which L1 loss acts as a proxy.", "questions": "- What is functionally/representationally different about BSPs in F_low vs F_high? \"Any square being threatened\" seems as high level as \"has a queen on the board\". I'm just trying to understand if there is a critical difference between these two classes of BSP from the perspective of the SAE or BSP classifier. It is nice that these concepts come from different sources, just wondering about the low-level vs high level implication.\n- What is the effect on the board reconstruction metric if there are BSPs for which there are no high precision features (line 131)?  Are these low precision features still used in calculating the final metric? It was not super clear to me from the notation in the equation below line 134 if **all** features are used used in this calculation, _but_ φfi,t(x) = 1 is only calculated for high precision features, or if **only** high precision features are included in this computation at all?\n- Why use a \"batch of train data\" to identify the high precision features? Why not use the whole dataset? How big should this batch be relative to the whole dataset?\n- Have the authors done any qualitative analysis of their metrics?\n\t- For example board reconstruction seems to overall be lower for high-level vs low-level BSPs. Have the authors observed that this translates into more difficulty interpreting board states retrieves by featured for the high level BSPs?\n- How many SAEs are trained for each of the 4 SAE training conditions {Gated, Standard} x {p-annealing, no p-annealing}?\n- Could you elaborate on line 220 where you say \"in the region with L0 below 100, which is the optimal range according to proxy metrics\"? Is this a reference to an existing finding, or how do we know that this is the optimal range for sparsity?\n\nA more speculative question:\n- Do the authors know if there is any correlation between their proposed metrics and the final reconstruction loss of the SAE? Particuarly since the linear probe performance on predicting BSPs seems to be high, might we expect to see a correlation to just straight reconstruction loss as another signal to validate these metrics or am I off base?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Having a better grasp on how to evaluate SAE's training methods would be very useful as they are generally fairly difficult objects to evaluate in terms of how interpretable they are (without a lot of manual inspection). The setting used is complex enough to be interesting while maintaining an ability to know ground truth concepts that should be present and separately probe that the models the SAEs are trying to give insight into actually have representations of those concepts. \n- The metrics proposed intuitively make sense and appear to relate well to established metrics like precision, recall and F1.\n- The motivation for p-annealing is well thought out and presented. And the method appears to have a strong effect on lowering overall sparsity.", "weaknesses": "- The figures and caption for figure 2 don't seem to match up (there may be a mixup with subfigures for figure 3). From reading the caption I would have expected the bottom row to show \"Coverage\" for high level BSPs in a similar pattern to the top row, however it shows \"Reconstruction\" of high level BSPs and reconstruction of low-level BSPs (which seem to appear as subfigures in figure 3). Could the authors correct this/post the appropriate charts?\n\n- I can't easily understand the comparison of the new metrics compared to the existing proxy metrics, the figures (2-5) seem to attempt to both validate the new metrics as well as compare the proposed p-annealing technique and thus get fairly busy making them hard to parse visually:\n    - The 3 variable plots on the left hand column of the figures are somewhat hard to read, especially since the proposed metric is encoded using color. Having a plot comparing each proxy metric (loss recovered and sparsity) to each proposed metrics would be easier to read. I do understand that the authors are trying to demonstrate that there is a region on the Pareto frontier where their metrics perform best, its not so apparent to me that sparser SAEs are always more \"interpretable\" SAEs so it is an interesting finding, but I still think it would be helpful to more easily compare each metric to the existing proxy metrics (for example I wouldn't necessarily expect that less sparse SAEs should have worse coverage—but maybe the authors have some thoughts on why that should be the case?)\n    - Having separate plots that focus on evaluating the p-annealing technique using the various metrics (proxy and proposed) when also be helpful (i.e. similar to Fig 5 in Rajamanoharan et al 2024). The overplotting of the various shapes for the different conditions (particularly in the left hand column) makes it a bit hard to understand the the trends and understand how well p-annealing works actually works.\n- From my read of the charts, the results for p-annealing seem quite different between Othello and Chess. I'm trying to map the final sentence about the efficacy p-annealing in the conclusion to the charts and i'm not sure if i'm able to see how the conclusion follows from the results. Could the authors comment on this?\n\n- There isn't much description of qualitative/manual analysis of the trained SAEs alongside the new metrics. Because measurement is relatively nascent in this space, it would be helpful to have some qualitative evaluation/results from manual inspection demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We’re glad that you think that our metrics intuitively make sense and that the motivation for p-annealing is well thought out and presented. \n\n**Mismatch between figures and caption:** Thank you; this is a mistake and we will correct this in the paper. We have included the updated plots in the attached PDF. Further updates are outlined in our global response.\n\n**Less sparse SAEs have worse coverage:** Here are two related perspectives on the connection between sparsity and coverage:\n- BSPs are typically sparse (e.g. only a sparse subset of chess boards have a knight on E3). Thus, if the SAE features are dense (as is the case for high L0 SAEs), then they have no chance of corresponding to sparse BSPs; thus sufficiently dense SAEs must have low coverage scores.\n- Consistent with prior work, we find that sparser SAEs have more interpretable features. Since coverage is a measure of whether our SAEs have features belonging to a certain class of interpretable properties, more interpretable SAEs will also tend to have better coverage scores.\n\n(These two perspectives are really two sides of the same coin: the connection between sparsity and interpretable properties is a key motivator for applying SAEs for interpretability [1, 2].)\n\n**Separate plots for evaluating p-annealing:** In the three variable plots, we wanted to demonstrate that p-annealing is a Pareto improvement in L0 / Loss recovered over the standard SAE and at the same time showcase how our new metrics can help to distinguish improvements invisible to existing metrics. Specifically, it becomes difficult to distinguish between the gated and p-annealing SAEs using existing metrics, while their differences are still visible in plots using our new metrics. However, we agree that it is difficult to distinguish between the gated and p-anneal SAEs in these plots and will make sure to improve the presentation in the final version.\n\n**Efficacy of p-annealing:** Thank you for pointing this out. The final sentence in the conclusion should be updated to better reflect our results. Specifically, we find that standard SAEs trained using p-annealing consistently perform better than those trained with constant L1 penalty by existing proxy metrics and in terms of coverage. However, Gated SAEs are on par with them in terms of coverage. We will include a more nuanced discussion in the results section. \n\n**Qualitative analysis of trained SAEs:** We did perform qualitative analysis of the trained SAEs, e.g. Figure 1 includes examples of game states that correspond to interpretable SAE features that we found. However, we found many more interpretable features that did not make it into the paper, such as an “en passant” feature that only activates when an en passant capture is available. We will provide additional qualitative results of trained SAEs alongside the new metrics in the appendix. \n\n**F_low vs F_high distinction:** You're right, our split into low vs. high seemed ambiguous and subjective. To address this, we moved to a more principled division of BSPs into (1) board state BSPs, which correspond to the presence of a particular piece at a particular board space, and (2) researcher-selected strategy BSPs for chess This is described in more detail in the global rebuttal. Please find our revised figures in the attached PDF.\n\n**Board reconstruction without high precision features:** Only high precision features are used in calculating the metric. If there are no high precision features for a BSP, the board reconstruction score for that BSP would be 0. We will clarify this in the text. \n\n**Batch of train data:** We apologize for the confusion caused by our terminology. To clarify, we use a consistent dataset of 1000 games as our training set for identifying high-precision features across all Board State Properties (BSPs). An additional, separate set of 1000 games serves as our test set. We will clarify this in our updated paper.\n\n**Qualitative analysis of metrics:** In qualitative analysis of high level BSP features, the features are inherently interpretable because we screen for 95% precision. Thus, for example, at least 95% of a pin feature’s activations will be when there is a pin on the board. We also noticed that high level features tend to activate in specific scenarios. For example, a pin feature may only activate on a more specific type of pin, such as “a pin by a bishop in the bottom left corner of the board”.\n\n**Number of SAEs trained** The plots contain 40 SAEs from each SAE training methodology for each board game. We had additional sweeps of 40 Standard SAEs on layer 1 of each trained model, and on layers 1 and 6 of the randomly initialized models.\n\n**Line 220, optimal L0 range** We try to make the same point as in our comment on **Less sparse SAEs have worse coverage:** above. Thank you for pointing this out, the word optimal is too strong here. We will replace “optimal L0 range” with “expected L0 range in line with previous work”.\n\n**Correlation between proposed metrics and final SAE reconstruction loss**  While there's some ambiguity in the term \"final SAE reconstruction loss”, we interpret this question as comparing `Loss Recovered` with our new metrics. Our metrics generally perform best at the elbow of the L0 / Loss Recovered plot, balancing sparsity and reconstruction quality. A higher Loss Recovered doesn't necessarily mean better performance on our metrics, as it often comes with higher L0 (less sparsity). Conversely, very low L0 tends to have low Loss Recovered. Thus, the relationship between our metrics and reconstruction loss isn't straightforward, as we're optimizing for a balance rather than maximizing either variable.\n\n\n\nReferences\n\n[1] A. Templeton et al., ‘Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet’, Transformer Circuits Thread, 2024.\n\n[2] Elhage, et al., \"Toy Models of Superposition\", Transformer Circuits Thread, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RG0QwgYLS1", "reviewer": "Reviewer_R4sd", "summary": "This manuscript applies sparse autoencoders (SAE) to detect interpretable features from autoregressive transformer models trained on Othello and chess. These controlled scenarios provide suitable testbeds, in the sense that we can extract ground truth features to measure progress in dictionary learning. Based on the observations that existing SAE optimizations may indeed be suboptimal, the authors propose *p-annealing*, a warm-start technique that iteratively optimize a sequence of (increasingly more non-convex) SAE objectives, in order to approximate the intractable $\\ell_0$ objective.", "questions": "* How sensitive is the SAE training to the choice of $p$-annealing schedule and other choice of hyperparameters?", "limitations": "The authors have adequately addressed limitations and potential societal impacts of their work", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The presentation of this paper is very clear. I really enjoyed reading this paper overall.\n* The topic of this paper is of high relevance NeurIPS community. SAEs are shown to be a promising approach for mechanistic interpretability. Providing a quick-to-iterate testbed as well as a set of improvement techniques would surely be valuable to this research.\n* The $p$-annealing technique is reasonable, is easy to implement, and has rich history in sequential and/or global optimization. It also demonstrate superior performance compared to standard SAE objectives.\n* Experiments are thorough, and demonstrate (1) the applicability of SAEs to learn interpretable features for Othello and chess tasks, and (2) the benefits of improved techniques on these tasks.", "weaknesses": "* While it is reasonable to operate in a controlled setup (i.e. Othello and chess) in this paper, there lacks concrete evidence on how these improved techniques can potentially improve SAEs in LLMs and natural language domains. Including a discussion and/or additional experiments on this could further strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We’re glad that you enjoyed reading the paper and think it’s of high relevance to the NeurIPS community. \n\n**Demonstrating transfer to natural language:** Yes, we agree that this is a limitation in our paper and future work should study this. However, note that our metrics reveal progress in SAE training which is invisible to existing proxy metrics. Specifically, SAEs with larger hidden dimensions perform better on our metrics, e.g. for the Standard architecture this is reflected in the parallel lines of purple diamonds (see attached PDF). The hope would be that training techniques which work better in the board game setting also work better in the natural language setting, though we don't validate that in this paper.\n\n**Sensitivity of SAE training to p-annealing schedule and other hyperparameters:** We did not conduct a systematic evaluation of the p-annealing schedule and other hyperparameters. From our experience it did not seem that sensitive to the schedule, but is a bit sensitive to the p_end value, i.e. the final p value at which we would stop the training. If p_end was below 0.15, the training started to destabilize. \n\nWe also noticed an effect on the initial lambda coefficient (strength of the sparsity penalty). Specifically, when using p-annealing, a small change to lambda could lead to larger differences in L0 than when training without it. This means it would often tighten the range of lambdas that we would explore. However, this only meant that we would explore over less orders of magnitude of lambda coefficients and get the same L0 spread.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "05VABCZzcf", "reviewer": "Reviewer_pBfq", "summary": "This paper proposes two measures to reflect the quality of the trained sparse auto-encoder when decomposing superposition features. In addition, this paper achieves approximate L0 optimization through dynamic adjustment of the p-norm and testing it on two types of board games.", "questions": "- Is there a separate training of SAE for MLPs, Attention outputs, etc., which write into the residual stream instead of activations between attention blocks?\n- Is there any relevant reference for the assumption about *High Precision* in line 130?\n- How were the hyperparameters adjusted in Figures 2, 3, 4, and 5 to obtain several different SAEs? (Causality between hyperparameters and SAE results needs further clarification [1].)\n- In [2], it's mentioned that Gated SAE can solve the shrinkage problem mentioned in line 167. Have you compared the relative reconstruction bias (gamma) proposed in [2] when using gated SAE w/wo p-annealing?\n\n[1] Kramár, János, et al. \"AtP*: An efficient and scalable method for localizing LLM behaviour to components.\" arXiv preprint arXiv:2403.00745 (2024).\n\n[2] Rajamanoharan, Senthooran, et al. \"Improving dictionary learning with gated sparse autoencoders.\" arXiv preprint arXiv:2404.16014 (2024).", "limitations": "I suggest the authors expand the scope of the research to include more domains and datasets, and validate the effectiveness of the proposed methods in other downstream interpretability tasks.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The research direction of this article is very valuable. Previous work using SAE for feature detection and circuit discovery did not directly measure the interpretability of the model decomposition results but relied on indirect measures such as reconstruction and sparsity levels of SAE. This work proposes using a few rules (low or high level) to automatically analyze the results of SAE decomposition under different hyperparameters, which is helpful for subsequent selection of SAE used for feature detection. Additionally, dynamically adjusting the p-norm is a straightforward method that intuitively approximates optimization of L0.", "weaknesses": "- The experimental objectives of the article are not clear: The article lacks of comparing the advantages of the two proposed measurement methods compared to reconstruction loss and sparsity. For example, demonstrating cases where both reconstruction loss and sparsity perform well but coverage and reconstruction are low, resulting in biases in downstream tasks (such as circuit discovery).\n- The rationality of the two proposed measurement methods in the article is questionable: The purpose of using SAE is to decompose the features of superposition into monosemantic features, with the expected decomposition results leaning towards several low-level features. However, based on the experimental results in Figure 2, there is no significant difference between low-level and high-level coverage results. This phenomenon is likely due to the unreasonable selection of rules.\n- Similarly, due to the fact that both coverage and reconstruction are ultimately detected using linear probing on individual features, some high-level features require linear combinations of multiple low-level features to be expressed [1,2,3]. If the measurement methods described in the article are used, it will not be effective in detecting these features.\n- Typos (not limited to): Line 154  \"than than\" should be corrected.\n- The image captions are incorrect: Figure 2 should be labeled \"coverage\" instead of \"reconstruction\" for the bottom row.\n\n\n[1] Templeton, Adly, et al. \"Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.\" Transformer Circuits Thread (2024).\n\n[2] He, Zhengfu, et al. \"Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT.\" arXiv preprint arXiv:2402.12201 (2024).\n\n[3] Rajamanoharan, Senthooran, et al. \"Improving dictionary learning with gated sparse autoencoders.\" arXiv preprint arXiv:2404.16014 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We’re glad that you think the research direction in our paper is very valuable. \n\n**Comparison of the advantages of the two proposed measurement methods compared to reconstruction loss and sparsity:** Thank you for giving us the opportunity to clarify this. In our paper, we compare three SAE training methodologies: Gated, Gated with p-annealing, and Standard with p-annealing. All three show Pareto improvements over the Standard approach in terms of reconstruction loss and sparsity but cannot be distinguished using existing unsupervised metrics (L0 and fraction of loss recovered). However, as you can see in the pdf attached to our global comment, the metrics we introduce clearly distinguish the SAEs. In other words, our metrics reveal differences in SAE quality which are invisible to prior unsupervised metrics. (As an especially clear example, the parallel curves that appear in our plots correspond to SAEs trained with the same algorithm but different hidden dimensions; our metrics clearly demonstrate that the SAEs with larger hidden dimensions are better than those with smaller hidden dimensions, as expected.)\n\n**No significant difference between low-level and high-level coverage, unreasonable selection of rules:** Thank you for pointing this out. In our original submission there was a mismatch between graphs and captions, and our actual high-level coverage graph was not included. We have provided an updated selection of graphs in the PDF.\n\nWe also agree that our initial split into low vs. high seemed ambiguous and subjective. To address this, we moved to a more principled division of BSPs into (1) board state BSPs, which correspond to the presence of a particular piece at a particular board space (giving 8 x 8 x 12 board state BSPs for chess and 8 x 8 x 2 for Othello), and (2) researcher-selected strategy BSPs for chess. Our goal was to confine researcher choice to the strategy BSPs as much as possible, while keeping the class of board state BSPs natural and principled.\n\n**Some high-level features require linear combinations of multiple low-level features to be expressed:** We would like to clarify that linear probing is not used in either the coverage or reconstruction metrics, which instead measure whether individual features found using SAEs correspond to individual BSPs. From our perspective, neither our work nor prior work suggests that interpretable high-level properties are necessarily expressed using an ensemble of low-level features. For example, in our work, we find individual features corresponding to high-level properties such as “an en passant capture is available”. Similarly, in Anthropic’s Scaling Monosemanticity [4] they find “a diversity of highly abstract features”. We hope this addresses your concern, but we also acknowledge that we might be misinterpreting your question. If this is the case, could you please provide further clarification? \n\n**Training of SAEs on various locations:** We only train SAEs on the residual stream between layers 6 and 7, and do not train SAEs on other locations. We chose this location because it achieved the best performance when reconstructing the board state with linear probes, which is consistent with prior work [1, 2, 3].\n\n**Assumption of High Precision:** We had several motivations for choosing high precision features. The first was that during manual inspection of SAE features, we found that features often seemed to activate on specific board configurations, rather than individual board square states. We also noted studies on chess players showing that chess experts excel at remembering realistic board configurations, but not random piece placements[4]. This suggests experts (and potentially AI models) encode board states as meaningful patterns rather than individual square occupancies.\n\nConcurrently, Anthropic has also evaluated features on their precision in Scaling Monosemanticity, which they called “specificity” [5].\n\n**Hyperparameter adjustment:** We sweep over the hyperparameters of sparsity penalty, learning rate, and expansion factor. A lower sparsity penalty tends to increase L0, while a larger expansion factor tends to increase Loss Recovered.\n\n**Relative Reconstruction Bias (Gamma from GDM paper [6])** Thank you for this suggestion. We have performed additional experiments measuring relative reconstruction bias. Our results show p-annealing achieves similar relative reconstruction bias to gated and gated w/ p-annealing, all improving over the standard SAE. The results can be seen in Figure 4 of our global pdf. We note that the y-axis scale for relative reconstruction bias differs between Chess and Othello SAEs. Chess shows a narrower range (minimum γ ≈ 0.98) compared to Othello (minimum γ ≈ 0.80), which more closely resembles the original GDM plot [6]. This may reflect differences in the underlying models or data.  In the Othello plot, we found erratic relative reconstruction bias values for SAEs with L0s very near 0; these are very bad SAEs outside of the sparsity range usually considered (including in [6]).\n\n[1] K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg, ‘Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task’, arXiv [cs.LG]. 2024.\n\n[2] N. Nanda, A. Lee, and M. Wattenberg, ‘Emergent Linear Representations in World Models of Self-Supervised Sequence Models’, arXiv [cs.LG]. 2023.\n\n[3] A. Karvonen, ‘Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models’, arXiv [cs.LG]. 2024.\n\n[4] Frey PW, Adesman P. ‘Recall memory for visually presented chess positions.’ Mem Cognit. 1976.\n\n[5] A. Templeton et al., ‘Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet’, Transformer Circuits Thread, 2024.\n\n[6] Rajamanoharan, Senthooran, et al. \"Improving dictionary learning with gated sparse autoencoders.\" arXiv preprint arXiv:2404.16014 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have taken a number of steps to improve, simplify, and clarify our analyses:
- Multiple reviewers raised that the distinction between low-level and high-level BSPs seemed ambiguous and subjective. To address this, we moved to a more principled division of BSPs into (1) board state BSPs, which correspond to the presence of a particular piece at a particular board space (giving 8 x 8 x 12 board state BSPs for chess and 8 x 8 x 2 for Othello), and (2) researcher-selected strategy BSPs for chess, discussed in more detail below [1]. Our goal was to confine researcher choice to the strategy BSPs as much as possible, while keeping the class of board state BSPs natural and principled.
- We identified and fixed an error which resulted in ground-truth game boards being mislabeled in around 10% of our data. As a result, our corrected results are much more crisp.
- We trained a new sweep of SAEs to span a larger range of L0s, with an emphasis on SAEs with a low L0 (which tend to be more interpretable).

The Standard SAE w/ p-Annealing method yields a Pareto improvement over Standard SAEs, visible in both existing unsupervised metrics (Figures 1a, 1c, 2a, 2c) and our supervised metrics (Figures 1b, 1d, 2b, 2d, 3a, 3b). While Gated SAE, Gated SAE w/ p-annealing, and Standard w/ p-annealing are not clearly separable by unsupervised metrics (L0 and Fidelity), our supervised metrics based on board state BSPs (Figures 1b, 1d, 3a, 3b) clearly differentiate these approaches.

Please find our revised figures 1-3 in the attached PDF. Additionally, Figure 4 contains the experimental results from the Relative Reconstruction Bias metric suggested by reviewer pBfq. Our results indicate that p-annealing achieves a similar relative reconstruction bias compared to gated and gated w/ p-annealing, all of which are significantly improved over the standard SAE.

[1] Strategy BSPs were selected by the authors based on domain knowledge and prior chess model interpretability work [2] to be properties relevant to playing strategically in chess (for example BSPs that classify the presence of a pin or a fork on the board). Because the Othello model was trained to play random legal moves (rather than to play strategically), we do not consider strategy BSPs for Othello.

[2] T. McGrath et al., ‘Acquisition of chess knowledge in AlphaZero’, Proceedings of the National Academy of Sciences, vol. 119, no. 47, Nov. 2022.

Meta-review: The paper proposes new ways to measure interpretability of features from SAEs in the setting of chess and othello based on known , interpretable BSP (board specific positions). The authors show that these metrics reveal things that normal reconstruction loss and L0 loss do not reveal; e.g. that larger SAE models are better than smaller SAE models on this. The authors also use an annealed training of SAE’s where  L^p_p Norm regularization is used and p is annealed from 1 to 0. The authors show that this results in more sparse features that are also better according to the metrics that the authors proposed. 
The reviewers found the paper to be quite interesting. They did raise the concern that it is not clear how this setup might be useful for understanding interpretibility in normal language setting. The authors suggest that their p-annealing procedure produces gains that were not previously measurable and hope that such annealing procedures could be useful elsewhere.
The reviewers pointed out a couple of errors with the subfigures and captions in figure 2 and I hope the authors will fix this in the final version.
Decision: Accept (poster)