Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models

Authors: adam  karvonen
benjamin  wright
rico  angell
jannik  brinkmann
logan  smith
claudio  mayrink verdun
david  bau
samuel  marks
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:13:31

Abstract: What latent features are encoded in language model (LM) representations? Recent work on training sparse autoencoders (SAEs) to disentangle interpretable features in LM representations has shown significant promise. However, evaluating the quality of these SAEs is difficult because we lack a ground-truth collection of interpretable features which we expect good SAEs to identify. We thus propose to measure progress in interpretable dictionary learning by working in the setting of LMs trained on Chess and Othello transcripts. These settings carry natural collections of interpretable features—for example, “there is a knight on F3”—which we leverage into metrics for SAE quality. To guide progress in interpretable dictionary learning, we introduce a new SAE training technique, $p$-annealing, which demonstrates improved performance on our metric.

Main paper: 
 introduction: 
 Mechanistic interpretability aims to reverse engineer neural networks into human-understandable components. What, however, should these components be? Recent work has applied Sparse Autoencoders (SAEs) [9, 16] , a scalable unsupervised learning method inspired by sparse dictionary learning to find a disentangled representation of language model (LM) internals. However, measuring progress in training SAEs is challenging because we do not know what a gold-standard dictionary would look like, as it is difficult to anticipate which ground-truth features underlie model cognition. Prior work has either attempted to measure SAE quality in toy synthetic settings [57] or relied on various proxies such as sparsity, fidelity of the reconstruction, and LM-assisted autointerpretability [6] .In this work, we explore a setting that lies between toy synthetic data (where all ground-truth features are known; cf. Elhage et al. [24] ) and natural language: LMs trained on board game transcripts. This setting allows us to formally specify natural categories of interpretable features, e.g., "there is a knight on e3" or "the bishop on f5 is pinned." We leverage this to introduce two novel metrics for how much of a model's knowledge an SAEs has captured:• Board reconstruction. Can we reconstruct the state of the game board by interpreting each feature as a classifier for some board configuration? • Coverage. Out of a catalog of researcher-specified candidate features, how many of these candidate features actually appear in the SAE?These metrics carry the limitation that they are sensitive to researcher preconceptions. Nevertheless, we show that they provide a useful new signal of SAE quality.Additionally, we introduce p-annealing, a novel technique for training SAEs. When training an SAE with p-annealing, we use an L p -norm-based sparsity penalty with p ranging from p = 1 at the beginning of training (corresponding to a convex minimization problem) to some p < 1 (a non-convex objective) by the end of training. We demonstrate that p-annealing improves over prior methods, giving performance on par with the more compute-intensive Gated SAEs from Rajamanoharan et al. [54] , as measured both by prior metrics and our new metrics.Overall, our main contributions are as follows:1.We train and open-source over 500 SAEs trained on chess and Othello models each.2. We introduce two new metrics for measuring the quality of SAEs.3. We introduce p-annealing, a novel technique for training SAEs that improves on prior techniques.2 Background 
language models for othello and chess: 
 In this work, we make use of LMs trained to autoregressively predict transcripts of chess and Othello games. We emphasize that these transcripts only give lists of moves in a standard notation and do not directly expose the board state. Based on behavioral evidence (the high accuracy of the LMs for predicting legal moves) and prior studies of LM representations [36, 47, 33] we infer that the LMs internally model the board state, making them a good testbed setting for studying LM representations.Othello. Othello is a two-player strategy board game played on an 8x8 grid, with players using black and white discs. Players take turns placing discs on the board, capturing their opponent's discs by bracketing them between their own, causing the captured discs to turn their color. The goal is to have more discs turned to display your color at the end of the game. The game ends if every square on the board is covered or either player cannot make a move.In our experiments, we use an 8-layer GPT model with 8 attention heads and a n = 512 dimensional hidden space, as provided by Li et al. [36] . This model had no prior knowledge of the game or its rules and was trained from scratch on 20 million game transcripts, where each token in the corpus represents a tile on which players place their discs. The game transcripts were synthetically generated by uniformly sampling from the Othello game tree. Thus, the data distribution captures valid move sequences rather than strategic depth. For this model, Li et al. [36] demonstrated the emergence of a world model-an internal representation of the correct board state allowing it to predict the next move-that can be extracted from the model activations using a non-linear probe. Nanda et al. [47] extended this finding, showing that a similar internal representation could be extracted using linear probes, supporting the linear representation hypothesis [46] .Chess. Othello makes a natural testbed for studying emergent internal representations since the game tree is far too large to memorize. However, the rules and state are not particularly complex. Therefore, we also consider a language model trained on chess game transcripts with identical architecture, provided by Karvonen [33] . The model again had no prior knowledge of chess and was trained from scratch on 16 million human games from the Lichess chess game database [38] . The input to the model is a string in the Portable Game Notation (PGN) format (e.g., "1. e4 e5 2. Nf3 ..."). The model predicts a legal move in 99.8 % of cases and, similar to Othello, it has an internal representation of the board state that can be extracted from the internal activations using a linear probe [33] .Figure 1 : We find SAE features that detect interpretable board state properties (BSP) with high precision (i.e., above 0.95). This figure illustrates three distinct chessboard states, each an example of a BSP associated with a high activation of a particular SAE feature. Left: A board state detector identifies a knight on square f3, owned by the player to move. Middle: A rook threat detector indicates an immediate threat posed by a rook to a queen regardless of location and piece threatened. Right: A pin detector recognizes moves that resolve a check on a diagonal by creating a pin, again, regardless of location and piece pinned.Given a dataset D of vectors x ∈ R d , a sparse autoencoder (SAE) is trained to produce an approximationEQUATIONas a sparse linear combination of features. Here, the feature vectors d i ∈ R d are unit vectors, the feature activations f i (x) ≥ 0 are a sparse set of coefficients, and b ∈ R d is a bias term. Concretely, an SAE is a neural network with an encoder-decoder architecture, where the encoder maps x to the vector f = [f 1 (x) . . . f dSAE (x)] of feature activations, and the decoder maps f to an approximate reconstruction of x.In this paper, we train SAEs on datasets consisting of activations extracted from the residual stream after the sixth layer for both the chess and Othello models. At these layers, linear probes trained with logistic regression were accurate for classifying a variety of properties of the game board [33, 47] .For training SAEs, we employ a variety of SAE architectures and training algorithms, as detailed in Section 4.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss our main limitation, the sensitivity of our metrics to human preconceptions, both in the introduction and in a dedicated limitations section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
measuring autoencoder quality for chess and othello models: 
 Many of the features learned by our SAEs reflect uninteresting, surface-level properties of the input, such as the presence of certain tokens. However, upon inspection, we additionally find many SAE features which seem to reflect a latent model of the board state, e.g., features that reflect the presence of certain pieces on particular squares, squares that are legal to play on, and strategy-relevant properties like the presence of a pin in chess (Figures 1 and 6 ).Fortunately, in the setting of board games, we can formally specify certain classes of these interesting features, allowing us to more rigorously detect them and use them to understand our SAEs. In Section 3.1, we specify certain classes of interesting game board properties. Then, in Section 3.2, we leverage these classes into two metrics of SAE quality.We formalize a board state property (BSP) to be a function g : {game board} → {0, 1}. In this work, we will consider the following interpretable classes of BSPs:• G board state contains BSPs which classify the presence of a piece at a specific board square, where the board consists of 8 × 8 squares in both games. For chess, we consider the full board for the twelve distinct piece types (white king, white queen, ..., black king), giving a total of 8 × 8 × 12 BSPs. For Othello, we consider the full board for the two distinct piece types (black and white), yielding 8 × 8 × 2 BSPs. • G strategy consists of BSPs relevant for predicting legal moves and playing strategically in chess, such as a pin detector. They were selected by the authors based on domain knowledge and prior interpretability work on the chess model AlphaZero [45] . We provide a full list of strategy BSPs in Table 3 in the Appendix. Because our Othello model was trained to play random legal moves, we do not consider strategy BSPs for Othello.In this section, we introduce two metrics of SAE quality: coverage and board reconstruction.Coverage. Given a collection G of BSPs, our coverage metric quantifies the extent to which an SAE has identified features that coincide with the BSPs in G. In more detail, suppose that f i is an SAE feature and t ∈ [0, 1] is a threshold, we define the functionEQUATION) where f max i is (an empirical estimate of) max x∼D f i (x), the maximum value that f i takes over the dataset D of activations extracted from our model, and I is the indicator function. We interpret ϕ fi,t as a binary classifier; intuitively, it corresponds to binarizing the activations of f i into "on" vs. "off" at some fraction t of the maximum value of f i on D. Given some BSP g ∈ G, let F 1 (ϕ fi,t ; g) ∈ [0, 1] denote the F1-score for ϕ fi,t classifying g. Then we define the coverage of an SAE with features {f i } relative to a set of BSPs G to beEQUATIONIn other words, we take, for each g ∈ G, the F 1 -score of the feature that best serves as a classifier for g, and then take the mean of these maximal F 1 -scores. An SAE receives a coverage score of 1 if, for each BSP g ∈ G, it has some feature that is a perfect classifier for g. Since Cov depends on the choice of threshold t, we sweep over t ∈ {0, 0.1, 0.2, . . . , 0.9} and take the best coverage score; typically this best t is in {0, 0.1, 0.2}.Board reconstruction. Again, let G be a set of BSPs. Intuitively, the idea of our board reconstruction metric is that, for a sufficiently good SAE, there should be a simple, human-interpretable way to recover the state of the board from the profile of feature activations {f i (x)} on an activation x ∈ R d .Here, the activation x was extracted after the post-MLP residual connection in layer 6.We will base our board reconstruction metric around the following human-interpretable way of recovering a board state from a feature activation profile; we emphasize that different ways of recovering boards from feature activations may lead to qualitatively different results. This recovery rule is based on the assumption that interpretable SAE features tend to be high precision for some subset of BSPs, in line with Templeton et al. [58] . For example, features that classify common configurations of pieces are high precision (but not necessarily high recall) for multiple BSPs. We use a consistent dataset of 1000 games as our training set D train for identifying high-precision features across all Board State Properties (BSPs). An additional, separate set of 1000 games serves as our test set D test . Using the training set D train , we identify, for each SAE feature f i , all of the BSPs g ∈ G for which ϕ fi,t is a high precision (of at least 0.95) classifier. Then for each g ∈ G our prediction rule isEQUATIONLet F 1 (P({f i (x)}); b) denote the F 1 -score for a given board state b, where P({f i (x)}) = {P g ({f i (x)})} g∈G represents the full predicted board (containing predictions for all 64 squares) obtained from the SAE activations. 3Then, the average F 1 -score over all board states in the test dataset D test can be calculated asRec({x i }, D test ) = 1 |D test | x∈Dtest max t F 1 (P({f i (x)}); b).(5)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide the relevant code to reproduce the main experimental results of the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
training methodologies for saes: 
 In our experiments, we investigate four methods for training SAEs, as explained in this section. These are given by two autoencoder architectures and two training methodologies-one with p-annealing and one without p-annealing-for each architecture. Our SAEs are available at https://huggingface.co/adamkarvonen/chess_saes/tree/main (chess) and https:// huggingface.co/adamkarvonen/othello_saes/tree/main (Othello).Let n be the dimension of the model's residual stream activations that are input to the autoencoder, m the autoencoder hidden dimension, and s the dataset size. Our baseline "standard" SAE architecture, as introduced in Bricken et al. [9] is defined by encoder weights W e ∈ R m×n , decoder weightsEQUATIONwhere f (x) is the vector of feature activations, and x is the reconstruction. For a standard SAE, our baseline training method is as implemented in the open-source dictionary_learning repository [43] , optimizing the lossEQUATIONfor some hyperparameter λ > 0 controlling sparsity.The L 1 penalty used in the original training method encourages feature activations to be smaller than they would be for optimal reconstruction [62] . To address this, Rajamanoharan et al. [54] introduced a modification to the original SAE architecture that separates the selection of dictionary elements to use in a reconstruction and estimating the coefficients of these dictionary elements. This results in the following gated architecture: 11) where I[• > 0] is the Heaviside step function and ⊙ denotes elementwise multiplication. Then, the loss function uses xfrozen , a frozen copy of the decoder:EQUATIONEQUATIONEQUATION+ ∥x -xfrozen (ReLU(π gate (x)))∥ 2 2 .Fundamentally, an L 1 penalty has been used to induce sparsity in SAE features because it serves as a convex relaxation of the true sparsity measure, the L 0 -norm. The L 1 -norm is the convex hull of the L 0 -norm, making it a tractable alternative for promoting sparsity [63] . However, the proxy loss function is not the same as directly optimizing for sparsity, leading to issues such as feature shrinkage [62] and potentially less sparse learned features. Unfortunately, the L 0 -norm is non-differentiable and directly minimizing it is an NP-hard problem [48, 18] , rendering it impractical for training. In this work, we propose the use of nonconvex L p p -minimization, with p < 1, as an alternative to the standard L 1 minimization in sparse autoencoders (SAEs). This approach has been successfully employed in compressive sensing and sparse recovery to achieve even sparser representations [11, 61, 64, 60] . To perform this optimization, we introduce a method called p-annealing for training SAEs, based on the compressive sensing technique called p-continuation [66] . The key idea is to start with convex L 1 -minimization through setting p = 1 and progressively decrease the value of p during training, resulting in closer approximations of the true sparsity measure, L 0 , as p approaches 0. We define the sparsity penalty for each batch x as a function of the current training step s:EQUATIONIn other words, the sparsity penalty will be a scaled L p p norm of the SAE feature activations, with p decreasing over time. At p = 1, the L p p norm is equal to the L 1 norm, and as p → 0, the L p p norm limits to the L 0 -norm, aslim p→0 i f i (x) p = i f i (x) 0 .The purpose of annealing p from 1 → 0 instead of starting from a fixed, low value for p is that the lower the p, the more concave (non-convex) the L p p norm is, increasing the likelihood of the training process getting stuck in local optima, which we have observed in initial experiments. Therefore, we aim to first arrive at a region of an optimum using the easier-to-train L 1 penalty and then gradually shift the loss function. This manifests as keeping p = 1 for a certain number of steps and then starting decreasing p linearly down to p end > 0 at the end of training. We set p end = 0.2.Coefficient Annealing. Changing the value of p changes the scale of the L p p norm. Without also adapting the coefficient λ, the strength of the sparsity penalty would vary too wildly across training. Empirically, we found that keeping a constant λ would lead to far too weak of a sparsity penalty for the larger p's at the start of training, making the process worse than simply training with a constant p from the beginning. Consequently, we aim to adapt the coefficient λ such that the strength of the sparsity penalty is not changed significantly due to p updates. Formally, the update step is:EQUATIONWe keep a queue of the most recent q batches of feature activations mid-training and use them to calibrate the λ s updates. Therefore, the strength of the sparsity penalty is kept locally constant.Combining p-annealing with other SAEs. Since the p-annealing method only modifies the L 1 terms in the loss function without affecting the SAE architecture, it is simple to combine p-annealing with other SAE modifications. This allows us to create the Gated-Annealed SAE method by combining the Gated SAE architecture and p-annealing. Concretely, we modify L gated (Equation 12) by replacing the sparsity term λ∥ReLU(π gate (x))∥ 1 in with λ s ∥ReLU(π gate (x))∥ ps ps . Our experiments showed that the optimum values for coefficients λ and λ s differ. 
results: 
 In this section, we explore the performance of SAEs applied to language models trained on Othello and chess. Consistent with Nanda et al. [47] , we find that interpretable SAE features typically track properties relative to the player whose turn it is (e.g. "my king is pinned" rather than "the white king is pinned"). To side-step subtleties arising from this, we only extract our activations from the token immediately preceding white's move. Specifically, we consider SAEs trained on the residual stream activations after the sixth layer using the four methods from Section 4 (see Table 2 for additional hyperparameters). In addition to our metrics introduced above, we also make use of unsupervised metrics previously appearing in the literature [9, 16, 54] :• L 0 measures the average number of active SAE active features (i.e., positive activation) on a given input. • Loss recovered measures the change in model performance when replacing activations with the corresponding SAE reconstruction during a forward pass. This metric is quantified as(H * - H 0 )/(H orig -H 0 ), where H orig is the cross-entropy loss of the board game model for next-token prediction, H * is the cross-entropy loss after substituting the model activation x with its SAE reconstruction during the forward pass, and H 0 is the cross-entropy loss when zero-ablating x.Our key takeaways are as follows.SAE features can accurately reconstruct game boards. In general, we find that SAE features are effective at capturing board state information in both Othello and chess (see Table 1 , Figure 2d and 4d ). In contrast, SAEs trained on a model with random weights perform very poorly according to our metrics, showing that SAE performance is driven by identifying structure in the models' learned representation of game boards. Nonetheless, SAEs do not match the performance of linear probes in terms of reconstructing the board state. This performance gap suggests that SAEs do not capture all of the information encoded in the model's internal representations.Standard SAEs trained with p-annealing perform on par with Gated SAEs. We find that standard SAEs trained using p-annealing consistently perform better than those trained with a constant L 1 penalty (Equation 8), as measured by existing proxy metrics and in terms of improvement in coverage (see Figure 2a and 4a ). In fact, standard SAEs trained using p-annealing show a coverage score that is comparable to Gated SAEs trained without p-annealing. Further, we find that both p-annealing and Gated SAEs significantly outperform Standard SAEs in addressing the shrinkage problem [62] , as detailed in Appendix E. However, we find cases where our coverage metric disagrees with existing metrics. In Figure 2 , for example, Gated SAEs perform achieve a higher loss recovered score than Standard SAEs trained using p-annealing. We emphasize that the training and inference of Gated SAEs is more computationally expensive, requiring 50% more compute per forward pass compared to Standard SAEs [54] .Coverage and board reconstruction reveal differences in SAE quality not captured by unsupervised metrics. Our metrics reveal improvements in SAE performance that traditional proxy metrics fail to capture. For example, we trained SAEs with hidden dimensions 4096 and 8192 (expansion factors of 8 and 16, respectively). We expect the SAEs with 8192 hidden dimensions to perform better since they have greater capacity. However, we observe that they perform equally well according to prior unsupervised metrics (see Figures 2 a, c and 4 a, c ). In contrast, our metrics reveal that SAEs with larger hidden dimensions are better. For the Standard architecture, this is reflected by the parallel lines (of purple diamonds) in Figures 2 b, d and 4 b, d. Thus, our metrics are able to capture improvements from larger expansion factors. In addition, we find that the performance of p-annealing closely resembles that of Gated SAEs when evaluated using standard proxy metrics; it demonstrates clear improvements under our proposed metrics. Coverage and board reconstruction are consistent with existing metrics. Figures 2, 4 , and 3 demonstrate that both coverage and board reconstruction metrics are optimal in the elbow region of the Pareto frontier. This region, where SAEs reconstruct internal activations efficiently with minimal features, also yielded the most coherent interpretations during our manual inspections. This provides precise, empirical validation to the common wisdom that SAEs in this region of the Pareto frontier are the best. 
limitations: 
 The proposed metrics for board reconstruction and coverage provide a more objective evaluation of SAE quality than previous subjective methods. Nevertheless, these metrics exhibit several limitations. Primarily, their applicability is confined to the chess and Othello domains, raising concerns about their generalizability to other domains or different models. Additionally, the set of BSPs that underpin these metrics is determined by researchers based on their domain knowledge. This approach may not encompass all pertinent features or strategic concepts, thus potentially overlooking essential aspects of model evaluation. Developing comparable objective metrics for other domains, such as natural language processing, remains a significant challenge. Moreover, our current focus is on evaluating the quality of SAEs in terms of their ability to capture internal representations of the model. However, this does not directly address how these learned features could be utilized for downstream interpretability tasks.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We discuss the training process in the main section and provide additional information in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
related work: 
 Sparse dictionary learning. Since the nineties, dictionary learning [22, 20] , sparse regression [26] , and later, sparse autoencoders [49] have been extensively studied in the machine learning and signal processing literature. The seminal work of Olshausen and Field [51] introduced the concept of sparse coding in neuroscience (see also [52] , building upon the earlier concept of sparse representations [19] and matching pursuit [42] . Subsequently, a series of works established the theoretical and algorithmic foundations of sparse dictionary learning [23, 30, 21, 1, 65, 32, 59, 2, 5, 7, 10] . Notably, Gregor and LeCun [28] introduced LISTA, an unrolled version of ISTA [17] that learns the dictionary instead of having it fixed.In parallel, autoencoders were introduced in machine learning to automatically learn data features and perform dimensionality reduction [29, 37] . Inspired by sparse dictionary learning, sparse autoencoders [49, 14, 13, 41, 35] were proposed as an unsupervised learning model to build deep sparse hierarchical models of data, assuming a certain degree of sparsity in the hidden layer activations. Later, Luo et al. [39] generalized sparse autoencoders (SAEs) to convolutional SAEs. Although the theory for SAEs is less developed than that of dictionary learning with a fixed dictionary, some progress has been made in quantifying whether autoencoders can, indeed, do sparse coding, e.g., Arpit et al. [4] , Rangamani et al. [55] , Nguyen et al. [50] .Feature disentanglement using sparse autoencoders. The individual computational units of neural networks are often polysemantic, i.e., they respond to multiple seemingly unrelated inputs [3] . Elhage et al. [24] investigated this phenomenon and suggested that neural networks represent features in linear superposition, which allows them to represent more features than they have dimensions. Thus, in an internal representation of dimension n, a model can encode m ≫ n concepts as linear directions [53] , such that only a sparse subset of concepts are active across all inputs -a concept deeply related to the coherence of vectors [26] and to frame theory in general [12] . To identify these concepts, Sharkey et al. [57] used SAEs to perform dictionary learning on a one-layer transformer, identifying a large (overcomplete) basis of features. Cunningham et al. [16] applied SAEs to language models and demonstrated that dictionary features can be used to localize and edit model behavior. Marks et al. [44] proposed a scalable method to discover sparse feature circuits, as opposed to circuits consisting of polysemantic model components, and demonstrated that a human could change the generalization of a classifier by editing its feature circuit. Recently, Kissane et al. [34] explored autoencoders for attention layer outputs. These works have benefited from a variety of open-source libraries for training SAEs for LLM interpretability [43, 8, 15] .Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We present results across a large sweep of training parameters to provide information about the significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Our work focuses on evaluating current methods to extract features from neural networks, and proposes a method to improve this process. We do not believe that this work has a direct impact on society. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 Most SAE research has relied on proxy metrics such as loss recovered and L 0 , or subjective manual evaluation of interpretability by examining top activations. However, proxy measures only serve as an estimate of interpretability, monosemantic nature, and comprehensiveness of the learned features, while manual evaluations depend on the researcher's domain knowledge and tend to be inconsistent.Our work provides a new, more objective paradigm for evaluating the quality of an SAE methodology; coverage serves as a quantifiable measure of monosemanticity and quality of feature extraction, while board reconstruction serves as a quantifiable measure of the extent to which an SAE is exhaustively representing the information contained within the language model. Therefore, the optimal SAE methodology can be judged by whether it yields both high coverage and high board reconstruction.Finally, we propose the p-annealing method, a modification to the SAE training paradigm that can be combined with other SAE methodologies and results in an improvement in both coverage and board reconstruction over the Standard SAE architecture. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not introduce new datasets or models that pose such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators of original assets (models and data) are properly cited in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We provide a Readme as well as in-code documentations alongside the code.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Despite the success of SAEs at extracting human-interpretable features, they fail to perfectly reconstruct the activations [16] . One challenge in the training of SAEs with an L 1 penalty is shrinkage (or 'feature suppression'); in addition to encouraging sparsity, an L 1 penalty encourages feature activations to be smaller than they would be otherwise. Wright and Sharkey [62] approached this problem by fine-tuning the sparse autoencoder without a sparsity penalty. Appendix E further quantifies shrinkage across a suite of SAEs trained on chess and Othello models. Jermyn et al. [31] and Riggs and Brinkmann [56] explored alternative sparsity penalties to reduce feature suppression during training. Rajamanoharan et al. [54] introduced Gated SAEs, an architectural variation for the encoder which both addresses shrinkage and improves on the Pareto frontier of L 0 vs reconstruction error. Recently, Gao et al. [27] systematically evaluated the scaling laws with respect to sparsity, autoencoder size, and language model size.The goal of dictionary learning in machine learning is to produce human-interpretable features and capture the underlying model's computations [9] . However, quantitatively measuring interpretability is difficult and often involves manual inspection. Therefore, most existing work assesses the quality of SAEs along different proxy metrics: (1) The cross-entropy loss recovered, which reflects the degree to which the original loss of the language model can be recovered when replacing activations with the autoencoder predictions. ( 2) The L 0 -norm of feature activations E z∼D ∥h(z)∥ 0 , measuring the number of activate features given an input [25] . Makelov et al. [40] proposed to compare SAEs against supervised feature dictionaries in a natural language setting. However, this requires a significant understanding of the model's internal computations and is thus not scalable.We used a single NVIDIA A100 GPU for training SAEs and experiments. It takes much less than 24 hours to train a single SAE on 300 million tokens. Given a trained SAE, our evaluation requires less than 5 minutes of computing time. In Figure 3 , we present a mean coverage score over strategy board state properties G BSP . Properties within G BSP vary significantly in complexity. For example, queen detection can be inferred directly from the move history, while fork detection requires an accurate representation of the board state. Table 4 shows that linear probe F1-score is below 0.95 for 6 out of 15 properties in G BSP . This suggests that chess-GPT [33] does not represent these properties linearly. Additional experiments are required to determine whether the representation is present at all.For the board state case, reconstruction is significantly higher than coverage. This is because there are many SAE features that are high precision classifiers for a configuration of squares, such as "white pawn on e4, white knight of f3". In cases where coverage is higher than reconstruction (such as for can_check), it is because there are not many features that are over 95% precision for "there is a check move available" from which we can recover if there is an available check move. Coverage is significantly higher because there is at least one feature that has an F 1 -score of 0.54 for can_check, which may not have a precision greater than 95%. Training Standard SAEs with an L 1 penalty, as described in Section 4, causes a systematic underestimation of feature activations. Wright and Sharkey [62] term this phenomenon shrinkage. Following Rajamanoharan et al. [54] , we measure the relative reconstruction bias γ of our SAEs, defined as:EQUATIONHere, D denotes a large dataset of model internal activations. Intuitively, γ < 1 indicates shrinkage. A perfectly unbiased SAE would have γ = 1.Our experiments show that p-annealing achieves similar relative reconstruction bias improvements to gated SAEs, both outperforming the baseline architecture. Figure 5 shows that improvements manifest differently across domains: Chess SAEs show a narrower range of bias (γ ≈ 0.98) compared to Othello (γ ≈ 0.80). This domain-dependent variation may reflect differences in the underlying models or data distributions. We observe unstable γ values for SAEs with L0 near zero, which represent degenerate cases outside the typical operating range of these models. Previous research of Othello-playing language models found that the model learned a nonlinear model of the board state [36] . Further investigation found a closely related linear representation of the board when probing for "my color" vs. "opponent's color" rather than white vs. black [47] . Based on these findings, when measuring the state of the board in Othello, we represent squares as (mine, yours) rather than (white, black).Similar to Othello models, prior studies of chess-playing language models found the same property, where linear probes were only successful on the objective of the (mine, yours) representation and were unsuccessful on the (white, black) representation [33] . They measured board state at the location of every period in the Portable Game Notation (PGN) string, which indicates that it is white's turn to move and maintain the (mine, yours) objective. Some characters in the PGN string contain little board state information as measured by linear probes, and there is not a clear ground truth board state part way through a move (e.g., the "f" in "Nf3"). We follow these findings and measure the board state at every period in the PGN string.When measuring chess piece locations, we do not measure pieces on their initial starting location, as this correlates with position in the PGN string. An SAE trained on residual stream activations after the first layer of the chess model (which contains very little board state information as measured by linear probes) obtains a board reconstruction F 1 -score of 0.01 in this setting. If we also measure pieces on their initial starting location, the layer 1 SAE's F 1 -score increases to 0.52, as the board can be mostly reconstructed in early game positions purely from the token's location in the PGN string.Masking the initial board state and blank squares decreases the F 1 -score of the linear probe from 0.99 to 0.98.We present two additional examples of learned SAE features that we (subjectively) match to board state properties based on their maximally activating input PGN-strings in Figure 6 .(a) "En passant available" detector learned by an SAE.(b) "Knight on e2 or e7" detector learned by an SAE. (c) Another example of the "Knight on e2 or e7" detector shown in subfigure (b) above. We interpret this feature as representing a mirrored perspective. It may be firing for "opponent knight on e column, 1 square away from opponent back rank", which is why it fires for both e2 (if black's turn to move) and e7 (if white's turn to move). Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Code, models, and data are available at https://github.com/adamkarvonen/SAE_BoardGameEvalWe do not score empty squares. Thus, the reconstruction score would be zero if no high precision features are active. 


Decision: The paper proposes new ways to measure interpretability of features from SAEs in the setting of chess and othello based on known , interpretable BSP (board specific positions). The authors show that these metrics reveal things that normal reconstruction loss and L0 loss do not reveal; e.g. that larger SAE models are better than smaller SAE models on this. The authors also use an annealed training of SAE’s where  L^p_p Norm regularization is used and p is annealed from 1 to 0. The authors show that this results in more sparse features that are also better according to the metrics that the authors proposed. 
The reviewers found the paper to be quite interesting. They did raise the concern that it is not clear how this setup might be useful for understanding interpretibility in normal language setting. The authors suggest that their p-annealing procedure produces gains that were not previously measurable and hope that such annealing procedures could be useful elsewhere.
The reviewers pointed out a couple of errors with the subfigures and captions in figure 2 and I hope the authors will fix this in the final version.

Reviews: review: paper_summary: paper_summary: The paper introduces a setting for evaluating dictionary learning methods (and in particular sparse autoencoders) for language model interpretability. The setting proposed is that of interpreting features learned by language models trained on data representing Chess and Othello games. This setting should allow testing based on a more easily enumerable set of ground truth features (compared to natural language) which we know are relevant to the task and are captured by the underlying model.

The authors propose two new F1 based metrics to evaluate the SAEs ability to recover features learned by the model, namely a "coverage" metric and a "board reconstruction" metric that measures the quality of high precision features in the SAE. Both these metrics are computed by treating SAE features as classifiers for binary board state properties.

The paper also introduces a new training technique for SAEs called p-annealing that modifies the L1 loss to more closely approximate optimizing the L0 norm for which L1 loss acts as a proxy.
strength_weakness: strength_weakness: - The figures and caption for figure 2 don't seem to match up (there may be a mixup with subfigures for figure 3). From reading the caption I would have expected the bottom row to show "Coverage" for high level BSPs in a similar pattern to the top row, however it shows "Reconstruction" of high level BSPs and reconstruction of low-level BSPs (which seem to appear as subfigures in figure 3). Could the authors correct this/post the appropriate charts?

- I can't easily understand the comparison of the new metrics compared to the existing proxy metrics, the figures (2-5) seem to attempt to both validate the new metrics as well as compare the proposed p-annealing technique and thus get fairly busy making them hard to parse visually:
    - The 3 variable plots on the left hand column of the figures are somewhat hard to read, especially since the proposed metric is encoded using color. Having a plot comparing each proxy metric (loss recovered and sparsity) to each proposed metrics would be easier to read. I do understand that the authors are trying to demonstrate that there is a region on the Pareto frontier where their metrics perform best, its not so apparent to me that sparser SAEs are always more "interpretable" SAEs so it is an interesting finding, but I still think it would be helpful to more easily compare each metric to the existing proxy metrics (for example I wouldn't necessarily expect that less sparse SAEs should have worse coverage—but maybe the authors have some thoughts on why that should be the case?)
    - Having separate plots that focus on evaluating the p-annealing technique using the various metrics (proxy and proposed) when also be helpful (i.e. similar to Fig 5 in Rajamanoharan et al 2024). The overplotting of the various shapes for the different conditions (particularly in the left hand column) makes it a bit hard to understand the the trends and understand how well p-annealing works actually works.
- From my read of the charts, the results for p-annealing seem quite different between Othello and Chess. I'm trying to map the final sentence about the efficacy p-annealing in the conclusion to the charts and i'm not sure if i'm able to see how the conclusion follows from the results. Could the authors comment on this?

- There isn't much description of qualitative/manual analysis of the trained SAEs alongside the new metrics. Because measurement is relatively nascent in this space, it would be helpful to have some qualitative evaluation/results from manual inspection demonstrated.
questions: questions: - What is functionally/representationally different about BSPs in F_low vs F_high? "Any square being threatened" seems as high level as "has a queen on the board". I'm just trying to understand if there is a critical difference between these two classes of BSP from the perspective of the SAE or BSP classifier. It is nice that these concepts come from different sources, just wondering about the low-level vs high level implication.
- What is the effect on the board reconstruction metric if there are BSPs for which there are no high precision features (line 131)?  Are these low precision features still used in calculating the final metric? It was not super clear to me from the notation in the equation below line 134 if **all** features are used used in this calculation, _but_ φfi,t(x) = 1 is only calculated for high precision features, or if **only** high precision features are included in this computation at all?
- Why use a "batch of train data" to identify the high precision features? Why not use the whole dataset? How big should this batch be relative to the whole dataset?
- Have the authors done any qualitative analysis of their metrics?
	- For example board reconstruction seems to overall be lower for high-level vs low-level BSPs. Have the authors observed that this translates into more difficulty interpreting board states retrieves by featured for the high level BSPs?
- How many SAEs are trained for each of the 4 SAE training conditions {Gated, Standard} x {p-annealing, no p-annealing}?
- Could you elaborate on line 220 where you say "in the region with L0 below 100, which is the optimal range according to proxy metrics"? Is this a reference to an existing finding, or how do we know that this is the optimal range for sparsity?

A more speculative question:
- Do the authors know if there is any correlation between their proposed metrics and the final reconstruction loss of the SAE? Particuarly since the linear probe performance on predicting BSPs seems to be high, might we expect to see a correlation to just straight reconstruction loss as another signal to validate these metrics or am I off base?
limitations: limitations: Yes
paper_summary: paper_summary: This manuscript applies sparse autoencoders (SAE) to detect interpretable features from autoregressive transformer models trained on Othello and chess. These controlled scenarios provide suitable testbeds, in the sense that we can extract ground truth features to measure progress in dictionary learning. Based on the observations that existing SAE optimizations may indeed be suboptimal, the authors propose *p-annealing*, a warm-start technique that iteratively optimize a sequence of (increasingly more non-convex) SAE objectives, in order to approximate the intractable $\ell_0$ objective.
strength_weakness: strength_weakness: * While it is reasonable to operate in a controlled setup (i.e. Othello and chess) in this paper, there lacks concrete evidence on how these improved techniques can potentially improve SAEs in LLMs and natural language domains. Including a discussion and/or additional experiments on this could further strengthen the paper.
questions: questions: * How sensitive is the SAE training to the choice of $p$-annealing schedule and other choice of hyperparameters?
limitations: limitations: The authors have adequately addressed limitations and potential societal impacts of their work
paper_summary: paper_summary: This paper proposes two measures to reflect the quality of the trained sparse auto-encoder when decomposing superposition features. In addition, this paper achieves approximate L0 optimization through dynamic adjustment of the p-norm and testing it on two types of board games.
strength_weakness: strength_weakness: - The experimental objectives of the article are not clear: The article lacks of comparing the advantages of the two proposed measurement methods compared to reconstruction loss and sparsity. For example, demonstrating cases where both reconstruction loss and sparsity perform well but coverage and reconstruction are low, resulting in biases in downstream tasks (such as circuit discovery).
- The rationality of the two proposed measurement methods in the article is questionable: The purpose of using SAE is to decompose the features of superposition into monosemantic features, with the expected decomposition results leaning towards several low-level features. However, based on the experimental results in Figure 2, there is no significant difference between low-level and high-level coverage results. This phenomenon is likely due to the unreasonable selection of rules.
- Similarly, due to the fact that both coverage and reconstruction are ultimately detected using linear probing on individual features, some high-level features require linear combinations of multiple low-level features to be expressed [1,2,3]. If the measurement methods described in the article are used, it will not be effective in detecting these features.
- Typos (not limited to): Line 154  "than than" should be corrected.
- The image captions are incorrect: Figure 2 should be labeled "coverage" instead of "reconstruction" for the bottom row.


[1] Templeton, Adly, et al. "Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet." Transformer Circuits Thread (2024).

[2] He, Zhengfu, et al. "Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT." arXiv preprint arXiv:2402.12201 (2024).

[3] Rajamanoharan, Senthooran, et al. "Improving dictionary learning with gated sparse autoencoders." arXiv preprint arXiv:2404.16014 (2024).
questions: questions: - Is there a separate training of SAE for MLPs, Attention outputs, etc., which write into the residual stream instead of activations between attention blocks?
- Is there any relevant reference for the assumption about *High Precision* in line 130?
- How were the hyperparameters adjusted in Figures 2, 3, 4, and 5 to obtain several different SAEs? (Causality between hyperparameters and SAE results needs further clarification [1].)
- In [2], it's mentioned that Gated SAE can solve the shrinkage problem mentioned in line 167. Have you compared the relative reconstruction bias (gamma) proposed in [2] when using gated SAE w/wo p-annealing?

[1] Kramár, János, et al. "AtP*: An efficient and scalable method for localizing LLM behaviour to components." arXiv preprint arXiv:2403.00745 (2024).

[2] Rajamanoharan, Senthooran, et al. "Improving dictionary learning with gated sparse autoencoders." arXiv preprint arXiv:2404.16014 (2024).
limitations: limitations: I suggest the authors expand the scope of the research to include more domains and datasets, and validate the effectiveness of the proposed methods in other downstream interpretability tasks.

Comments: title: Reply for Authors
Reply for Authors
Any final questions?
comment: Thanks to the author for the reply, the two points listed answered my question, so I will increase my score.
The reviewer summarizes our original response as

> The author believes that the existing L0 and reconstruction loss cannot accurately measure the training results of SAE (I also agree), Therefore, the author proposed two new measurement methods and observed that the extreme value area of ​​the new measurement method is at the Pareto front of the original measurement method, so the two new measures are considered to be better than the original method.

To be clear, while it is true that one of our results is that our new supervised metrics obtain their best values in the "elbow" region of the Pareto frontier for unsupervised proxy metrics (L0 and loss recovered), this is **not** the reason we claim our supervised metrics improve on prior unsupervised metrics. We include this result as a "sanity check" to show that our metrics do not totally diverge from prior notions of SAE quality. (And we agree that—as the reviewer's "harmonic averaging" example shows—this result alone leaves open the possibility that our supervised metrics contribute nothing new.)

Rather, the reasons our metrics improve on prior measures of SAE quality are:

1. (Empirical result) Our metrics show differences between SAEs which are invisible to prior unsupervised metrics (see the pdf attached to our general response). As discussed in our original response, an especially striking example is given by the parallel curves that appear in our plots: our metrics clearly show that SAEs with a larger hidden dimension are better, but prior unsupervised metrics failed to show this.

2. (Theoretical property) Our metrics are supervised—based on researcher operationalization of what counts as an "interpretable" feature and which features we expect good SAEs to learn—whereas prior SAE quality measures (L0 and loss recovered) are unsupervised.
I appreciate the authors' response. My main question focuses on the first point. Let me briefly explain my understanding of the first question. 
> The author believes that the existing L0 and reconstruction loss cannot accurately measure the training results of SAE (**I also agree**), Therefore, the author proposed two new measurement methods and observed that the extreme value area of ​​the new measurement method is at the Pareto front of the original measurement method, so the two new measures are considered to be better than the original method.

But this only shows that the new measurement method considers L0 and reconstruction loss at the same time (similar to harmonic averaging), but does not reflect the loopholes of the original measurement (for example, different training settings obtained two groups of L0 and reconstruction loss with the same SAE, but the subsequent performance of these two groups of SAE is very different). And this is also my main concern with this article. Therefore, I maintain my rating. 
And I am open to revising my rating based on discussions with other reviewers/ AC.
Dear reviewers,
The discussion period is ending soon. If you have additional questions for the authors, please post them now.

One of the questions that two of the reviewers raised was the paper does not address enough domains, especially not natural language and so the utility of the approach may be harder to judge. 

Reviewer pBfq, your rating differs significantly from the others -- have the author's responses addressed any of your concerns ? 
Thanks.
Thank you for these feedbacks. I maintain my current score and look forward to seeing future works on applying this type of SAEs to natural language setups.
Thanks for your clarifications and updated figures. I have updated my score.