Evaluating SAE interpretability without generating explanations
Authors: Gonçalo Paulo
Nora Belrose
Venue: ICLR 2026 Poster (2026)
Keywords: interpretability
explanation
sae
transcoder
Primary area: interpretability and explainable AI
TL;DR: Instead of evaluating whether explanations match activating contexts, we evaluate how much are activating contexts similar between themselves.

Abstract: Sparse autoencoders (SAEs) and transcoders have become important tools for machine learning interpretability. However, measuring how interpretable they are remains challenging, with weak consensus about which benchmarks to use. Most evaluation procedures start by producing a single-sentence explanation for each latent. These explanations are then evaluated based on how well they enable an LLM to predict the activation of a latent in new contexts. This method makes it difficult to disentangle the explanation generation and evaluation process from the actual interpretability of the latents discovered. In this work, we adapt existing methods to assess the interpretability of sparse coders, with the advantage that they do not require generating natural language explanations as an intermediate step. This enables a more direct and potentially standardized assessment of interpretability. Furthermore, we compare the scores produced by our interpretability metrics with human evaluations across similar tasks and varying setups, offering suggestions for the community on improving the evaluation of these techniques.

EVALUATING SAE INTERPRETABILITY WITHOUT EX- PLANATIONS
Gonçalo Paulo, Belrose
EleutherAI
(goncale o, nora) Beleuther.ai
ABSTRACT
Sparse autoencoders (SAEs) and transcoders have become important tools for ma- chine learning interpretability. However, measuring how interpretable they are re- mains challenging, with weak consensus about which benchmarks to use. Most evaluation procedures start by producing single-sentence explanation for each latent. These explanations are then evaluated based on how well they enable an LLM to predict the activation of a latent in new contexts. This method makes it difficult to disentangle the explanation generation and evaluation process from the actual interpretability of the latents discovered. In this work, adapt existing methods to assess the interpretability of sparse coders, with the advantage that they do not require generating natural language explanations an intermediate step. This enables a more direct and potentially standardized assessment of inter- pretability. Furthermore, we compare the scores produced by our interpretability metrics with human evaluations across similar tasks and varying setups, offering suggestions for the community on improving the evaluation of these techniques.
INTRODUCTION
Sparse autoencoders (SAEs) and transcoders now popular tools for interpreting large language models (Lieberum et al., 2024 Gao et al.) 2024 Templeton et al. 2024). vision models (Surkov)
al. 2024] Gorton 2024) and other neural networks (Adams et al., 2025 Le et al., 2024). Naively interpreting the activations of neural networks tends to fail due to pol polysemanticity (Mu & Andreas, 2020 /hang & Wang 2023) neurons usually fire in diverse, seemingly unrelated contexts. SAES aim to overcome polysemanticity by representing each hidden state using an overcomplete basis, while requiring the coefficients of the linear combination to be sparse and non-negative. This allows SAE basis vectors (often called latents) to learn more specific and interpretable latents than those captured by neurons. Initial works showed that the latents of SAES have very interpretable activa- tions (Cunningham al.) (023),, specially in the highest quantiles. Recent work (Karvonen al.) 2025) has focused on generating good benchmarks that go beyond minimizing the reconstruction loss and increasing sparsity, instead focusing possible downstream applications and addressing pathologies found in some SAEs, like feature absorption (Chanin et al., 2024).
The most popular methods for evaluating SAE interpretability involves generating a natural language explanation for each latent. This explanation is then evaluated by how useful it is for predicting the activation of the latent, the effects of intervening on the latent, in a given context (Templeton) et al. 2024 Paulo et al. 20244; Gur-Arieh et al. 2025). The use of natural language explanations is problematic, however.
complicates the evaluation pipeline, introducing additional hyperparame- ters, such as how many examples to show the explainer model and which type of examples to show for instance. It also requires the the exploration of different prompting techniques which are likely to affect the final results, for instance allowing the model to do Chain of Though (Wei et al. (2022), how to highlight the activating tokens as well as on which format to present the explanation as a short sentence, longer explanation with examples, something in between. Ideally, would like to abstract away from the details of explanation generation, focusing on the properties of the latents themselves.
The asplanation-centrrt approach also assumes that, in order for a latent to be interpretable, must have meaning succinetly expressible in words (Ayonrinde et al. 2024 Paulo & Belrose 2025). We

---

question this philosophical assumption: we deem latent interpretable if human can distinguish activating from con-activating examples, without producing an explanation. Given this new defini- tion of interpretability, do not require generating natural language explanations for latents. Our methods are applicable to both human and LLM evaluators, and we compare the scores produced by LLMs to those produced by humans
introduce two methods for assessing sparse coder interpretability which
2 RELATED WORK
2.1 EVALUATING INTERPRETABILITY
We consider human evaluation to be the gold standard for interpretabilityy and LLM evaluators are valid only insofar as their judgments correlate strongly with those of humans. Chang et al. (2009)
introduced the word intrusion task, where human participants identify an out-of-place word within set of words associated with a specific topic discovered by a topic models. /Subraaanian et al. (2018) apply this approach to sparse word embeddings. The two alternative forced choice task (Borowski et al. 2020 Zimmermann et al. 2023) asks humans to discriminate between pairs of images where one strongly activates feature map in a convolutional neural network, and one image does A model for which humans achieve high accuracy on these tasks, as well as a high inter-participant agreement rate, can then be considered interpretable.
Most methods of evaluating SAE interpretability involve generating an explanation for each latent, then evaluating the usefulness of these explanations for predicting whether, and/or how strongly, the latent will activate in given context. This is similar to the simulation scoring method proposed in Bills al. (2023) for neuron- based interpretability. However, this approach is entirely correlational, and may not capture the causal effects of latent activations on the model's 's output. Causal evaluations involve intervening on the model along latent direction, and measuring how well the effect of this intervention can be predicted given natural language explanation (Paulo al. 2024 cct al. 2025).
Gur-Arich
If SAE latents strongly predictable by humans, we might imagine that we could replace the true latent activations from the SAE encoder with predicted activations from a human or LLM during a forward pass, and observe little performance degradation relative to the pristine model. This was attempted by Paulo & Belrose (2025), with mostly negative results. The idea is similar to concept bottleneck models (Koh al. 2020), where the concepts learned to explain an existing model, instead of learned as the model is trained.
2.2
AUTOMATIC EXPLANATION GENERATION AND EVALUATION
The standard method for generating explanations for latents involves collecting activations over a large corpus of text (Bills al., 2023). Contexts that trigger the activation of given latent are then presented to an LLM, which is tasked with summarizing them. Different strategies can be employed when sampling examples show the LLM, such as stratified sampling from selected quantiles in the activation distribution (Templeton et al. 20244] Paulo et al. 2024).
Explanations can be evaluated via simulation, where an LLM is asked to predict latent's activation strength at a specific token in a sentence given the explanation (Bills et al.) 2023). However, this technique is computationally expensive and requires fairly heavyweight language models to produce meaningful results (Bills et al.] 2023 Paulo et al. 2024). More recent methods aim to simplify this
process by focusing on distinguishing between activating and non-activating examples (Paulo al.) 2024).
One such method is detection, where the model only needs to predict whether any token anywhere in an example activates specified latent. In fuzzing, the model is asked whether a specific highlighted token in a context activates the latent or not. In Templeton et al. (2024). the authors propose a simple rubric score, where the model rates the relevance of given interpretation in relation to the context and corresponding activations. Notably, these scoring techniques are more suitable for human evaluation due to their simplicity.
2

---

Collect activations
Step Score interpretability activations
T
The dog was walkinng
Latent 10
2000 1: of steel were
Latent 200
2000 kg of steel
Intruder detection
10 tons of wheat
Step 2: Generate explanations
dog was walking in
2000 ig
Units measurement
10 tons wheat
LLM P
kg
2
60 mg
40 tons
Step 3: Score explanations
t
Score Example embedding
9/5
Units measurement
ug
g of sand
Latent
days
hours
40 min
days since
45%
Figure I: Evaluating the interpretability of SAE latents. To evaluate the interpretability of SAE latents, activations of the latents are collected over colection of text. Traditionally these are then used to generate explanations, which are then scored. This process an indirect measurement of the latents interpretability, and specific choices when generating explanations scores. Instead, we propose to evaluate the interpretability of SAE latents by directly looking at their activations, either through intruder detection example embedding scoring.
influence the
Other approaches focus on the downstream effects of activating certain latents. For instance, Gur- Arieh al. (2025) and Paulo et al. (2024) generate explanations by analyzing the impact that latent has on the base model's output when used as a steering vector. Their scoring metrics quantify how well the explanation does at predicting these steering effects.
3 METHODS
3.1 TRAINING SAES
In this work focus evaluating the interpretability of four different sparse autoencoders trained on the output of the MLPs at layers 9, 15, 21, and 27 of SmolLM2 135M (Allal et al. 2025). They were trained minimize the mean squared error between their output and the MLP output, with no auxiliary loss terms. We adopt the state-of-the-art TopK activation function proposed by Gao et al. (2044, which directly enforces desired sparsity level the latent activations without the need to tune an L1 sparsity penalty, and consider the case where l 32. All SAEs have 32k features. All sparse coders are trained using the Adam optimizer (Kingma & Ba 2017), sequence length of 2049, and batch size of 64 sequences. We train the SAES on 10 billion tokens sampled from reconstruction of the SmolL.M training corpus.
We also use our method on publicly available SAEs, namely SAEs trained on the residual stream of layer 20 of Gemma 9b (Team al. 20244 Lieberum et al. 2024) (gemmascope), on the residual stream of layers 17,19 of Llama 3.1 8b (Team) AleALEEEeehheAAAlletteerrrr and skip transcoders (SSTs) (Dunefsky et al. 2024) trained on the MLP layer of Pythia-1600 160m (Biderman et al, 2023 (EleutherAI). The Gemmascope SAEs have JumpRelu activations (Rajamanoharan et al. with an average LO of 57 and 32k features. The LLama SAEs have 131k latents and are Topk SAEs with k=32. The SST' trained on Pythia have 32k latents and are TopK SSTs with k=64.
2024

---

3.2 INTRUDER DETECTION
Inspired by intruder word detection (Chang et al. 2009; Klindt et al. 2025), we designed an intruder sentence detection task to evaluate the interpretability of SAE latents without relying natural language explanations. For each latent, we sample four activating examples and one -cttiviii intruder example. The intruder is drawn from pool of examples that do activate other latents but do not trigger the latent being evaluated. These examples are collect from the SAE training corpus.
To evaluate the interpretability of each latent, we present the five examples the LLM numbered list, and ask it to identify the intruder by index. Activating tokens in the activating examples are highlighted using << and >> to provide additional context, while in non-activating examples, random selection of tokens is highlighted. The number of highlighted tokens matches the average count in the activating examples, rounded down. Each example is constrained to 32 tokens in length to maintain consistency. Examples are preceded by a few-shot prompt demonstrating the task on synthetic examples.
The activating examples show are chosen at random from one of ten deciles of the activation dis- tribution. Decile sampling done for two reasons: first there has been previous work (Paulo al.] 2024) that showed that using decile sampling one could evaluate the interpretability of different parts of the distribution. We are also interested in understanding the similarity of the activations of the same feature, and dividing the distribution into bins makes it easier for us compare different parts of the activation distribution.
The score of the latent is then defined the accuracy of the evaluator at intruder detection for that latent, averaged over deciles. We also report the accuracy for individual deciles, allowing for the function of activation strength.
evaluation of interpretability
We depart from traditional intruder word detection by showing a full context, because we find that significant number of latents activate on more than one token in context, sometimes on several adjacent tokens.
3.2.1 HUMAN INTRUDER DETECTION
The authors manually performed the intruder task on a subset of examples to provide a human base- line. The procedure is similar to what is done by the LLM. The authors were shown five examples, with tokens highlighted as described above, and had to select which of the examples is the intruder. In LLMs, each intruder test is done from one set of examples to the next. The same is not true for the human participants. To minimize this effect, we randomly select which latent to sample the examples from, such that there aren't t many consecutive examples of the same latent. The decile from which the examples are drawn are also chosen randomly and not shown to the participants.
fresh context, and so there no information that carries
3.3 EXAMPLE EMBEDDING SCORING
Although intruder detection able to skip the explanation generation process, we find that a pow- erful I.L. is still needed to produce meaningful scores. Since high-quality embedding models can be quite small, we explore a method we call example embedding scoring to speed up the evalua- tion process. It is inspired by the two alternative forced choice task used in Borowski et al. (2020); Zimmermann et al. (2024).
Example embedding scoring evaluates the interpretability of a latent by measuring whether activat- ing examples cluster in embedding space. We sample sentence embeddings for a set of activating examples E {er- ........ e and a set of non-activating examples E {ej ,... .....,,,. We also sample embeddings for an activating example q called the positive query, and a activvatva example q- called the negative query. We compute how much closer each query is to examples
4

---

from its own class than to examples from the other class:
A+ N
Σ
q | e
(1)
q
e EE+
EB-
q e,
q
A =
(2)
6 EE-
EE
As in intruder detection, all positive examples are sampled from the same decile of the activation dis- tribution, and the negative examples are random non-activating contexts. All examples are sourced from the examples found in the training data of the SAE.
4+ and A_ can used to compute the AUROC, which is the scoring metric for this method. For each latent we iterate over different sets of explanations and queries, and average over them. This metric is similar to that of embedding scoring (Paulo et al. 2024), where explanations substituted by activating examples.
3.3.1 FINETUNING AN EMBEDDING MODEL
We found that off-the-shelf embedding models did not perform the example embedding task very well, and suspect that part of it is that they can't 't infer the meaning of the highlighting tokens we use to indicate which tokens are active stella 400M ememmdddi model, as it was lightweight but demonstrated good performance in embedding benchmarks. We selected latents from the TopK skip transcoders trained on the MLP of layer of Pythia- 160m, with k=32(]a as these would be not be used in this study and so would reduce the chances that the model would be overfit to contexts had seen. First, we filter latents that have score higher than 0.7 on both fuzzing and detection, until we have c.a 300 features. Secondly we create contexts similar to the ones used in example embedding, choosing 9 activating examples and an activating query and 9 non activating examples and non activating query, with 100 of these sets for each feature. We finetune using Multiple Negatives Ranked Loss, which takes pairs of (query, positive example) and increases their similarity with respect with all the other elements in the batch. This is not exactly the same task as example embedding but we find that it approximates it in reasonable manner.
sentence, see Table A1 We chose to finetune the
RESULTS
4.1 INTRUDER DETECTION
The average intruder detection accuracy for human is 64%, as measured on 105 different latents. On the highest activating deciles, human accuracy averaged 78%. In total, one third of latents had score over 80%, and only one in seven had a score lower than 30%, meaning that they few were considered non-interpretable. This supports the claim that explicit natural language interpretations are not needed to decide wether a given example should be active or not, and that showing other activating examples is sufficient.
To validate the usage of LLMs instead of humans in the evaluation process, we compute the scores given by the LLM on the same latents. While humans only see on average 10 to 20 prompts per latent, Claude Sonnet 3.5) the LLM we compare to in the left panel of Figure 2 is evaluated on 50 prompts. The Spearman correlation between the LLM scores and the human score 0.83, higher agreement than what was seen in previous SAE evaluation metrics (Paulo et al. 2024). although in those works the human evaluators were not performing exactly the same task, as the case here. The expected accuracy of random guessing is 20%, so each latent that has intruder detection accuracy of 20% or less is probably uninterpretable just by looking at activating contexts. We create four additional interpretability bins, ranging from low degree of interpretability (20-40%) to very high degree of interpretability (80- 100%). In the right panel of Figure 2 we report the agreement between binned human and binned LLM scores. We observe that the human mostly finds latents more interpretable than their LLM counterparts. This is good news, as it suggests LLMs are not finding convoluted patterns in the latent activations that humans cannot find.
5

---

1.0
Very High
3
12
Degree interpretabilityy (LLM)
of LLM detection
High
Medium
1
9
Low
8
8
3
0.2
No
0.67
0.0
0.6
0.8
1.0
No
High Very High Degree of Interpretability (Human)
Medium
Accuracy of Human detection
Figure 2: Correlation between human and LLM intruder detection accuracy. In the left panel we compare the accuracy on the intruder task for the LLM this case Claude Sonnet 3.5 and that of human. In the right panel we show more coarse grained classification. All latents which have less than that 0.2 accuracy are considered non interpretable, and different degrees of interpretability are assigned to the other 4 bins of 0.2. The agreement between the "quality" of the latents given by humans and the I.LM significant, even though the I.LMS underestimate the degree of inter- pretability. We evaluate 105 latents from SAEs trained on SmolLM 2 around 26 latents per layer on four different layers. The size of each dot represents how many prompts the human saw for each latent, and is meant represent the uncertainty associated with the score, with the smaller dots rep- resenting 8 prompts.
Accuracy on the intruder task is dependent on the the activation decile from which the examples were sampled. Examples from the highest activating decile can have up 20% higher accuracy than examples from the smallest activations (Figure left panel).
Previous work (Templeton et al.) 2024 Paulo et al. 2024) has found that examples with the lowest activations are hardest to interpret, and we observe the same thing. On the other hand, even though intruder detection accuracy lower in the lower activating deciles, still significantly higher than random. If divide latents into interpretability bins, remain interpretable even the lower activation deciles, with accuracies greater than 0.75 (Figure AD). which believe show that even low activating examples carry information about the behaviour the latent.
that the most interpretable latents
4.1.1 DETECTING INTRUDER DECILES
Intruder detection makes it easy to investigate how the meaning of latent varies as a function of the activation strength. Specifically, we modify our setup so that all five examples contain a token that activates the same latent, but one of them, the intruder, is sampled from a different activation decile as the rest. For a perfectly monosemantic latent representing a binary feature, we would expect the accuracy of this task to be close to random, as examples from different deciles would be very similar. If we assume that some latents represent scalar features with a notion of degree or intensity, would expect that nearby deciles would exhibit close to random accuracy, and that distant pairs of deciles would be easier to distinguish. We would also expect symmetry, where the difficulty of finding high-activating intruder among low-activating examples is similar to the accuracy of finding a low-activating intruder among high-activating examples.
Llama 3.1 70b (Team 2024) perform this detection task. As expected, Llama achieves highest accuracy when the majority decile highest and the intruder decile is lowest (51 (5.%). This result is far from symmetric, however: when the majority decile is low,
In the right panel of Figure 3

---

1.0
Lama
High
0.16
0:14
0.15
0.14
0:13
0.12
0.12
0.13
Human
0.30
0.10
0.17
0:16
0.15
0:15
0.13
0:13
0.15
0.45
0:33
0.25
0:14
0.14
0.40
Accuracy
0.6
Intruder Dapie
-0.44
0:41
0.33
0.28
0.24
0:17
0.17
0.16
0.17
0.17
0.18
0.21
0.35
-0.47
0.36
0:30
0.25
0.21
0.18
0.18
0.17
0.20
0.25
00.99
0.43
0:33
0.29
0:25
0.24
0:23
0:10 0.21
0.2
0.20
0.22
0.20
0.51
0.45
0:38
0.33
0.29
0.24
0:25
0.22
0.20
Highest
Lowest
Highest
Lowest
Activation
Correct Decile
Figure 3: Interpretability of the distribution of activations. (Left) The activation examples used on the intruder task come from different deciles of the distribution of the latent's activations. On the lower activating deciles, it's harder to distinguish which is the intruder example, but the accuracy is still significantly above random. This shows that the full distribution of the activations of the SAE latents remains somewhat interpretable. We evaluate 105 latents using human labeling, and 1800 latents using the LLM judge (Llama 3.1 70b). (right) If instead of using antiiviva examples we use examples from different decile, we can observe that the accuracy of close deciles is close to random, while it increases further deciles, remaining lower than when using non activating examples. We evaluate 3700 latents using the LLM judge (Llama 3.1 70b).
Llama essentially never achieves higher than random accuracy, no matter the decile of the intruder. Interestingly, when the intruder decile is high and the majority decile is low, significantly worse than random accuracy (around 13%).
While there likely some 'perfect' scalar features in these SAEs, do not clear evidence for them in the right panel of Figure 3 The accuracy matrix is nonsymmetric, and even in the lower triangular is not completely explained by the distance between majority and intruder deciles except by the highest activating deciles. We can rule out the hypothesis that most features are both binary and monosemantic, since we see far better than random accuracy for many (intruder, majority) pairs.
4.1.2 INTER -LEM AND INTER- HUMAN AGREEMENT
We investigate how the intruder accuracy for given latent is correlated across different LLMs. We find that different models' scores have around the same correlation than with humans, except for the smaller Llama 3.1 8b, which has the worst accuracy (27%), and the lowest correlation with human judgment.
We also interested in inter-human agreement. We find that, in smaller set of 40 latents, each with only 3-5 prompts, the correlation between the two human labelers is of 0.87, while the one between Claude-Sonnet and each of the humans is 0.86 and 0.69. This level of inter-LLM and inter- human agreement is promising for this proposed method of evaluating SAE latent interpretability. In Table we observe that scores given by humans, computed on 105 latents, have high correlation scores, >0.00, for several strong LLMs, but that it is lower for weaker LLM.
4.2 EXAMPLE EMBEDDING SCORING
Example embedding scoring also finds that high deciles are easier than from low deciles (Figure left panel). Unlike intruder detection accuracy, example embedding scoring is symmetric by struction (Figure right panel). Example embedding scoring also finds that highly activating deciles easy to tell apart from random examples, but has harder time to distinguish between examples

---

Human
Llama
Llama
QwQ
Gemini Flash 2.0
Claude
3.1 70b
3.1 8b
32b
Sonnet 3.5
Human
0.76
0.52
0.78
0.80
0.83
Llama 3.1 70b
0.76
0.58
0.89
0.85
0.88
Llama 3.1 8b
0.52
0.54
0.59
0.60
0.57
QwQ 32b
0.78
0.80
0.59
0.86
0.90
Gemini Flash 2.0
0.80
0.85
0.60
0.86
0.87
Claude Sonnet 3.5
0.84
0.88
0.57
0.90
0.87
Table I: Correlation between different evaluators. We measure the Pearson correlation between accuracy of intruder detection in 56 latents. Each latent was evaluated 10-20 prompts by the human, and on 100 prompts by the LLMs.
1.0
Highest
0.66
0.70
0.58
0.55
0.58
0.59
0.60
0.61
0.62
0.65
0.68
0.66
0.62
0.38
0.54
0.56
0.57
0.58
0.59
0.59
0.06
AUROC
Intruder
0.64
0.69
0.50
0.57
0.55
0.54
0.56
0.56
0.62
0.7
0.60
Bottom 20%
0.66
0.62
0.61
0.59
0.57
0.56
0.55
0.55
0.60
20-40%
40-60%
Lowest
0.56
Top 20%
.0.71
0.68
0:65
0.66
0.63
0.62
0.60
0.60
0.57
Highest
Highest
Lowest
Activating decle
Figure 4: Interpretability evaluation using example embedding scoring. In the left panel, we show that the highest activating decile easier to distinguish from non activating examples, was the case with intruder detection. On the panel on the right, is possible to see that, if using exam- ples from different deciles, close deciles hard distinguish, while further deciles the accuracy slightly increases above random, up to 0.7 AUC for some distances. We evaluate this method on 700 latents.
belonging the same feature but from different deciles, which the AUC for the largest possible difference reaching only 0.7.
Example embedding scores correlate as strongly with human intruder scores (r 0.78) LLM intruder scores do (r = 0.75). See Table for full results. Due to it being fast and computationally efficient technique relative other approaches, might be an appropriate technique for large scale evaluation of SAEs.
In Tables A2 and A3 we show that SAES trained in different models, with different activation func- tions, training objectives and sparsity levels can be evaluated using our methodology, by showing they have similar levels of correlations what shown Table 2
5 CONCLUSION
Prior work in the evaluation of SAEs relied on generating natural language explanations for SAE latents, which were then used to predict either the activations those latents, or their causal effects. In this work we introduced two new methods for evaluating SAEs which do not rely on explanations: intruder detection, and example embedding scoring. We tested intruder detection using both human and LLM evaluators, finding that the human accuracies highly correlated with those from LLMs. We use both intruder detection and example embedding scoring to measure the interpretability of different parts of the activation distribution, finding that higher deciles more interpretable. As
8

---

Example Embedding
Intruder Detection (Human) 0.78
Intruder Detection (LLM) 0.77
Fuzzing (LLM)
Detection (LLM)
Example Embedding
1.0
0.87
0.52
Intruder
0.78
1.0
0.75
0.77
0.48
Detection
(Human)
Intruder
0.77
0.75
1.0
0.78
0.56
Detection
(LLM)
Fuzzing (LLM)
0.87
0.77
0.78
1.0
0.64
Detection
0.52
0.48
0.56
0.64
1.0
(LLM)
Table 2: Correlation between different methods. We measure the Pearson correlation between the accuracy of intruder detection and other scoring methods 105 latents. Each latent was evaluated in 10-20 prompts by the human, and 100 prompts by the other methods.
expected, example embedding scoring has lower correlation with human intruder detection scores than LLM intruder detection scores do, but we believe it to be promising method due to its speed: uses small embedding model rather than large language model to perform the scoring.
ACKNOWLEDGMENTS
We would like to acknowledge our discussion with David Klindt that initiated this work. We are thankful to Open Philanthropy for funding this work. We are grateful to Core for providing the compute resources.
AUTHOR CONTRIBUTIONS
Gonçalo Paulo developed the scoring methods and performed the experiments. GP wrote the initial draft. Nora Belrose performed some intruder detection experiments, provided guidance and did major reviews to the draft.
CODE AVAILABILITY
Both the intruder detection scorer and the example embedding scorer can be used in our pipeline for automated interpretability of sparse coders, delphi Sparse autoencoders where trained using our training library, (sparsff
REFERENCES
Etowah Adams, Liam Bai, Minji Lee, Yiyang Yu, and Mohammed AlQuraishi. From mechanistic interpretability mechanistic biology: Training, evaluating, and interpreting sparse autoencoders on protein language models. bioRxiv, pp. 2025-02, 2025.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big-data-centric training of small language model. arXiv preprint ar arXiv:2502.022 02250.00277..... 2025.
Kola Ayonrinde, Michael T Pearce, and Lee Sharkey. Interpretability compression: Reconsidering sae explanations of neural activations with mdl-saes. arXiv preprint arXiv:2410.11179, 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle OrBrien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al.

---

Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in lan- guage models. URL http:://oeeaiipublicc blob. core. windows. net/neuron-exppee hwtp:pmn-cccaiieeerpppennnded html. accessed: 14.05. 2023), 2, 2023.
Judy Borowski, Roland S. Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain enn activations better than cate-of-the-ane feature visualization, 2020. URL https://arrx...or/////aa 12606
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. Reading tea leaves: How humans interpret topic models. Advances in neural information processing systems, 22, 2009.
David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. A for absorption: Studying feature splitting and absorption in sparse autoencoders, 2024. URL
14507
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08610 2023.
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits. Advances in Neural Information Processing Systems, 37:24375-2444 24410,, 2024.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093 2024.
Liv Gorton. The missing curve detectors of inceptionvl: Applying sparse autoencoders to incep- tionvl early vision, 2024. Uttptttps:/axxiivcoom 2bc///////2222222 03662
Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. Enhancing automated interpretability with output-centric feature descriptions, 2025. URL httts:://aaxi/t org/
abs/2501.0830 08319
Adam Karvonen, Can Rager, Rager,Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability, 2025. URUntt.s.//aaci.voor/////
09532
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https://arxiie org/ abs 1412.4124 6980
David Klindt, Charles O'Neill, Patrik Reizinger, Harald Maurer, and Nina Miolane. From perposition to sparse codes: interpretable representations in neural networks. arXiv preprint arXiv:2503.01824, 2025.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. ArXiv, abs/2007.04612, 2020. URL https:
22:0024444:: //api. www.sssmmatiicchola.i... wwwpolla...rcoooouussss.o..
Nhat Minh Le, Ciyue Shen, Neel Patel, Chintan Shah, Darpan Sanghavi, Blake Martin, Alfred Eng, Daniel Shenker. Harshith Padigela, Raymond Biju, Syed Ashar Javed, Jennifer Hipp. John Abel, Harsha Pokkalla, Sean Grullon, and Dinkar Juyal. Learning biologically relevant features in pathology foundation model using sparse autoencoders, 2024. URL https: /tt./iivvvco orr//
abss /ess/247..cm 10785
10

---

Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once gemma 2, 2024. URL
wwtpxii.orrrgaaa.. org/abs/
1408.05144
Jesse Mu and Jacob Andreas.
neurons.. ArXiv, abs/2aa6.100003333 2020.
URL http://ppw..oaaeee.cccco.orrrrrs wttp:/aaar..rroooooaoooo 2200555996
Gonçalo Paulo and Nora Belrose. Partially rewriting a transformer in natural language, 2025. URL
https://aaxiv http://pxww..rrrra///5/111..1111
Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpretin millions of features in large language models, 2024. /ttpppppss//arrvv..orrr/aa///// 2ttp://aaab/211..192...
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 24435.14443, 2024.
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg- KirgKKirkpatrick, and Eduard Hovy. Spine: Sparse interpretable neural embeddings. In Proceedings of the AAAI conference on artifi- cial intelligence, volume 32, 2018.
Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, and Caglar Gulcehre. Unpacking sdxl turbo: Interpreting text-to-imaae models with sparse autoencoders, 2024. htt.:ttss//xxiiv.ooott
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint 2rXiv:2488.001 18, 2024.
Llama Team. The llama 3 herd models, 2024. URL https://arxiv.org/aaasa
21783
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Tumner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- former Circuits Thread, 2024. URL httpp::/ttr/tt fornsffrrmrrrrr cetommerciieee cwcciirrtitss. pubb/20247 scal 1 iiiiin ing-monosemant.e..mmmmmmommmmmm. ci tyy wtttttyy..nnneeeeee index.id...... html
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of -f prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24883 2022.
Changwan Zhang and Yue Wang. A sample survey study of poly-semantic neurons in deep cnns. In Conference on Computer Graphics, Artificial Intelligence, and Data Processing, 2023. URL https: pettps://ap... semant icschollaa
Roland S. Zimmermann, Thomas Klein, and Wieland Brendel. Scale alone does not improve mech- anistic interpretability in vision models. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: : /ownn//eeeeerreeeeeeemooo net. wwt.teeettooeeee fortmmormm? i d==02 7 ImD uu
Roland S. Zimmermann, David A. Klindt, and Wieland Brendel. Measuring mechanistic inter- pretability at scale without humans. In ICLR 2024 Workshop on Representational Aligument, 2024. http:ttpss///peeneeeeeeee.eee//e
A APPENDIX
11

---

Bottom 20% accuracy
0.9
20-40%
40-60%
0.8
60-80%
Top 20% accuracy
0.7
0.6
0.5
0.4
0.3
0.2
Highest activating
Lowest activating
Sample quantile
Figure Al: Interpretability of the distribution of activations, stratified. we observe each the 5 bins of accuracy separately, the bin with higher aceuracy consistently has high accuracy throughout all the distribution, while the other bins have sharper drops accuracy when going from the highest activating examples to the lowest ones.
Example Embedding
Example Embedding (no finetuning) 0.47
Intruder Detection (Human)
Intruder Detection (LLM)
Fuzzing (LLM)
Detection (LLM)
Example
1.0
0.78
0.77
0.87
0.52
Embedding
Example
0.47
1.0
0.42
0.46
0.50
0.60
Embedding
(no finetuning)
Intruder
0.78
0.42
1.0
0.75
0.77
0.48
Detection
(Human)
Intruder
0.77
0.46
0.75
1.0
0.78
0.56
Detection
(LLM)
Fuzzing (LLM)
0.87
0.50
0.77
0.78
1.0
0.64
Detection 0.52
0.60
0.48
0.56
0.64
1.0
(LLM)
Table A1: Correlation between different methods. We measure the Pearson correlation between the accuracy of intruder detection and other scoring methods in 1 112 latents. Rach latent was evalu- ated in 10-20 prompts by the human, and on 100 prompts by the other methods.
12

---

Example Embedding
Intruder Detection (LLM) 0.84
Fuzzing (LLM)
Detection (LLM)
Example
1.00
0.90
0.58
Embedding
Intruder
0.84
1.00
0.83
0.67
Detection
(LLM)
Fuzzing (LLM)
0.90
0.83
1.00
0.71
Detection
0.58
0.67
0.71
1.00
(LLM)
Table A2: Correlation between different methods in Gemmascope SAEs trained onGemma 9b. We measure the Pearson correlation between the accuracy of intruder detection and other scoring methods 50 latents. Each latent was evaluated on by 100 prompts for each method.
Example Embedding
Intruder Detection (LLM)
Fuzzing (LLM)
Detection (LLM)
Example
1.00
0.67
0.84
0.40
Embedding
Intruder
0.67
1.00
0.70
0.55
Detection
(LLM)
Fuzzing (LLM)
0.84
0.70
1.00
0.59
Detection
0.40
0.55
0.59
1.00
(LLM)
Table A3: Correlation between different methods in SSTs trained on Pythia-160m. We measure the Pearson correlation between the accuracy of intruder detection and other scoring methods in 230 latents. Each latent was evaluated on by 100 prompts for each method.
13


Reviews:
[{"review_id": "UhdnwsXEao", "reviewer": "Reviewer_VHFv", "summary": "The authors introduce two novel methods for evaluating interpretability of SAE latents without the requirement for generative methods. The authors construct an intruder detection task as the first approach and compare performance of human and LLM detectors, showing a high correlation albeit at a small sample size of 56 latents. Example embedding scoring, on the other hand, measures proximity of positive and negative sentence samples in the latent space. Example embedding scoring reports a moderate correlation with human scores, which could be caused by the fact that sentence embedders might poorly reflect individual token relevances.\n\nThe problem of evaluating SAE interpretability is an important one, and the proposed methods have merit. The high correlation between human and LLMs on the intruder detection task is promising. The presentation of the paper, however could be improved upon. I find more extensive experiments lacking, such as adding more latents in the intruder detection task or performing subsequent analses on what causes the low correlation between human and example embedding scores — is the embedder quality a factor driving this gap? Furthermore, it is not clear to me where the data used for positive and negative SAE samples is sourced from, which is a crucial detail. An interesting question would also be also how do the methods fare across different data domains of source text?", "questions": "See above", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The authors study an interesting problem of interpreting SAE latents without the use of generative LMs\n- The authors propose two methods, one of which exhibits a high correlation with human annotators", "weaknesses": "- The presentation of the paper would benefit from improvement\n- Some important experimental details are missing: where is the data used as positive/negative samples for SAE latents sourced from? \n- Experimental limitations: increasing the number of latents, or analysing the cause of poor correlation between the example embedding method and human scoring would be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iAzccAF55A", "reviewer": "Reviewer_Bo6j", "summary": "This paper proposes two explanation-free methods for evaluating the interpretability of sparse autoencoders (SAEs): intruder detection and example embedding scoring. The authors test the proposed methods on SmolLM2 135M across 56 latents and find a strong correlation between human and LLM evaluators in intruder detection. The intruder detection method successfully bypasses natural language explanation generation while maintaining interpretability assessment; however, the embedding method shows limited correlation with human judgments. Higher activation deciles prove more interpretable across both methods, and the evaluation reveals that most SAE latents demonstrate interpretability without requiring explicit verbal descriptions.", "questions": "1. Why does the example embedding score not correlate as strongly with human intruder scores as it does with LLM intruder scores? Authors say example embedding scores tend to underestimate the interpretability of latents due to the small size of the embedding. Does the correlation improve when the embedding size is increased?\n\n\n2. How sensitive are the intruder detection results to the highlighting strategy? Have you tested alternative approaches, such as not highlighting any tokens or using attention-based highlighting to focus on the most relevant tokens?\n\n3. What is the rationale for randomly selecting a single decile and sampling all activating examples from it?\n\n4.  Proposed SAEs use TopK activation with $k=32$. How do results change with different $k$ values, different activation functions, or different sparsity levels?\n\n5. Can you provide examples of latents that score poorly on intruder detection but might still be considered interpretable by other measures?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Figure 1 effectively illustrates the conceptual shift from explanation-based to activation-based evaluation, and the writing is generally accessible.\n\n2. The paper introduces evaluation methods that bypass natural language explanation generation, addressing a significant limitation in existing sparse autoencoders' interpretability assessment. This is a significant contribution that streamlines the evaluation pipeline and minimizes the impact of confounding factors.\n\n3. Example embedding scoring offers a computationally lightweight alternative using small embedding models, making large-scale SAE evaluation more feasible.", "weaknesses": "1. The evaluation focuses exclusively on SmolLM2 135M across only 4 layers with 56 total latents. This narrow scope raises questions about generalizability to larger models, different architectures, or other SAE training approaches beyond TopK.\n\n2. Example embedding scores do not correlate as strongly with human intruder scores (r = 0.48), and AUROC are close to random, which limits the practical utility of the proposed scoring method. \n\n3. The paper lacks a discussion of failure modes or which types of latents are poorly captured by the proposed methods. \n\n4. Limited investigation of why LLMs consistently underestimate interpretability compared to humans", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAEsBaGlad", "reviewer": "Reviewer_6pUU", "summary": "This paper proposes a novel evaluation approach to assess the interpretability of sparse autoencoders (SAEs). Instead of generating natural language explanations as an intermediate step, the authors introduce two explanation-free methods: intruder detection and example embedding scoring. The paper demonstrates that direct assessment of latent interpretability is viable and correlates well with human judgments when using an LLM-as-a-judge approach.", "questions": "- Line 34, the coefficients is not non-negative necessarily, if this refers to activation value of latents. Please verify with examples from Neuronpedia.\n- Line 44 - 46, the conclusion on natural language explanations introduced additional hyper parameters and prompts can be expanded further. It’s not very clear how they introduce additional parameters, which might refer to simulations. But it’s important to explain this clearly at the beginning of the paper. I feel the authors should spend more time polishing the introduction section to stand out their motivation and make it accessible. The last paragraph of the introduction is hard to follow. The introduction to their own methods is not clear at all.\n- Heatmap in Figure 2 is not very illustrative. What does \"All latents which have less than that 0.2 accuracy are considered non interpretable, and different degrees of interpretability are assigned to the other 4 bins of 0.2.\" mean? Would overlapping histogram be more illustrative here?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper demonstrates the feasibility of the proposed method intruder detection, achieving strong correlation between human and LLM assessments.\n- The methods used in this paper are straightforward and easy to understand.\n- The paper examines interpretability across different activation deciles, providing nuanced insights into how interpretability varies with activation strength.", "weaknesses": "- However, the performance of embedding score is not promising. The AUROC scores are barely above random (0.5-0.7), and correlation with human judgments is weak (r=0.48). This undermines one of the paper's main contributions, as this method was proposed as a fast, scalable alternative.\n- Lack of direct performance comparison with traditional interpretability evaluation methods.\n- Results are presented on very small set of latents (56) and small models. So we don't know if this holds when dataset scales up.\n- The bottleneck of evaluation seems to be extensive data collecting process, why avoiding natural language explanation is a critical problem?\n- Despite claiming to simplify evaluation, intruder detection still relies heavily on LLM queries, contradicting the motivation of reducing computational costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Q918WG5rU", "reviewer": "Reviewer_KYqC", "summary": "The paper introduces a new method for evaluating Sparse Autoencoders (SAEs). It argues that explaining latent directions in the SAE’s latent space through short textual descriptions is suboptimal for two main reasons. First, this approach complicates the evaluation process by adding hyperparameters and prompt-related variability. Second, a latent factor can be interpretable even if it cannot be concisely expressed in words.\n\nAs an alternative, the paper proposes an intruder detection framework. For each latent, four activating examples and one non-activating “intruder” example are sampled. Interpretability is then assessed based on how effectively humans, large language models (LLMs), and an embedding-based algorithm can identify the intruder. This approach emphasizes intuitive recognition of the pattern a latent does or does not encode.\n\nThe results show strong agreement between human and LLM performance in intruder detection, indicating that LLMs may be well-suited for automating SAE interpretability evaluation.", "questions": "Looking at the interpretability of distributions of activations, the LLM's results are very far from symmetric: \nit is much better at detecting a low-activating intruder among highly-activating samples than vice versa.\nThis is something I could not have predicted, do you have any intuition for why this is? And, do you have any data on how symmetrical humans are?\n\nHave you compared your interpretability scores to the explanation-centered approach you contrast to in the introduction? Can you find examples of latents which would be deemed uninterpretable according to other methods, but are considered interpretable under your framework?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Proposes new method for evaluating SAEs that is more permissive in the types of interpretability it allows for (interpretable, but not easily expressible in words). \n\nThe proposed method looks very promising, with LLM accuracies tracking those of humans.\n\nMultiple approaches toward the task are evaluated (LLM vs. embedding).", "weaknesses": "I think the presentation could be significantly improved.\nOn line 155, it is explained that 'We randomly select one of the ten deciles of the activation distribution, then sample all of our activating examples from the same decile.', but this is then not at all motivated. I found it quite difficult to understand why we would want to do this, and it wasn't until re-reading some of the results section for the second time that I understand the point. Specifically, the paragraph on lines 295-304 goes into the different ways we might (not) assign meaning to the activation strengths of the latent. I think that goes a long way towards motivating why we care about deciles, but it appears in the results section, rather than in an earlier section, where I would expect it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)