reviews: 
Comments: This paper discusses some interesting features found in sparse features extracted from Gemma 2 2B and 9B models with the Gemma Scope SAEs. These features, when queried on the final token of an entity mention, are predictive of whether the model correctly recalls facts about that entity in a synthetic task constructed from Wikidata. Some features generalize across multiple (four) entity types, and interventional experiments on chat-tuned models find that they mediate refusals in these models, suggesting that RL training may preferentially lead models to repurpose these mechanisms developed in pretraining. Further analysis shows that these features, when activated, disrupt the normal mechanisms for factual recall by blocking attribute extraction heads. A final experiments identifies SAE latents in a similar spirit which are predictive of model mistakes.
Strengths: * The paper is excellently written. The prose is very clear and the arguments are well structured.
* The paper fruitfully pursues multiple lines of evidence for its core claims about the latents it identifies in the model it studies. Having correlational and causal analysis on multiple fronts including with the RLHF'd model makes the central idea feel pretty solid.
* The paper is well scoped.
Weaknesses: Some of the claims of the paper feel not quite adequately supported by the experiments. I'm not really doubtful of the reasonableness of the claims but I think the paper should quantify and qualify them better.
* I think the choice to describe these latents as representing "self-knowledge" is a little bit dicey. The description only clearly applies to the chat model. For the base model I think a sufficient explanation of the evidence with fewer assumptions is to say something like they're latents for "entities it knows about" versus "entities it doesn't know about or treats as novel". You could make progress disentangling this from 'self-knowledge' by looking at how these latents behave on fictional narrative versus newswire versus encyclopedic text (the former two should have been written after training cutoff). I would expect the 'known entity' latents to fire increasingly more as you move from fiction->news->encyclopedia and opposite for 'unknown entity' latents. Analyzing these results could help us understand why the features are learned, e.g., if the the model needs to know whether it is writing stories or stating facts.
* The paper claims that the features generalize over multiple entity types. This is a binary statement but of course is about a graded judgment regarding the amount of generalization. But there is no quantitative claim in the paper about the degree to which this is true in an absolute sense—only that it is _more_ true in the middle layers. That doesn't mean that it's particularly true anywhere. The maxmin metric makes sense for relative comparisons but is difficult to understand on an absolute level. Indeed, when you select the "most general" latents based on this metric it's unclear if these features are actually general across more than the four entity types you use or if you may just be selecting on noise. To measure this, you should measure what happens when you select latents based on their generality across a subset of entity types, and test their separation score on held-out entity types. Some measure of how predictive this selection is of separation for new entity types gives an actual measure of the generality of the latent.
* In the mechanistic analysis (p.6–7) the paper points to figures which seem to show that attention to an entity is decreased when the unknown entity latents are steered (increased). It's not clear to me from the text if this is averaging across all attention scores or some subset ("these attention heads", L345, reads to me like the latter, but I'm not sure which subset) and the results of this experiment aren't quantified. There should be some straightforward measure here which you can use to quantify the suppression effect here (which will also allow you to claim statistical significance — you do say "significantly" on L346 without a relevant quantity or p-value, which leaves it unclear if you actually mean statistical signifiance)
* The results in Figure 5 seem dubious. They are really hard to read because of the overlap of all of the confidence intervals but overall nothing significant seems to be happening here. I think you'd need to run this study on much more data to get something interesting/clearly meaningful and I also find this the least interesting or informative experiment in the paper. Mild recommendation to appendicize it. (and strong rec to make the statistical claims precise either way: is the effect you're talking about significant or not?)
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4