Kati8/ICLR_2023_2025 huggingface dataset:
authors: Javier Ferrando
authors: Oscar Balcells Obeso
authors: Senthooran Rajamanoharan
authors: Neel Nanda
reviews: 
Comments: This paper investigates the mechanisms behind LLM hallucinations by using sparse autoencoders -- SAEs to analyze internal model representations. The key finding is the discovery of "entity recognition" directions in the representation space that detect whether a model can recall facts about specific entities. These directions generalize across different entity types (movies, cities, players, songs) and causally influence whether the model refuses to answer questions or hallucinates information. More interestingly, the directions found with a base model can transfer to a chat model and control the model's tendency in refusing to answer questions.
Strengths: - Empirically link the "entity recognition" directions and the models' refusal in answering questions
- The experiment includes clear experimental design using diverse entity types
- Shows interesting discovery that base model mechanisms are repurposed during chat model fine-tuning
Weaknesses: - **Lack of comparison to methods other than SAEs**: If we remove the sparsity of the SAE and reduce the module to a regular AutoEncoder, would the same properties hold? If we further remove the auto encoder and just use a linear probing module, would the same properties hold? Is it really necessary to use sparse and auto encoder modules to get the interpretability? If we borrow the same idea from ITI [1] to steer model towards a more truthful direction in the latent space, can the same property holds? Lack of ablation on the necessity of SAE is a big flaw of this paper.
- **Lack of model diversity**: Only the Gemma models are tested. It would be better if we can see the same findings on LLaMA or other models to show how universal this idea holds.

[1] https://arxiv.org/abs/2306.03341
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The paper revealed that the knowledge of a large language model can be detected from a sparse latent space. There are several important findings:

- The last token of known and unknown entities have similar latent representation respectively in a sparse latent space across different entity types.

- The latent representation of entities has an influence on LLM generation and steering the representations will affect the refusal and fact recover behaviors of LLMs.

- The latent representation can predict the correctness of generated answers.
Strengths: The findings of the paper (see summary) are well supported by the experiments and can provide valuable scientific insights to the community. The presentation is great and readers can easily follow.
Weaknesses: The experiments are now only for Gemma models. It would be interesting to see the model behavior on other LLMs.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 4
reviews: 
Comments: Using sparse autoencoders (SAEs), this paper shows the existence of directions in the latent space of LLMs responsible for entity recognition i.e. detecting whether the entity is a known entity or not. The paper shows that these identified latents are causally responsible for LLMs refusing to answer questions. The paper shows how these latent directions are responsible for regulating attention scores, and also responsible for directly expressing knowledge uncertainty. All experiments are conducted with Gemma 2 models (2B and 9B) where SAEs are available.
Strengths: 1. The result showing that directions identified using base model, causally affect refusal in chat-finetuned models is very interesting — the transfer from base to instruction/chat finetuned model is not obvious, and indicates a mechanism for how refusal finetuning can generalize beyond the specific entities it was finetuned on. 

2. All of the experiment and results figures seem thorough and convincing (e.g. error bars, useful and clear visualizations etc.)

3. The paper also shows that interestingly, the same entity recognition directions regulate the model's attention mechanism (the factual recall mechanism) and directly express knowledge uncertainty (line 355, although the effect is not as strong).
Weaknesses: 1. Definition of known and unknown entities — the paper categorizes entities as known if it gets at least two attributes right, and unknown if it gets all attributes wrong. While this is fine, it does leave open the possibility that what is currently categorized as unknown is actually known by the model and can perhaps be elicited with a better prompt or few-shot examples. Alternatives to the current categorization could be – (a) checking across a diverse set of prompts to decide known and unknown as in https://arxiv.org/abs/2405.05904 ; (b) relying on n-gram matching in the pretraining data (would need to use LLMs with open pretraining data) ; (c) focusing on entities which occur after the knowledge cutoff for the unknown category. 

2. Explanation for existence of generalized latents that distinguish between known and unknown entities — the paper currently suggests that this is evidence of ‘self-knowledge’. An alternative and much simpler explanation which currently can’t be ruled out is that these latents are activated on high likelihood tokens (known) and low likelihood tokens (unknown). It would be useful to show some kind of control experiment to show that is not the case. 

3. (minor) In general, I also find it hard to directly interpret the results in Fig 2. The min-max value peaks at around 0.4 for unknown. How high should that value be to conclude the existence of generalized latents? I feel like if the set of the entities is small enough and the set of latents large enough, then there might always exist such latents? (Fwiw the additional results later do make it convincing that these are generalized latents)
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This work demonstrates that knowledge encoded within large language models can be identified and extracted from a compressed latent representation space. The paper presents several notable discoveries: (1) Terminal tokens corresponding to familiar and unfamiliar entities exhibit analogous latent encodings within the sparse representation across diverse entity categories; (2) Entity latent encodings play a significant role in shaping LLM outputs, and modifying these representations impacts how models handle knowledge refusal and factual recovery; (3) The latent encoding space enables prediction of answer accuracy in generated responses.
Strengths: The conclusions presented in this work are rigorously substantiated through comprehensive experimental validation and offer meaningful contributions to the scientific understanding of the field. The writing quality is excellent, enabling readers to comprehend the material without difficulty.
Weaknesses: At present, the empirical evaluation is restricted to the Gemma model family. Extending the analysis to evaluate whether these phenomena generalize across alternative large language models would strengthen the claims and broaden the applicability of the findings.
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 4