Interpretability of LLM Deception: Universal Motif

Authors: Wannan Yang, Gyorgy Buzsaki
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: safety, honesty, deception, lie, interpretability, Large Language Model

Abstract:
Conversational large language models (LLMs) are trained to be helpful, honest and harmless (HHH) and yet they remain susceptible to hallucinations, misinformation and are capable of deception. A promising avenue for safeguarding against these behaviors is to gain a deeper understanding of their inner workings. Here we ask:  what could interpretability tell us about deception and can it help to control it?  First, we introduce a simple and yet general protocol to induce 20 large conversational models from different model families (Llama, Gemma, Yi and Qwen) of various sizes (from 1.5B to 70B) to knowingly lie. Second, we characterize three iterative refinement stages of deception from the latent space representation. Third, we demonstrate that these stages are \textit{universal} across models from different families and sizes. We find that the third stage progression reliably predicts whether a certain model is capable of deception. Furthermore, our patching results reveal that a surprisingly sparse set of layers and attention heads are causally responsible for lying. Importantly, consistent across all models tested, this sparse set of layers and attention heads are part of the third iterative refinement process. When contrastive activation steering is applied to control model output, only steering these layers from the third stage could effectively reduce lying. Overall, these findings identify a universal motif across deceptive models and provide actionable insights for developing general and robust safeguards against deceptive AI. The code, dataset, visualizations, and an interactive demo notebook are available at \url{https://github.com/safellm-2024/llm_deception}.

Decision: reject
Reviewer scores: 10, 5, 1, 6

Reviews and discussion:
<thread>
creation date:
1730513348058

thread_type:
official_review

<thread_object>
creator: reviewer_8jsy

<content>
summary:
This paper investigates several large language models (LLMs) to understand when and why these models may exhibit deceptive behavior. The authors first find that the tendency to lie increases with model size. They then explore how latent representations associated with lying evolve through three iterative refinement stages, concluding that smaller models lack the ability to lie as they cannot rotate truth directions during the third stage. The study further examines if this third stage is causally linked to lying, with findings that suggest a causal relationship.

strengths:
1. The study examines a diverse range of language models across various sizes and model families.
2. The research findings are intriguing, particularly the observation of truth direction rotation in the third refinement stage for models capable of lying.
3. The verification of findings, especially the conclusion that the third stage is causally linked to lying, appears robust and well-supported.

weaknesses:
1. The protocol used to instruct models to lie intentionally is overly simplistic. As a result, the conclusion that lying scales with model size may be somewhat limited. It would be valuable to explore whether smaller models could also exhibit deceptive behaviors if prompted with more sophisticated, carefully engineered instructions.

2. While it is interesting to identifying that the third refinement stage in the rotation of truth direction is causally linked to a model's ability to deceive, it would add depth to the study to investigate why smaller models fail to achieve this directional rotation.

3. The paper's readability, particularly in Section 4.4 and Figure 5, could be significantly improved. It is challenging to understand how the results presented in Figure 5, along with the text in the second paragraph of Section 4.4, effectively demonstrate the causal relationship between the third refinement stage and lying behavior.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730592748136

thread_type:
official_review

<thread_object>
creator: reviewer_fk9k

<content>
summary:
First conduct a simple prompt engineering to induce LLMs from four families (Qwen, Yi, Llama and Gemma) with different sizes to knowingly lie.
Then use the simple key word matching ("true"/"false") proportion as the evaluate metric for LLM deception.
Using the well-kown interpretability tool, i.e., activation steering, to study the key behind deception behavior and try to reduce deception.
Name 3 stages and some directions, give the finding that if a LLM complete the third stage by rotating the truth direction, it can perform deception knowingly.

strengths:
Study an important problem of safety.
Do some good visualization.

weaknesses:
1. The paper writing is not clear. Introduction part is too short to introduce the motivation and proposed method. Arrange of this paper is also bad. Figure 1 is far away from its corresponding explanations. There are also some typos resulting in difficulty in reading.
2. The authors do simple prompt engineering to induce lies from various models. They reach the susceptable conclusion: small models ''cannot'' lie. They cannot conclude like this unless exhausitive induction. I highly suspect this conclusion is incorrect if you try another prompt engineering approach.
3. LLMs' responses can vary a lot with multiple runs even with the same prompt. Results in Figure 2 with the simple metric ''accuracy'' are not reliable in this circumstance. And the authors say nothing about specific temperatures for these LLMs, which further makes the results untrustworthy.
4. With little innovation, this paper just uses a famous interpretability tool to study the deception scenario, presenting some findings seems not robust or reliable.

questions:
1. Can you explain why the four subfigures in Figure 4 part A are totally identical?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683419255

thread_type:
official_review

<thread_object>
creator: reviewer_maek

<content>
summary:
The paper studies the internal representation produced by LLMs prompted to lie or respond truthfully to binary questions. In particular, they analyse the activations at all layers of the model and plot the first 2 principal components (PCs) of the activations. The resulting scatterplot can be grouped in three stages: first, the PCs of the activations corresponding to truthful and untruthful scenarios separate; then, within each of the clusters obtained in the first stage, the activations corresponding to true and false answers separate, with the vector going from the centroid of the true to the false cluster ("truth direction") being roughly parallel across truthful/untruthful scenarios; finally, the activations for the true and false answers in the untruthful scenarios get swapped, thus leading to antiparallel truth directions. These three stages occur in all models capable of lying in the ones they tested. Next, they perform some experiments to probe the causal nature of the rotation: first, they perform "patching", where the activations corresponding to the lying scenarios are patched onto the truthful scenario to see if the model ends up lying or not; they find that a small set of token positions and heads lead to changing model behaviour. Finally, they perform model steering, namely, adding a constant steering vector obtained as the average difference of the activations in the truthful and lying scenarios; they find that only steering the layers from the third stage reduces lying. Therefore, they conclude that these layers are causally responsible for the lying behaviour.

strengths:
## originality
- the findings on the three stages of activations across layers are original.

## quality
- The experiments are thoroughly conducted and analyzed.

## clarity
- The various figures present the main findings very clearly and succinctly.

## significance
- Understanding how lying and deception emerge in LLMs is an important question; moreover, the paper also suggests that activation steering could neutralise (or at least reduce the prevalence) of deception.

weaknesses:
- The related works are missing reference to https://arxiv.org/abs/2407.12831, which shares several similarities with the paper (in particular, they also identify a 2-dimensional space which represents lying/truthful state and true/false answer.

- Some parts of the paper are written in a too informal manner for a published work, and seem rather to have been obtained from internal research notes. In particular, I refer to Sections 3.4, 3.5, 3.6, 3.7, 4.2, 4.4.

- Other minor presentation details: Section 3 could introduce what the various subsections should discuss; Fig6B has a "Loading Mathjax" box which should be removed.

- The paper claims that it "introduces a simple yet general protocol to induce large conversational models to knowingly lie"; however, from the description in 3.2, this protocol seems extremely simplistic (two prompt templates fixed across LLMs), not advancing in any way with respect to previous settings, such as Pacchiardi et al. 2023. As such, I don't think this should be presented as a main contribution, nor as "careful prompting design", as it is described in Sec 3.3. Or, does the description of Sec 3.2 overlook important details (see question below)?

- The paper claims that the patterns they find are "universal". While interesting that they are coherent across the considered LLMs, the setup they consider is still quite narrow (a single prompting setting, and only ~100 binary questions linked to scientific facts). As such, I believe the use of the term "universal" is unsuitable, particularly in Secs 2 and 5. 

- Relatedly, the fact that a fairly narrow set of examples is considered makes me wonder how general the patterns of activations (shown in Fig 3) are; in particular, it may as well be that the activations are different for a different set of samples. Or, it may be that the overall patterns are preserved, but that the different clusters are less distinct due to larger variety in the prompts.


- More experiments could be done to complement the ones provided and provide additional evidence of the causal nature of the 3-stage patterns identified. For instance: 
	- To complement the study in Sec 4.5, the authors could try steering honest models to lie.
	- Also, steering using the truth direction rather than the honest direction could be done

questions:
Sec 3.2: 
- is the "protocol" only composed of those two fixed prompt templates, or are other templates used (for instance, for different models)?

Sec 3.3:
- Is the "match" with the ground-truth label done as a simple substring check? If that is the case, "The answer is not *true" would be incorrectly marked.

Sec 3.4: 
- Truth direction: what do "true" and "false" refer to? Is it the answer that the LLM produces, or the ground truth?
- That is an arithmetic mean, not a geometric one.

Sec 3.5.1: 
- It is not clear from the formulas there that the prompts are contrastive; in particular, the definitions in Eq 4 do not make clear that the same number of prompts are used.

Fig 1:
- The lie example in Fig 1 is not very "convincing", but it is rather as if the model was joking. Are all generated lies across models of this form? If that is the case, then maybe the authors could try methods that generate more convincing lies. If that is not the case, then it would be interesting to investigate if there is any distinction in the activations between convincing and non-convincing lies.

Sec 4.3:
- have the authors got any interpretation for why the "truth direction" influences the ability to lie?

Sec 4.4: 
- is the patching experiment done on all models? Or only on those that are capable or incapable of lying?
- if the activations of a complete layer are replaced, does that mean that all downstream activations are equivalent to those that the model would produce if the truthful example was presented to the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706729295

thread_type:
official_review

<thread_object>
creator: reviewer_nvcr

<content>
summary:
This paper investigates Deception in LLMs and uses activation engineering to steer the model towards truthfulness in different layers. The experiments are specifically focussed on truthfulness and lying behavior with a contrastive steering vector computed. Each factual question in the dataset is prompted to give a true or false answer. The authors find that only larger models can knowingly lie, whereas smaller ones cannot. Further, the authors find that the latent representations of lying go through three distinct phases including separation of honest and lying (where two distinct clusters form), separation of true and false (where with the previous two clusters, two subclusters respectively form) and rotation of truth direction (where the lying cluster reverses direction by inverting the positions of the two subclusters, whereas the truthful cluster further separates out the original subclusters in the original direction). Beyond this, the authors find that intervention with a steering vector to reduce lying is only effective in the third stage layers. It appears that in models that cannot lie the three stages also exist but the rotation does not happen, whereas in models that can lie, the rotation is consistent. Interestingly, this motif is consistent across models of size 1.5B to 70B. This research sets an interesting precent for further research into deception, with the potential to extend this work into further types of deception and to more deeply understand the roots of this phenomenon. Perhaps it suggests opportunities for the development of interventions on the activation level during deployment.

strengths:
Originality: The model finds a novel rotation effect in latent space of models when prompted on a honesty and lying dataset. Though related work has engaged with Deception in various ways, this type of interpretability using activation steering has seen little precedent in deception research. 

Quality: While the paper has a narrow focus, the claims made are generally well substantiated. Perhaps the term 'universal' is a bit confident given this paper tested the effect across four models of different sizes, and it's possible that further models might not display the same effect. 

Clarity: The figures are exceptionally clear at illustrating the rotation effect and the distinctions between the phases. The math explaining the computation of the activation steering vector is easy to follow. The section explaining how the patching was done could have been a bit more clear on how the patching operation was done. 

Significance: This work presents an interesting starting point for lots of future work investigating the nature of deception in LLMs. If this is indeed a universal motif across all LLMs, this work has found a highly unique pattern which may allow for safety intervention based on activations for models in inference. Due to activation steering's computational lightness, this might be a very tractable intervention for models in deployment. Additionally, follow on work might shed more clarity on which stages and layers are most strongly involved in deception, and how to mitigate risks from Deception more effectively.

weaknesses:
The authors also acknowledge that only a narrow subset of deception, namely binary honesty and lying, is considered. It remains open whether these results would generalize to other forms of deception or dishonesty, such as when the statements are not purely factual or when there might be partial truths or a continuum of honest to dishonesty rather than a binary truth or false response. Would this, for instance, correspond to a partial rotation? Perhaps the authors intentionally focussed on purely factual true or false questions for ease of evaluation. 

It might be further interesting to see whether this effect replicated in more complex deceptive setups. For instance, if an agent needs to be lie to achieve a goal in a multi-turn problem setting, does the same effect appear? It might be interesting to try to analyse whether rotation happens, and if so, at which time point the rotation takes places, and how soon after this the outputs reflect the deception. 

It would be interesting to see more analysis of the differences in the rotation effect between different models. Does it rotate at approximately the same location (proportionately) in layer space in each model? Are the sizes of the stages the same size for each model respectively? (ie. for Llama - 3- 8B there seem to be 4 layers in each stage. Does one of the larger models show this same consistency in size of layers?). Are the layers involved always consecutive? What proportion of a model's layers is involved vs not involved in this effect? 

Typo in section 4.2 in "Lllam-3-8b-chat".

questions:
Can you share more details on the methodology used for patching? 

Does this effect extend to further models of sizes between those tested, or larger than those tested? 

How might these results extend to non-binary lying (ie. where partial truths are told, and the response isn't clearly labelled with true or false)?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies deception in LLMs using interpretability methods to analyze and control deceptive behaviors across models of various sizes and families. The topic is crucial for AI safety and alignment, and understanding the mechanisms behind deceptive behavior in LLMs is an important research challenge. The paper presents original and significant findings, such as the discovery of a consistent latent rotation effect across various model sizes using activation steering, which the reviewer praised as a valuable contribution with potential for impactful future research in AI safety.

However, concerns have been raised regarding the methodology, presentation quality, and the strength of the conclusions. For instance, there remain controversies surrounding the design of prompts and the selection of reviewers. Although the authors have made improvements in the second version, the reliability of prompt engineering in this paper is still questionable. Additionally, issues persist in the reviewers' perspectives on binary classification. As one reviewer noted, certain variants of "true" have been overlooked, leaving gaps in the analysis. Moreover, the conclusion that small models "cannot" lie is not sufficiently substantiated in its current form.  

While the findings are intriguing and the topic is significant, revisions are necessary to address these methodological limitations, ensure the conclusions are better supported by evidence, and improve overall clarity. The current version falls short of the standards for acceptance but could be significantly strengthened with further revisions for future submissions.