Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models

Authors: Sarah Ball, Frauke Kreuter, Nina Panickssery
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: jailbreaks; activation engineering; alignment

Abstract:
Conversational Large Language Models are trained to refuse to answer harmful questions. However, emergent jailbreaking techniques can still elicit unsafe outputs, presenting an ongoing challenge for model alignment. To better understand how different jailbreak types circumvent safeguards, this paper analyses model activations on different jailbreak inputs. We find that it is possible to extract a jailbreak vector from a single class of jailbreaks that works to mitigate jailbreak effectiveness from other classes. This may indicate that different kinds of effective jailbreaks operate via similar internal mechanisms. We investigate a potential common mechanism of harmfulness feature suppression, and provide evidence for its existence by looking at the harmfulness vector component. These findings offer actionable insights for developing more robust jailbreak countermeasures and lay the groundwork for a deeper, mechanistic understanding of jailbreak dynamics in language models.

Decision: reject
Reviewer scores: 5, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730020276834

thread_type:
official_review

<thread_object>
creator: reviewer_5pmp

<content>
summary:
This paper provides an empirical understanding about jailbreak success, specifically focused on the activation vectors. Authors hypothesized that jailbreak attacks with different attacks will have lead to similar underlying intermediate vectors from the language model, and this "jailbreak vector" can be used to promote or suppress the harmful behaviors. Their empirical results validate their claim, showing that (1) different jailbreak steering vectors have high cosine similarities (in Figure 3) and (2) these vectors can be transferred so that jailbreak vector from tactic A can be used to make model safer on tactic B (and other tactics). 

The paper further discussed about the hypothesis that jailbreaking attacks lead LMs fail to detect the harmfulness of the prompt, which is previously argued in Zou et al, 2023. The paper showed the result of PCA anaylsis (in Figure 5) and cosine similarity plots (in Figure 6) to show that jailbreaking often leads to the reducing the harmful features from the harmful instructions. But it also mentioned that low harmfulness always lead to low ASR score.

strengths:
The paper is well-written and the claim is clear.

This paper presents clear empirical observations through multiple experiments about their claim that jailbreak vector exists. The authors tested on multiple LMs (7B-14B models) and tested multiple jailbreak tactics. Also, this claim is meaningful to understand the jailbreak success, and can be used to steer LMs to behave safer without additional fine-tuning.

The paper seems not to over-claim but tried to present their results objectively. For example, in Section 5.4, authors discussed about the harmfulness suppression, mentioning the exceptional cases where low harmfulness scores do not lead to low ASR scores. This helps readers and future researchers to understand the phenomenon in better ways.

weaknesses:
I can't find significant weaknesses from the paper, though I think the paper could be improved if the authors include automatic jailbreak attacks like PAIR or TAP. These attacks don't specify the type of attacks but LLMs automatically find the attack. I think understanding their jailbreak success using latent space dynamics can be helpful to build successful defenses on such attacks.

questions:
- There are bunch of automatic jailbreaking attacks that are trying not to use pre-defined tactics but discover different tactics using LLMs or other optimization-based approaches (for example, TAP or PAIR.) Do these attacks conclude to similar results?
- I wonder this observation can be also applicable to the safety-trained models -- for example, Llama2 or Llama3 are known to have low ASR on well-known jailbreak attacks due to safety training such as SFT or RLHF. Have you tried studying the vectors from these models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618707999

thread_type:
official_review

<thread_object>
creator: reviewer_2nr9

<content>
summary:
This paper aims to understand existing jailbreak methods over LLMs by analyzing the latent space dynamics. First, they constructed jailbreak vectors based on the activations of LLMs calculated on harmful datasets and harmless datasets. They find that the jailbreak vector from a single class of jailbreaks can also work to mitigate jailbreak effectiveness from other semantically-dissimilar classes. Second, they analyze the evolution of cosine similarity between harmfulness direction and activations at each token position for one harmful question without jailbreak (none) and for different jailbreak types. They find that successful jailbreaks have significantly lower representations of harmfulness at the end of instruction for most models, which indicates that the jailbreaks suppress the harmfulness feature on the prompts.

strengths:
1.	It is interesting to use jailbreak vectors for analysis. The paper also demonstrated that jailbreak vectors can  effectively prevent the success of jailbreaks across different types via activation steering, pointing to a shared underlying mechanism.

2.	The paper discussed the concept of models’ perception of prompt harmfulness, and suggested that some jailbreaks succeed by reducing the perception of prompt harmfulness, preventing the refusal response.

weaknesses:
1.	The presentation of methodology part can be improved by providing a workflow figure. This can help people better understand how and where the jailbreak vectors are calculated, and also how the mitigation is performed. 

2.	The findings in Section 5.2 need to be re-evaluated carefully. They find that jailbreak steering vectors have a positive cosine similarity with one another, and hypothesize that jailbreak vectors from one class will work to steer away from successful jailbreaks of other classes. However, the positive cosine similarity could result from the representation degeneration issues, which find that the representations learned by Transformer models tend to clustered in a cone in representation space.

3.	Subtracting jailbreak vectors may reduce ASR, but also potentially destroy the model parameters, making it hard for practice use. For example, in line 371, it may induce content repetition in the response. In addition, it also better if the performance of a general task can be included in this part for comparison.

questions:
1.	In model settings, are the included LLMs trained with RLHF for safety issues? Will this affect the validity of analysis in this paper?

2.	The paper said no sampling is used when decoding. However, in practice, LLM based chatbots usually use sampling for diversity. This discrepancy may affect the conclusions in this paper. A study on the temperature selection can also verify the robustness of the proposed method.

3.	The paper uses middle layers for calculating jailbreak. Did you try the lower or upper layers? Or maybe an ablation study can further enhance our understanding. 

4.	In Section 4.4, the harmless dataset was generated by instructing ChatGPT. Were these samples checked by human annotators to guarantee the data quality?

5.	In line 256, the LLMs do not behave the same. What affects this, data or architecture?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650707167

thread_type:
official_review

<thread_object>
creator: reviewer_c6to

<content>
summary:
This paper investigates how different types of jailbreaks in LLMs might share underlying mechanisms, making it possible to counter multiple jailbreaks using shared “jailbreak vectors.” The study finds that effective jailbreaks tend to reduce a model’s perception of harmfulness, suggesting a way to develop more resilient safeguards against these manipulations.

strengths:
Strengths
1.	Some new Insights on Jailbreaks: the paper explores how different jailbreak types in LLMs might share common mechanics, providing a relatively fresh view on understanding and countering them.
2.	Possible Mitigation Solution: By identifying shared jailbreak properties, the study suggests ways to create more robust defenses against jailbreak attacks.

weaknesses:
Weaknesses
1.	Limited Model Range: The study focuses on only a few models, which may make its conclusions less generalizable. Moreover, the choice of Vicuna lacks sufficient justification, as it may not adequately represent the current SOTA among 7B/13B models.
2.	Related work is not highly related to the research topic of this paper and did not include the most recent studies on understanding the how jailbreak works in the LLMs.

questions:
The authors said "layer 16 for 7B and layer 20 for 13B and 14B parameter models" in line 162, my question is "Multiple layers, rather than a single layer, should be considered as the middle layers. what if we select a different middle layer, e.g., layer 14 for 7B, layer 24 for 13B. will this have some influences on the results?"
</content>
</thread_object>

</thread>
<thread>
creation date:
1731113242841

thread_type:
official_review

<thread_object>
creator: reviewer_yynu

<content>
summary:
This paper explores how jailbreaking techniques work by analyzing latent space dynamics in large language models. The authors found that different types of jailbreaks might share a common internal mechanism and that a jailbreak vector from one class can mitigate others, and that effective jailbreaks can reduce the model's perception of prompt harmfulness. These findings provide valuable insights for understanding jailbreaks and developing better safeguards in large language models.

strengths:
* This paper provides a new perspective to understand jailbreak by analyzing latent space dynamics in large language models. Their findings are very interesting that different types of jailbreaks might share a common internal mechanism and a jailbreak vector can be used to improve/reduce safety of large language models.
* The paper covers many different kinds of jailbreak methods, making it more convincing that there may exist a common internal mechanism in large language jailbreaking.

weaknesses:
* This paper only evaluate 4 models and some of them are not well aligned, e.g. Vicuna is only trained on ShareGPT dataset without RLHF/DPO Stage. In addition, these four models do not behave consistently, e.g. Vicuna shows better cluster patterns than Qwen 14B Chat. In addition, from Vicuna 7B to 13B, I also find that 13B model activation cluster is scattered. This made me question that when the model goes to large sizes and more aligned, will these patterns be less obvious and visible? Hence, I think the authors may compare more aligned models like Llama 3.1 in various sizes and study if they follow similar patterns.

* The authors study how to use the  jailbreak vector to reduce/improve jailbreak success. My question is that in practice, how do we use such techniques? If we minus jailbreak vectors for both harmful and harmless questions, will it reduce the models' helpfulness/correctness/truthfulness? It will be interesting to know these results in general benchmarks like MMLU or AlpacaEval.

questions:
See comments in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732244487278

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response to all reviewers

comment:
We are delighted that the reviewers recognize our paper as providing an important contribution to understanding jailbreak success mechanisms, which we believe to be a critical step toward developing more robust jailbreak mitigation techniques. 

We view our main contributions as the following: 
- Successfully constructing jailbreak steering vectors that work to mitigate the success of other jailbreak types, providing evidence for a shared underlying mechanism.
- Demonstrating that this underlying mechanism could be the jailbreaks’ ability to reduce the models’ perception of prompt harmfulness at the end of instruction. 
- Investigating a comprehensive set of different jailbreak types and models that are aligned via different alignment strategies. 

In the reviewers’ comments, we identified two main themes that we would like to address globally, in addition to providing detailed responses to individual reviewers. These themes are: 
1. Justifying the model choice.
2. Using jailbreak vectors as a defense strategy.

**Concerning point 1: model choice justification**\
Some reviewers were unsure whether the models we chose underwent safety training and were curious whether our results transfer to more strongly aligned models like Llama 3. Regarding the safety training question, the four models we selected are aligned via finetuning, RLHF, or DPO, thereby representing a range of different alignment strategies. To ensure these models were adequately aligned, we checked their refusal rates on 352 harmful scenarios without a jailbreak, showing that the models are capable of refusing the vast majority of these harmful scenarios. If a model was not capable of refusing a specific harmful request, we excluded this request from the steering analyses. 

Our criteria for model selection was such that we included models of different sizes that are aligned but still susceptible to jailbreaks. This choice is determined by the primary goal of our study: to investigate the mechanisms by which jailbreaks succeed. Our selection criteria also explains why we did not include strongly aligned models like Llama 3 since they are robust to our diverse set of jailbreaks. Hence, studying the newest, strongly aligned models would limit our ability to analyze a meaningful range of effective jailbreak types. 

We argue that our focus on susceptible-yet-aligned models is relevant given the lack of general understanding concerning how jailbreaks work. Furthermore, one can expect that models that appear robust today may become more jailbreakable in the future. By providing evidence for a shared underlying mechanism across different jailbreak types and model alignment strategies, our findings provide insights that are likely to generalize to other aligned models.

**Concerning point 2: using jailbreak vectors as a defense strategy** \
We were pleased to see that some reviewers recognized the potential of using our steering vectors as a jailbreak defense strategy. We agree that evaluating the effectiveness of such an approach in practice is crucial, particularly in terms of maintaining response quality on harmless queries. It is furthermore necessary to find the optimal intervention layer and strength as pointed out by some reviewers. However, the primary focus of our paper is on studying the underlying dynamics of jailbreaks, rather than on the comprehensive development or assessment of a defense strategy. We view our work as a foundational step that can inform future research on jailbreak-vector-based defenses. We made this distinction clearer in our paper. 

To answer the reviewers’ questions on whether our results transfer to different steering settings, we repeated our analyses for different layers, a temperature value different from zero, and evaluated a possible steering implementation on the MMLU benchmark. The results of these analyses are discussed in detail in our individual responses to the reviewers. While we incorporated these findings into the paper, as they strengthen our findings, we would like to reiterate that the main contribution of our work is in increasing our understanding of jailbreak dynamics across various jailbreak types and models. This focus does not allow for an exhaustive development of an optimal jailbreak defense strategy, which is an important but distinct line of future research. 

We would like to thank the reviewers again for their valuable feedback and think that we are able to address their remaining concerns in the individual responses and the changes we made to the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studying common underlying mechanisms for different types of jailbreaks in language models and how they can be potentially used to counteract jailbreaking. The reviewers agreed that the paper makes an interesting contribution to our understanding of jailbreak success mechanisms. Also, they agree that finding that jailbreak steering vectors can be used to mitigate the success of other jailbreak types is interesting and has potential implications for developing jailbreak defenses.

Weakness: The reviewers have primarily cited two reasons to reject -- (1) limited number of models used in the study (which that authors have justified, and reviewers didn't raise more concerns), and (2) the defense strategy could have unintended consequences like reduced general capabilities. The authors have argued that understanding jailbreaking is the main problem they study, and defense is not the objective of this paper. Despite this claim, defense is central to main story of this paper as it currently stands and does not seem supplementary, it is mentioned as early as page 1 (figure 1). Shifting focus will require major rewrite and hence, I recommend rejection.

The authors should consider the following suggestions for revision:

1. Include a broader range of models in their study, including models from different alignment strategies and sizes.
2. Provide a more thorough justification for their choice of models.
3. Expand the related work section to include the most recent studies on understanding how jailbreaks work in LLMs.
4. Conduct a more comprehensive analysis of the potential negative effects of using jailbreak vectors as a defense strategy.