Title: Attention Slipping: A Mechanistic Understanding of Jailbreak Attacks and Defenses in LLMs
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Large Language Models, Attention, Jailbreaks

Abstract: As large language models (LLMs) become more integral to society and technology, ensuring their safety becomes essential. Jailbreak attacks exploit vulnerabilities to bypass safety guardrails, posing a significant threat. However, the mechanisms enabling these attacks are not well understood.  In this paper, we reveal a universal phenomenon that occurs during jailbreak attacks:  **Attention Slipping**. During this phenomenon, the model gradually reduces the attention it allocates to unsafe requests in a user query during the attack process, ultimately causing a jailbreak. We show **Attention Slipping** is consistent across various jailbreak methods, including gradient-based token replacement, prompt-level template refinement, and in-context learning. Additionally, we evaluate two defenses based on query perturbation, Token Highlighter and SmoothLLM, and find they indirectly mitigate **Attention Slipping**, with their effectiveness positively correlated with the degree of mitigation achieved. Inspired by this finding, we propose Attention Sharpening, a new defense that directly counters **Attention Slipping** by sharpening the attention score distribution using temperature scaling. Experiments on four leading LLMs (Gemma2-9B-It, Llama3.1-8B-It, Qwen2.5-7B-It, Mistral-7B-It v0.2) show that our method effectively resists various jailbreak attacks while maintaining performance on benign tasks on AlpacaEval. Importantly, Attention Sharpening introduces no additional computational or memory overhead, making it an efficient and practical solution for real-world deployment.

TL;DR: A Mechanistic Understanding of Jailbreak Attacks and Defenses in LLMs Via Attention Perspective.