Logicbreaks: A Framework for Understanding Subversion of Rule-based Inference

Authors: Anton Xue, Avishree Khare, Rajeev Alur, Surbhi Goel, Eric Wong
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Logic, reasoning, inference, language model, jailbreak

Abstract:
We study how to subvert large language models (LLMs) from following prompt-specified rules.
We first formalize rule-following as inference in propositional Horn logic, a mathematical system in which rules have the form "if $P$ and $Q$, then $R$" for some propositions $P$, $Q$, and $R$.
Next, we prove that although small transformers can faithfully follow such rules, maliciously crafted prompts can still mislead both theoretical constructions and models learned from data.
Furthermore, we demonstrate that popular attack algorithms on LLMs find adversarial prompts and induce attention patterns that align with our theory.
Our novel logic-based framework provides a foundation for studying LLMs in rule-based settings, enabling a formal analysis of tasks like logical reasoning and jailbreak attacks.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730627614160

thread_type:
official_review

<thread_object>
creator: reviewer_7lac

<content>
summary:
One of the most visible limitations of large language models (LLMs) is their vulnerability to jailbreak attacks (i.e. manipulating input prompts with the intent to bypass the LLM's safeguards). Most of works focusing on jailbreak attacks in LLMs attempt to improve LLM defence to jailbreaking using different techniques (like fine-tuning, activation patching, prompt detection, etc), but without addressing the theoretical understandings of this vulnerability.     

This paper investigates this problem and unveil some insights about LLMs that are behind this vulnerability, by demonstrating with a concrete case of propositional logic reasoning. Intuitively, a small language model (LM), composed with one layer and one self-attention head, is constructed to autoregressively predict propositional logic inference. This LM serves as a theoretical reasoner to identify three attacking rules to breakdown inference properties, i.e. monotonicity, maximality and soundness.
Proofs of theorems regarding the validity of these attacks are provided. Additional evidence of the success of these attacks on LLMs (i.e., GPT-2 and Llama-2-7B-chat) is presented in the empirical study.


Overall, I am favourable for an acceptance of this paper.

strengths:
- This work addresses an important and timely problem in LLMs and reveals significant aspects that could help devise better techniques against jailbreaking.
- Overall, the paper is technically rigorous, well-organized, and clear. Moreover, it enhances understanding of the aims of this work by providing intuitive examples.
- The experimental evaluation is comprehensive, and the empirical results support the theoretical findings.

weaknesses:
- The paper does not discuss possible alternatives to address jailbreak vulnerability in LLMs

questions:
- Is it possible to relate the theoretical fundings of this work with safety issues in code generation with LLMs?


Minor comments:

line 357: This is evidence that = > evident?
line 497: positional encoding => propositional encoding?
line 498:  'quantifiers' can be use in first-order logic or above but not propositional logic.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645037028

thread_type:
official_review

<thread_object>
creator: reviewer_p6rj

<content>
summary:
The paper investigates adversarial attacks against large language models (LLMs) through the lens of propositional logic. The authors represent sets of propositions in a vectorized form and model rule disjunctions as summations of binary elements of these vectors. Attacks are performed by removing some (of the indicators of) propositions to break the logical soundness of rule applications. Additionally, small transformer architectures are used to predict the next state, mimicking rule application. Experiments with LLMs are conducted on synthetic data.

strengths:
- The formalization of different types of rule-based adversarial attacks is novel, to the best of my knowledge, and provides a structured way of thinking about how LLMs might fail in following logical rules. 

- The authors’ attempt to develop a theoretical framework that connects LLM behavior with logical rule-following is conceptually interesting and could inspire further work in this area. 

- The use of synthetic data and small transformer architectures for experimentation allows for controlled exploration of the proposed ideas, though this setup has its limitations.

weaknesses:
- The paper's clarity needs improvement, particularly in terms of defining key terms and methods. The architecture of the theoretical model is not sufficiently explained and the theoretical results not easy to trust given the lack of intuition about why they come about. 

- The paper makes a fundamental assumption that LLMs can follow prompt-specified rules, but this assumption is not validated. In fact, prior work, e.g. (Zhang at al., 2022a), disprove this, and the experiments indicate that LLMs struggle with logical reasoning, a point that is not fully acknowledged in the paper. A stronger link between theoretical expectations and empirical findings is needed. 

- The theoretical model is not a strong representation of how LLMs operate, as it treats propositions as binary events, and simulate inference by summation which is not clear whether can generally represent of LLMs' behavior, e.g. performing next-token prediction. The correspondence between the transformer architecture and the theoretical model is weak, and the results do not demonstrate rule-following as claimed. 

- The experiments are not adequately explained, and it’s unclear how well the theoretical attacks translate to learned models. For example, the empirical results with GPT-2 lack sufficient explanation regarding how the theoretical attacks were applied to the LLM. The lack of clarity in the methodology undermines confidence in the results.

questions:
Content-related: 

- L49: The paper claims that the proposed logic-based framework can detect and describe rule disobedience by LLMs. I am not convinced that the claim holds since I cannot see a concrete correspondence between the proposed theoretical model and a transformer architecture like the ones you experiment with. This is probably due to presentation, since I did not find Eq (4) and (5) to be adequately described or justified. They read like a list of ingredients to me. Also the theorems are too informal and at least an intuition of why they are the case should be given. 

- L53: The paper states that attacks in the theoretical setting transfer to learned models and that LLMs exhibit consistent reasoning behaviors with the theoretical model—can you provide evidence for this? I did not find the presented evidence clearly supporting this claim. 

- L141: "Might violate rule-following differently" refers to how predictions diverge from the ground truth. L159 mentions "non-determinism in rule-application order"—does this mean the result of Apply() is not unique? Could you expand on this source of uncertainty? 

- L155: What do you mean by "good coverage"?  

- L172: What result are you referring to here? The following model description is unclear, and some background and definitions are missing. 

- L185: Are the rules represented as a set of tuples? $\{0,1\}^{2n}$ is simply a set. 

- L195: When and how does thresholding come into play? 

- L209: How does the network's dimension support the results in Theorem 3.1? 

- L255: Is the reason for negative values of the attacks due to the embedding space and the way language models "reason"? 

- L262: Empirical results with GPT-2 are mentioned without explaining how theoretical attacks were translated to learned models. How does a model like GPT-2 fit into this framework? 

- L299: GCG is not introduced or described appropriately. 

- L322: How is the search for expected behavior conducted? 

- L357: Linear classifier probes are mentioned but not defined. 

- L365: Is there a difference between what is measured by accuracy and F1? I find the description of the metrics confusing. 

- L371: What are models T=1,3,5? You referred to models as $\mathcal{R}$ earlier—could you please clarify. 

- L374: SSR is not properly defined. 

- Table 2: What is the baseline here? How can you say that the jailbreak succeeded if results without the attack are not shown? 

- Table 4: Why is the suppression effect more pronounced in layers 6, 7, and 8? Is there something that can be inferred from this? 

- Figure 6: The figure is unclear. What is the prompt, the rule, the attack, and the output? 

Minor points: 

- L134: Equivalence to HORN-SAT should be referenced. 

- L187: Typo: "autogregressively" should be "autoregressively." 

- L464: Typo: "does" should be corrected.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660712804

thread_type:
official_review

<thread_object>
creator: reviewer_9pde

<content>
summary:
The authors investigate how adversarial suffixes can cause LLM to subvert from following rules specified in the input prompt. To do so, the authors introduce a theoretical framework using propositional Horn logic, which involves multiple inference steps. At each inference step, new facts are derived using a set of rules and a set of known and derived facts. Within this framework, the authors investigate how and which adversarial suffixes make LLMs violate the rules defined in the prompts. Here, the authors investigate three attack scenarios: fact amnesia (deletion of facts), rule suppression (the rule is not applied even though it could have been), and state Coercion (the current state is manipulated).

strengths:
- Understanding why and how these jailbreak attacks work on LLMs is an interesting and important direction of research.
- The paper is overall well-written and follows a clear structure. However, as an informed outsider, I find it hard to follow; while the theory is described in great detail, it could benefit from some more high-level insights, which would make it more accessible.
- The authors provide both theoretical and empirical support for their framework to explain the behavior of LLMs under such jailbreak attacks.

weaknesses:
- The paper focuses on small language models, mostly GPT2, and for some experiments, llama2-7B-Chat. I am unsure how the results scale for even larger and deeper models and whether they could be applied to current LLM architectures (e.g., llama-3). 
- The Authors include an invalid link within the reproducibility statement for code and experiments.
- As an informed outsider, the paper seems to be very technical, and I struggle to see the main insides you draw from the theory and empirical findings. Maybe a short elaboration on how and why these suffixes work could clarify that.

questions:
- why do we need so many repeats for the attack in Fig. 3, and why is the success rate dropping if the number is too high?
- Is it more challenging to apply your code to larger and deeper LLMs from an implementation standpoint?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710897682

thread_type:
official_review

<thread_object>
creator: reviewer_ktce

<content>
summary:
The paper investigates methods to manipulate LLMs into deviating from prompt-specified rules. The study models rule-following as propositional Horn logic. The authors demonstrate how malicious prompts can exploit theoretical weaknesses, leading even robust models to fail in rule-based reasoning. This approach is validated both theoretically and empirically, linking the behavior of real-world adversarial attacks (jailbreaks) to their framework.

strengths:
1. A theoretical model is provided to study how the reasoning of transformer-based language models can be subverted. This approach bridges theory and practice, demonstrating that popular jailbreak attacks align with theoretical predictions. The presentation of the idea is quite clear.

2. The framework is extendable to various LLMs, making it relevant for broader applications in model safety and adversarial robustness. Experimental results align with the theoretical analysis.

weaknesses:
1. The theoretical model is simplified by considering only Horn logic without quantifiers. 


2. A graph-based representation, rather than binary vectors, might provide a stronger representation of propositions but could be more challenging to analyze. 


3. While the study includes empirical validation, the range of language models and datasets tested may not fully capture the diversity of existing models.  Although recent models like GPT-3+ are not controllable at will, experiments with these models may yield unexpected results, and additional insights could be explored.

questions:
1. How should we interpret the curve in the left and middle figures, where the ASR initially increases but then decreases with additional repetitions?

2. Does this alignment between theory and behavior appear only in your self-trained GPT-2 and Llama reasoners, or can we expect similar results in more advanced LLMs, such as GPT-4 or Llama 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731290169590

thread_type:
official_review

<thread_object>
creator: reviewer_pvai

<content>
summary:
This manuscript explores the subversion of logical entailment in transformer-based LLMs, in three steps:
1. a theoretical/mathematical representation of a single layer transformer with one attention head (§3.1): Theorem 3.1 establishes that this architecture can encode (propositional) logical entailment, characterized by _monotonicity_, _maximality_, and _soundness_.
1. theory-based attacks against GPT-2 reasoners:
   1. Theorem 3.3 presents binary encodings of theory-derived suffixes that implement attacks on each of the three logical properties, via the mathematical representation of the transformer.
   1. Figure 3 then shows the success rate of each of the mathematical attacks.  Those against monotonicity and maximality are highly successful, while that on soundness generally fails, but can induce variance.  These attacks are somewhat validated by 'learned' attacks that minimise the BCE loss between a desired sequence of reasoning, and an actual induced sequence of reasoning.
1. finally, the authors use the Greedy Coordinate Gradients (GCG) algorithm for generating adversarial attacks on GPT-2 and Llama-2 to induce the specific logical violations sought. Linear classifier probes tend to recover the induced adversarial states, rather than the true states, indicating the attacks' success.

strengths:
**originality**

While 'jailbreak'/'adversarial attack' papers are common, I have yet to see a paper that embeds such attacks in the transformer architecture underlying LLMs.

**quality**

The paper seems well conducted.

**clarity**

The paper is generally well structured.  For specific exposition, see below.

**significance**

I think that the paper contributes to the literature on LLM's ability to implement logical operations.

weaknesses:
1. throughout, as a non-expert in adversarial attacks, I found that steps in the paper's reasoning were hard to understand.  For example, I would like clearer explanation of:
   1. the relation between the binary encodings and the Minecraft prompts: are terms like "if I have" and "then I can" somehow encoded to binary in this form, or are the implied logical operators encoded instead?
   1. is there anything special about adversarial _suffixes_ (rather than e.g. adversarial prefixes)?
   1. why is the ASR non-monotonic in the number of repeats (Fig. 3)?  I would have thought that a repeated attack is more likely to succeed.
   1. I did not understand the variance explanation (Fig. 3) of the soundness attacks: my best guess is that adversarial suffixes can induce similar reasoning states across a range of prefixes; if so, this seems unsurprising - the tokens generated when a common token is appended to idiosyncratic tokens are more correlated than those generated in response to the idiosyncratic tokens alone?
   1. what is a 'budget' (problem 3.2)?
   1. Table 1: this seems to show that learned attacks are successful.  
      1. Do they 'mirror' the theoretical attacks in the sense of implementing similar reasoning sequences, or using similar adversarial prefixes?
      1. The caption claims that $v_{tgt}$ is larger on average than $v_{other}$ for fact amnesia; the figures in the table show the opposite?
      1. The theory based attacks failed against soundness, yet the learned attacks seem to be quite successful.  In Table 2, the soundness attack seems the most successful.  Why is there this difference between the attack types' success?
   1. The Suffix in Figure 4 contains various typos (e.g. "I and have", "and and").  Is this an intrinsic part of the attack, or just incidental?
   1. Table 3: why are there no confidence intervals on the substitution ASR?
   1. Table 4: is there any intuition behind the most targeted layers being consecutive?  Why is the pattern in Table 5 so different?
   1. Figure 6: should the block between 'Suppressed Rule' and 'Output' be labeled?
1. neither of the two LLMs used, GPT-2 and Llama-2, have been state-of-the-art for some time.  It would be helpful to either:
   1. (ideally) demonstrate the same results on SOTA LLMs; or
   1. (less ideally) explain why the non-SOTA LLMs still provide insight into the weaknesses of SOTA LLMs.

None of the expositional issues, above, are 'deal breakers'.  Thus, I have rated this paper as a weak accept: I would like to papers in top ML conferences like ICLR to be strong across the board, rather than good analyses that seem a bit rushed in their exposition.

questions:
See 'Weaknesses', above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732055537616

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Preliminary Global Response to Reviewers

comment:
We thank the reviewers for their helpful and constructive feedback. Their comments, questions, and suggestions have helped us identify important areas for improvement. While we work on an updated manuscript, we first give a preliminary response below to address some common questions. Then, once the updated manuscript is ready, we will give detailed responses to each reviewer.


## Addition of Newer Models
Several reviewers pointed out that adding newer LLMs would strengthen our experiments. We have re-run the LLM experiments with Llama3 (specifically, meta-llama/Meta-Llama-3-8B from HuggingFace) and observed behaviors similar to those of our existing Llama2 results. Below are the layer-wise attention values of Llama3 that compare the attention values for a benign (No Atk) vs. adversarial prompt (Yes Atk) as in Table 5, where the prompts are taken from the dataset as described in Fig. 10.

| Layer | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| No Atk | 0.64 | 0.27 | 0.73 | 0.11 | 0.59 | 0.66 | 0.7 | 0.47 | **0.84** | 0.67 | 0.78 | 0.43 | 0.25 | 0.53 | **0.8** | **0.98** |
| Yes Atk | 0.46 | 0.21 | 0.31 | 0.1 | 0.17 | 0.34 | 0.29 | 0.23 | 0.52 | 0.33 | 0.35 | 0.28 | 0.11 | 0.43 | 0.42 | 0.44 |


| Layer | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| No Atk | **0.89** | 0.57 | 0.5 | 0.63 | **0.85** | 0.53 | 0.69 | 0.56 | 0.78 | 0.57 | 0.52 | 0.66 | 0.47 | 0.25 | 0.44 | 0.24 |
| Yes Atk | 0.43 | 0.5 | 0.25 | 0.23 | 0.31 | 0.37 | 0.34 | 0.18 | 0.32 | 0.4 | 0.27 | 0.15 | 0.2 | 0.19 | 0.13 | 0.07 |


## Non-monotonicity of the Attack Success Rate in Fig. 3
Some reviewers pointed out the counterintuitive result in Fig. 3, wherein the attack success rate (ASR) was not monotonic in the attack strength (number of tokens). We attribute this to our use of stringent criteria of "success," wherein the model must exactly induce all bits of the theoretically predicted adversarial target when given an adversarial token sequence. When we relax this criterion of success and require only that the adversarial suffix induces any incorrect output, the ASR is mostly monotonic, with the exception of some noise. Importantly, there is no sharp drop-off similar to that of Fig. 3, which uses a strict measure of ASR.

Below is a table of results for the **Fact Amnesia** attack.

| Num Repeats | n16 | n32 | n48 | n64 |
| --- | --- | --- | --- | --- |
| 1 | 0.021 | 0.201 | 0.268 | 0.015 |
| 2 | 0.855 | 1.000 | 0.995 | 0.892 |
| 3 | 1.000 | 1.000 | 0.999 | 1.000 |
| 4 | 1.000 | 1.000 | 0.987 | 1.000 |
| 5 | 1.000 | 0.999 | 0.976 | 1.000 |
| 6 | 1.000 | 0.995 | 0.998 | 1.000 |
| 7 | 1.000 | 1.000 | 1.000 | 0.990 |
| 8 | 1.000 | 1.000 | 0.975 | 0.983 |

Below is a table of results for the **Rule Suppression** attack.

| Num Repeats | n16 | n32 | n48 | n64 |
| --- | --- | --- | --- | --- |
| 1 | 0.021 | 0.006 | 0.005 | 0.004 |
| 2 | 0.083 | 0.078 | 0.247 | 0.056 |
| 3 | 0.569 | 0.962 | 0.944 | 0.817 |
| 4 | 0.957 | 1.000 | 0.999 | 0.998 |
| 5 | 0.999 | 1.000 | 1.000 | 1.000 |
| 6 | 1.000 | 1.000 | 1.000 | 1.000 |
| 7 | 1.000 | 1.000 | 1.000 | 1.000 |
| 8 | 1.000 | 1.000 | 1.000 | 1.000 |

We will upload the updated manuscript once it is ready and provide detailed responses to each reviewer.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper considers the problem of manipulating LLMs so that they deviate from prompt-specified rules. Overall, the reviews lean towards acceptance, and I agree. The overall take-away message "Language Models are Susceptible to Inference Subversions" is important. The in-depth discussion between one reviewer and the authors helped clarify potential downsides. As one reviewer has put it, the study models rule-following as propositional Horn logic, and the authors demonstrate how malicious prompts can exploit theoretical weaknesses, leading even robust models to fail in rule-based reasoning. This is interesting. However, please clearly state that the Horn rules considered are quite simple as they only involved three literals. Generally, a Horn clause is a clause with at most one positive, i.e., unnegated literal. This has to be pointed out.