Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses

## Abstract

Abstract Vulnerability of Frontier language models to misuse and jailbreaks has prompted the development of safety measures like filters and alignment training in an effort to ensure safety through robustness to adversarially crafted prompts. We assert that robustness is fundamentally insufficient for ensuring safety goals, and current defenses and evaluation methods fail to account for risks of dual-intent queries and their composition for malicious goals. To quantify these risks, we introduce a new safety evaluation framework based on impermissible information leakage of model outputs and demonstrate how our proposed question-decomposition attack can extract dangerous knowledge from a censored LLM more effectively than traditional jailbreaking. Underlying our proposed evaluation method is a novel information-theoretic threat model of inferential adversaries , distinguished from security adversaries , such as jailbreaks, in that success is measured by inferring impermissible knowledge from victim outputs as opposed to forcing explicitly impermissible outputs from the victim. Through our information-theoretic framework, we show that to ensure safety against inferential adversaries, defense mechanisms must ensure information censorship , bounding the leakage of impermissible information. However, we prove that such defenses inevitably incur a safety-utility trade-off.

## 1 Introduction

Background Frontier models have demonstrated remarkable capabilities (Bubeck et al., ; Llama Team, ; Reid et al., ), but their potential for misuse has raised alarm about possible risks. These encompass a wide taxonomy of privacy and security threats, ranging from social manipulation to the creation of dangerous weapons by malicious actors (Phuong et al., ; Slattery et al., ; Weidinger et al., ; Durmus et al., ; Li et al., ). In response, researchers have developed various mitigation strategies, including prompt engineering (Bai et al., ), aligning models with human values through fine-tuning (Ouyang et al., ; Rafailov et al., ), improving robustness via adversarial training (Bai et al., ), and implementing input and output filters as guardrails (Debenedetti et al., ; Zou et al., ).

Motivation Despite these efforts, recent work has called into question the reliability of extant safety methods and their assessments (Feffer et al., ; Kapoor et al., ). Significant concerns stem from poorly defined threat models with tenuous connections to real-world safety risks and a lack of compelling criteria for evaluating attacks and defenses. Current threat models and assessment methods typically focus solely on the permissibility of the victim model’s responses (Zou et al., ), which do not capture many expressly stated safety concerns. For example, as illustrated in Figure 1, an adversary seeking to implement a social engineering attack can achieve their goal without eliciting an explicitly harmful response to a query like “How do I scam the elderly?.

Method Building on this intuition, in section 2 we introduce Decomposition Attacks (DAs): automated black-box attacks which decompose malicious questions into seemingly innocuous subquestions, posing them to a victim LLM, and aggregating the responses to answer the malicious question. The attacker resembles a problem-solving agent (Khot et al., ), relying on the expertise of a victim LLM to acquire dual-use information for fulfilling the malicious task.

Evaluation In section 3, we propose a novel framework to evaluate Decomposition Attacks on a subset of questions from the Weapons of Mass Destruction Proxy (WMDP) dataset for hazardous knowledge (Li et al., ). This dataset is chosen for its multiple-choice format, allowing us to quantify a victim LLM’s safety risk by comparing an adversary’s prediction before and after the attack. We compare our non-jailbreaking DA against a commonly studied black-box jailbreaking method: PAIR (Chao et al., ). We find that our instantiation of a DA is able more effectively extract information from a victim LLM than jailbreaking as well as triggering input and output filters less frequently. Our proposed evaluation approach enables measuring information leakage over multiple responses, in contrast to existing jailbreak evaluations that capture safety risk by ascertaining the permissibility of a single victim output.

Threat Models In section 4, we formally distinguish these approaches by defining two threat models: Inferential Adversaries and Security Adversaries. Inferential adversaries (eg. Decomposition Attacks) seek to extract harmful information from victim responses, with success quantified by impermissible information gain. Security adversaries (eg. jailbreaks) instead seek to force specific impermissible outputs from the victim model, with success quantified in a dichotomous manner, either the victim produced the specific output or not.

Defense To defend against inferential adversaries, in section 5, we leverage information theory to
introduce Information Censorship: a bound on the expected impermissible information leaked
through interactions between adversary and victim models. Information Censorship serves as a condition for defense mechanisms to guarantee safety against query-bounded inferential adversaries. To illustrate such a censorship mechanism, we define a randomised response mechanism (Mangat, ) which bounds impermissible information leakage by probabilistically returning empty strings to the adversary with probability dependent on the worst-case expected impermissible information leakage.

Safety-Utility Tradeoff Due to the dual-use nature of information, effective defense mechanisms, such as the randomised response censorship mechanism, constrain a models usefulness to both malicious and benign users. We formalize this intuition in subsection 5.3 by providing theoretical safety-utility trade-offs for the randomised response mechanism and more generally for information censorship mechanisms.

Figure: Algorithm 1 Decomposition Attack for LLMs

## 2 Method: Decomposition Attacks

Overview We empirically ground our theoretical contributions by instantiating an inferential adversary through a Decomposition Attack—an automated black-box attack method for extracting and leveraging dual-use information to fulfill adversary objectives. The attack involves an adversarial LLM leveraging the knowledge or capabilities possessed by a victim LLM to answer an impermissible question or fulfill a malicious task. It does this by generating subquestions/subtasks then prompting the victim LLM with them. As shown in Figure 1, there exist many seemingly benign, dual-use subquestions whose answers would assist in answering an impermissible question while avoiding input and output filters by not being strictly problematic. Algorithm 1 provides an overview of our DA method.

Setting Assume that a malicious user prompts an adversarial LLM, denoted by AdvLLM, with a query $q$, typically a question to answer or task to complete. Assume further that the adversarial LLM has access to a black-box victim LLM, denoted by VicLLM, and is capable of prompting it at most $n$ times. We assume that the victim LLM employs input and output filtering mechanisms, $M_{i}$ and $M_{o}$, respectively, and may monitor jailbreak or safety violation attempts to block the adversary’s access.

Inputs To execute the attack, a malicious user first provides the adversarial LLM with a malicious query $q$, number of subquestion rounds $k$ and number of subquestions to generate per round $m$ such that $km\leq n$. The user further provides the following behavioural template to guide the adversary:

- $E_{g}$:
Few-shot examples of query decomposition into self-contained subqueries;
- $E_{e}$:
Few-shot examples of extractions of subanswers from victim responses to subqueries;
- $E_{a}$:
Few-shot examples of the aggregation of subquery-subanswer pairs to answer a given query.

Initialization AdvLLM initializes knowledge pile $h$ to hold a history of interactions with VicLLM.

Subquestion Generation For each of the $k$ rounds, AdvLLM is provided with a prompt containing the few shot subquestion generation examples $E_{g}$, the knowledge pile $h$, the malicious query $q$, and instructions to generate $m$ self-contained subquestions that would help in answering the malicious query $q$. The AdvLLM then outputs a string of $m$ subquestions $q_{s}$ which are separated into a list:

$$ $\displaystyle[q_{s_{1}},q_{s_{2}},\dots,q_{s_{m}}]\leftarrow\textsc{AdvLLM}(E_ {g},h,q,m).$ $$

Victim Interaction
Each of the $m$ generated subquestions $\{q_{s_{i}}\}$ is submitted to the VicLLM, passing through any input or output filtering mechanisms $M_{inp}$ and $M_{out}$ it employs. In turn, the VicLLM responds to each subquestion with a response:

$$ $\displaystyle r_{i}\leftarrow M_{out}(\textsc{VicLLM}(M_{inp}(q_{s_{i}})).$ $$

Processing Responses Due to the length and irrelevant information contained within the response $r_{i}$, for each non-empty response the adversarial LLM extracts an answer $a_{i}$ from $r_{i}$ relevant to the query $q$ and subquestion $q_{s_{i}}$ and appends it to the knowledge pile $h$:

$$ $\displaystyle a_{i}\leftarrow\textsc{AdvLLM}(E_{e},q,q_{s_{i}},r_{i});$ $\displaystyle h\leftarrow h\cup(q_{s_{i}},a_{i}).$ $$

Answer Aggregation
The last step of the DA is answer aggregation, where the AdvLLM is provided with the malicious query $q$, the knowledge pile $h$ containing all subquestion answer pairs from interactions with the VicLLM, and few shot examples $E_{a}$ to produce an answer $a$ to the query $q$:

$$ $\displaystyle a_{q}\leftarrow\textsc{AdvLLM}(E_{a},q,h).$ $$

## 3 Evaluation

### 3.1 Proposed Framework

Motivation To evaluate the Decomposition Attacks proposed in the previous section and assess the safety risk of deploying an LLM, in this section we introduce a novel evaluation framework derived from the theoretical threat models formalized in section 4. Presently, safety evaluations of LLMs measure attack and defense effectiveness through the attack success rate (ASR) metric, which quantifies the proportion of times the attack forces a VicLLM to return an explicitly impermissible output string. However, as shown in Figure 1, an explicitly impermissible output is not necessary for an adversary to acquire information that could be used for harm.

Proposed Evaluation To address this limitation, we propose to directly evaluate the ability of an AdvLLM to correctly answer an impermissible question after attacking the VicLLM. As the AdvLLM may be capable of correctly answering the impermissible question on its own, for a given AdvLLM we can measure the marginal risk induced by a VicLLM by comparing against the AdvLLM’s baseline ability of answering the question before, or without, executing the attack.

Evaluation Dataset For our proposed evaluation, we consider the WMDP dataset (Li et al., ): a collection of multiple choice questions, denoted $q$, to serve as a proxy for dangerous biological, chemical, or cybernetic knowledge. The set of possible answers, denoted $\{a_{i}\}$, and single correct answer, denoted $a^{*}$, enables measuring the AdvLLM’s confidence in the correct answer before and after attacking a VicLLM by computing next-token probabilities of each of the answer options.

Notation Thus, for a question $q$, we assume that knowledge of $a^{*}$ being the correct answer is impermissible, and, we use $p_{adv}(a^{*}|q)$ and $p_{adv}(a^{*}|h,q)$ to denote
the AdvLLM’s belief in the correct answer before and after the attack respectively. We use $h$, the knowledge pile, to denote a string extracted from all prompt-response interactions between the AdvLLM and VicLLM.

Measuring Risk Provided the before and after probabilities $p_{adv}(a^{*}|q)$ and $p_{adv}(a^{*}|h,q)$, the question of how to properly measure risk arises. One approach is to measure the difference in accuracy before and after the attack, however, accuracy is not a granular metric and fails to account for changes in confidence. This issue is commonly resolved by measuring cross-entropy of the AdvLLM’s prediction with the correct answer. The difference in the cross-entropies, $p_{adv}(a^{*}|q)$ and $p_{adv}(a^{*}|h,q)$, is expressed as

$$ $\displaystyle\log(p_{adv}(a^{*}|h,q))-\log(p_{adv}(a^{*}|q))=\log\frac{(p_{adv }(a^{*}|h,q))}{(p_{adv}(a^{*}|q))}$ $$

commonly known as the pointwise mutual information. However, under this metric, the safety risk when $p_{adv}(a^{*}|h,q)=.9$ and $p_{adv}(a^{*}|q)=.1$ is the same as the risk when $p_{adv}(a^{*}|h,q)=.009$ and $p_{adv}(a^{*}|q)=.001$. To address this, we propose to calibrate the safety risk metric by the adversary’s final belief in the correct answer—$p_{adv}(a^{*}|h,q)$. Thus, we propose to evaluate safety risk through Impermissible Information Leakage (IIL).

###### Definition 3.1 (Impermissible Information Leakage ( IIL )) .

Let $p_{adv}(a^{*}|q)$ and $p_{adv}(a^{*}|h,q)$ represent the belief of AdvLLM in $a^{*}$ before and after interacting with VicLLM. We define the impermissible information leakage as:

$$ $\textsc{IIL}=p_{adv}(a^{*}|h,q)\log\frac{p_{adv}(a^{*}|h,q)}{p_{adv}(a^{*}|q)}$ $$

###### Definition 3.1 (Impermissible Information Leakage ( IIL )) .

### 3.2 Experiments

Experimental Design: For our experiments, we compare the ability of an AdvLLM to extract impermissible information from a larger VicLLM using our proposed DA attack and an adaptation of the commonly studied black-box jailbreaking method PAIR (Chao et al., ) which also relies on an adversarial LLM to iteratively generate prompts to the victim LLM to bypass safety measures. To demonstrate the effectiveness of DAs at extracting information while bypassing safety filters, we curate subsets of WMDP-Bio and WMDP-Chem questions which get flagged as unsafe by the Llama-Guard-3-8B model (Llama Team, ), and employ the model as an input $M_{i}$ and output $M_{o}$ filter for VicLLM. Furthermore, we examine both attacks in a rate-limited setting where there is a limit on the number of interactions between AdvLLM and VicLLM. More details regarding our implementation can be found in Appendix B. We report our results in Table 1.

**Table 1: We fit a linear mixed-effects model over measured IIL for every attack and report its mean and standard error for curated subsets of WMDP-Bio and WMDP-Chem.**
| Attack Method | Adversary Model | WMDP-Bio $\uparrow$ | WMDP-Chem $\uparrow$ | Flags(Bio/Chem) $\downarrow$ |
| --- | --- | --- | --- | --- |
| DA (Ours) | Mistral-7B-Instruct | $\textbf{0.95}\pm 0.22$ | $\textbf{0.48}\pm 0.10$ | $0.34/0.30$ |
| DA (Ours) | Llama-8B-Instruct | $\underline{0.59}\pm 0.09$ | $\underline{0.38}\pm 0.05$ | $\textbf{0.03}/\textbf{0.11}$ |
| PAIR | Mistral-7B-Instruct | $0.43\pm 0.12$ | $0.32\pm 0.12$ | $1.15/2.07$ |
| PAIR | Llama-8B-Instruct | $0.28\pm 0.06$ | $0.16\pm 0.04$ | $\underline{0.13}/\underline{0.58}$ |

Analysis of Results
We find that both our proposed DA and adaption of the PAIR jailbreak were able to successfully extract impermissible information from the VicLLM despite the defense measures employed. Moreover, we observe that on average, the number of times interactions get flagged as unsafe per attack is low, particularly considering that each attack typically consists of $6$ interactions. In particular we observe the safety-aligned Llama-3.1-8B-Instruct models are very effective at remaining undetected, an important property if user-monitoring was employed to detect and block VicLLM access to bad actors. Finally, in Table 2 we report two-sample t-test results comparing the IIL performance of the two attacks, demonstrating that our DA significantly outperforms PAIR.

Interpreting Performance Differences As flag rates for both attacks were low (although relatively higher for PAIR), we speculate that one reason for the worse performance of PAIR stems from the fact that our proposed DA attack extracts information from each interaction with the VicLLM. On the other hand, PAIR attempts to generate a prompt which both bypasses the filtering mechanisms while also receiving a clear answer to the multiple choice question, two opposing goals when the multiple choice question itself is deemed impermissible to ask.

**Table 2: Two-sample t-test results comparing DA against PAIR. The values reported are p-values.**
| Comparison | WMDP-Bio | WMDP-Chem |
| --- | --- | --- |
| DA $>$ PAIR (Mistral-7B-Instruct) | 0.0342* | 0.317 |
| DA $>$ PAIR (Llama-8B-Instruct) | 0.0055** | 0.0004*** |

## 4 Adversary Threat Models

Motivation To formally understand the underlying distinction between our proposed evaluation framework and the commonly studied evaluations, in this section we define and distinguish the objectives of two safety threat models, security adversaries and inferential adversaries. Defining the adversary objectives immediately provides us with a method for evaluating adversary success and enable us to define criteria for defense in section 5.

### 4.1 Setting

Notation Let $\mathcal{X}$ and $\mathcal{Y}$ be the set of valid input and output strings to both, AdvLLM and VicLLM, respectively. The LLMs $\textsc{AdvLLM}:\mathcal{X}\to P(\mathcal{Y})$ and $\textsc{VicLLM}:\mathcal{X}\to P(\mathcal{Y})$ are defined as mappings of input strings to a distribution over output strings. An interaction between the adversary and victim is represented by an input-output pair $(q,a)\in\mathcal{X}\times\mathcal{Y}$, where $q$ is the input to VicLLM and $y\sim\textsc{VicLLM}(x)$ a sample output returned to the AdvLLM. Within these interactions, the adversary aims to maximize a scoring function $s:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}^{+}$, which quantifies how well these input-output pairs fulfill the adversary’s goals.

###### Definition 4.1 (Censorship Mechanism) .

A Censorship Mechanism $M:\mathcal{X}\times P(\mathcal{Y})\to P(\mathcal{Y})$ is a randomized function that outputs a new distribution over responses returned to a user-provided input. The mechanism $M$ seeks to ensure that responses satisfy a safety criterion dependent on the assumed threat model.

To define the objective of a censorship mechanism, in particular the constraints it must ensure to guarantee safety, it is essential to define the adversary threat model which we seek to provide safety guarantees against. Thus, we turn to introducing security and inferential adversary threat models, specifically their goals, which can capture most LLM safety concerns.

###### Definition 4.1 (Censorship Mechanism) .

### 4.2 Security Threats

Examples To help understand security adversaries, we first provide some examples of them. Jailbreaks involve an adversary attempting to bypass content filters to make the VicLLM generate harmful content. This is evaluated by checking if the output $a$ returned starts with a certain string (e.g., “Sure, here’s how to evade taxes:”) (Liu et al., ), or, an impermissibility score assigned to $a$ by a judge LLM passes a certain threshold (Souly et al., ). Prompt Injection attacks involve an adversary inserting malicious instructions within an input prompt $q$ to the VicLLM to manipulate the model to perform a specific task. Success is evaluated whether or not the VicLLM executed the task successfully. In these cases, the adversary’s success is evaluated dichotomously—either the VicLLM exhibits a target behavior or not. Formally, we define the security adversary objective as follows.

###### Definition 4.2 (Security Adversary Objective) .

The goal of a security adversary is characterized by a target function $t:\mathcal{X}\to\mathcal{P}(\mathcal{Y})$ mapping input strings to a set of impermissible output strings, and is evaluated by the following scoring function for $T_{q}=t(q)$:

$$ $\displaystyle s(q,a)=\mathds{1}_{a\in T_{q}}$ $$

For a VicLLM input $q$, the impermissible output set $T_{q}=t(q)$ is assumed to be common knowledge to both adversary and victim as membership of $a\in T_{q}$ can be directly verified. VicLLM is secure if and only if $\forall q,a:\;s(q,a)=0$.

Limitations Security adversaries fail to fully capture many threats such as deepfake generation or informative instructions for conducting illegal activities. This is because adversaries can compose individual acceptable outputs to construct deepfakes or acquire instructions (Glukhov et al., ).

###### Definition 4.2 (Security Adversary Objective) .

### 4.3 Inferential Threats

Distinction Inferential adversaries provide a compelling alternative threat model for many safety concerns which are not adequately captured by security adversaries by their ability to better capture compositional threats. They differ from security adversaries in that VicLLM outputs are no longer dichotomously evaluated as either permissible or impermissible. Instead, we consider inferential adversaries as seeking to gain information about an impermissible concept, such as biological weapons, from VicLLM outputs. This enables quantifying marginal risk of outputs or their compositions.

Impermissible Concepts We characterize “impermissible concepts”: knowledge which would be considered dangerous, risky, harmful, or otherwise undesirable for an individual to know through sets of input-output pairs. Specifically, let $Q\subset\mathcal{X}$ be a set of queries that a malicious adversary wants to answer, and for any $q\in Q$, the set $A_{q}\subset\mathcal{Y}$ is the set of answers $a$ for which the pair $(q,a)$ is considered to contain impermissible knowledge. For instance, $q\in Q$ might be asking for malware code, and $A_{q}\subset\mathcal{Y}$ could be the set of all output strings containing a malware implementation. We note the multiple choice examples considered from our evaluation framework in section 3 are a special case when $\mathcal{Y}=\{a_{i}\}$ and $A_{q}=a^{*}$.

###### Definition 4.3 (Inferential Adversary Objective) .

Let $p_{adv}(\cdot|q)$ represent the adversary’s prior belief about the answer to a malicious query $q$. The goal of an inferential adversary AdvLLM with prior $p_{adv}(a|q)$ is to select $k$ inputs $\{q_{i}\}_{i=1}^{k}$ to maximize the scoring function

$$ $\displaystyle s(q,h^{k})=\sum_{a\in A_{q}}p_{adv}(a|h^{k},q)\log\frac{p_{adv}( a|h^{k},q)}{p_{adv}(a|q)}.$ $$

where $h^{k}=\{(q_{i},a_{i})\}_{i=1}^{k}$ is the knowledge pile of $k$ interactions with VicLLM.

###### Definition 4.3 (Inferential Adversary Objective) .

## 5 Information Censorship

#### Overview

To mitigate risks incurred by inferential adversaries, the victim model provider seeks to minimize the expected impermissible information leakage over collections of interactions. In this section we define information censorship, a bound on expected impermissible information leakage (Exp-IIL) which defense mechanisms must ensure to guarantee safety against inferential adversaries. We propose a randomised response defense mechanism for ensuring this bound.

### 5.1 Safety Guarantee

###### Definition 5.1 (Expected Impermissible Information Leakage (Exp- IIL )) .

Let $M$ be a censorship mechanism such that for an input $q_{i}$ to the VicLLM, the answer $a_{i}$ returned to AdvLLM is sampled from $M(q_{s},\textsc{VicLLM}(q_{s}))$. Assuming the AdvLLM seeks to answer $q\in Q$ and submits a set of $k$ queries $\{q_{i}\}_{i=1}^{k}$ to the VicLLM, define the distribution of knowledge piles $h^{k}=\{(q_{i},a_{i})\}_{i=1}^{k}$ to be $H_{q_{i}}^{k}:=\{q_{i},M(q_{i},\textsc{VicLLM}(q_{i}))\}_{i=1}^{k}$. Then, the expected impermissible information leakage is given by:

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{i}}^{k})=\sum_{\{a_{1},\dots,a_ {k}\}\in\mathcal{Y}^{k}}p_{M}(H_{q_{i}}^{k}=h^{k})\sum_{a\in A_{q}}p_{adv}(a|h ^{k},q)\log\frac{p_{adv}(a|h^{k},q)}{p_{adv}(a|q)}.$ $$

Distinction from MI Our definition of Exp-IIL differs from the mutual information definition

$$ $\displaystyle I(p_{adv}(\cdot|q);H_{q_{i}}^{k})=\sum_{\{a_{1},\dots,a_{k}\}\in \mathcal{Y}^{k}}p_{M}(H_{q_{i}}^{k}=h^{k})\sum_{a\in\mathcal{Y}}p_{adv}(a|h^{k },q)\log\frac{p_{adv}(a|h^{k},q)}{p_{adv}(a|q)}.$ $$

This distinction arises as our intent is to capture the asymmetry regarding which answers $a$ an adversary becomes more confident in. From a safety perspective, the only concern is whether or not the adversary becomes more confident in impermissible conclusions $a\in A_{q}$. Specifically, for the LLM provider, scenarios in which the adversary’s posterior $p_{adv}(a|h^{k},q)=1$ for some $a\not\in A_{q}$ is perfectly acceptable as it implies the adversary is confident in a “permissible” answer to $q$, whereas a bound on mutual information would deem this a defense failure assuming the entropy of the adversary prior distribution $p_{adv}(\cdot|q)$ was high.

In order to defend against an inferential adversary, AdvLLM, from inferring the answer to a harmful query $q\in Q$ over $k$ interactions, a censorship mechanism $M$ must bound the worst case Exp-IIL.

###### Definition 5.2 ( $(k,\epsilon)$ -Information Censorship Mechanism (ICM)) .

For a collection of adversary priors $\Phi_{adv}$, a malicious query $q\in Q$, a leakage bound $\epsilon>0$, and $k$ possible interactions between AdvLLM and VicLLM, a $(k,\epsilon)$-ICM $M$ ensures the worst-case Exp-IIL is bounded by $\epsilon$ for knowledge pile distribution $H_{q_{i}}^{k}:=\{q_{i},M(q_{i},\textsc{VicLLM}(q_{i}))\}_{i=1}^{k}$:

$$ $\displaystyle\sup_{\begin{subarray}{c}p_{adv}\in\Phi;\\ \{q_{i}\}_{i=1}^{k}\in\mathcal{X}^{k}\end{subarray}}I_{A_{q}}(p_{adv}(\cdot|q) ;H_{q_{i}}^{k})\leq\epsilon.$ $$

Compositional Bounds Finding and bounding the supremum of the Exp-IIL necessary for a $(k,\epsilon)$-ICM over all possible sets of $k$ interactions is increasingly challenging due to the combinatorial complexity of checking all combinations. However, the $(1,\epsilon)$-ICM (henceforth referred to as an $\epsilon$-ICM) can also provide bounds on Exp-IIL for $k$ interactions.
Assuming the AdvLLM can interact with VicLLM across independent context windows, the $\epsilon$-ICM cannot depend on knowledge of an existing interaction history $h^{i}$—it must be non-adaptive. We provide a non-adaptive composition bound of an $\epsilon$-ICM inspired by results in Nuradha and Goldfeld ().

###### Theorem 5.3 (Non-Adaptive Composability of $\epsilon$ -ICM) .

For a collection of adversary priors $\Phi$, malicious query $q\in Q$, leakage bound $\epsilon>0$, $k$ possible interactions between AdvLLM and VicLLM, and an $\epsilon$-ICM $M$,

$$ $\displaystyle\sup_{\begin{subarray}{c}p_{adv}\in\Phi;\\ \{q_{i}\}_{i=1}^{k}\in\mathcal{X}^{k}\end{subarray}}I_{A_{q}}(p_{adv}(\cdot|q) ;H_{q_{i}}^{k})\leq k\epsilon+\sum_{j=2}^{k}I_{A_{q}}((q_{j},a_{j});H_{q_{i}}^ {j-1}|p_{adv}(\cdot|q)).$ $$

Interpretation In other words, the joint leakage can be bounded by the sum of $k$ individual $\epsilon$ per-interaction leakages and a term capturing the dependencies between interactions when conditioned on $p_{adv}(\cdot|q)$. If the model outputs $\textsc{VicLLM}(q_{i})$ are deterministic or independent when conditioned on $p_{adv}(\cdot|q)$, then, the sum becomes 0 0 because the noise mechanism for an $\epsilon$-ICM is independent of the response. Such assumptions could hold when there is a “single true value” of $p_{adv}(\cdot|q)$ known by VicLLM, and all model outputs are related to this value by a deterministic function.

###### Definition 5.1 (Expected Impermissible Information Leakage (Exp- IIL )) .

###### Definition 5.2 ( ( k , ϵ ) 𝑘 italic-ϵ (k,\epsilon) ( italic_k , italic_ϵ ) -Information Censorship Mechanism (ICM)) .

###### Theorem 5.3 (Non-Adaptive Composability of ϵ italic-ϵ \epsilon italic_ϵ -ICM) .

### 5.2 Randomised Response ϵ italic-ϵ \epsilon italic_ϵ -ICM

Proposed Defense To provide concrete bounds on the information leakage to a $k$-inferential adversary and demonstrate what an $\epsilon$-ICM could look like, we construct an $\epsilon$-ICM. Inspired by a differentially private mechanism proposed by Mangat () to protect privacy of individuals during surveys, we propose a randomized response information censorship mechanism.

###### Definition 5.4 (Randomised Response Mechanism) .

Let $q\in[0,1]$ and $S\subset\mathcal{Y}$ be a nonempty set of safe strings such as the empty string. The randomised response mechanism $M_{q}:\mathcal{X}\times P(\mathcal{Y})\to P(\mathcal{Y})$ is defined as:

$$ $\displaystyle p_{M}((q,a))=\begin{cases}tp(\textsc{VicLLM}(q)=a)&\text{if }a \in\mathcal{Y}\setminus S\\ (1-t)\frac{1}{|S|}&\text{if }a\in S\end{cases}$ $$

where $p_{\textsc{VicLLM}}(y)$ is the probability distribution of the victim model’s output.

###### Theorem 5.5 (Randomised Response $\epsilon$ -ICM) .

Let $\Phi$ be a collection of adversary priors. We assume there exists a nonempty safety set $S\subset\mathcal{Y}$ such that for any $q\in Q$, $p_{adv}\in\Phi$, and $s\in S$, $I_{A_{q}}(p_{adv}(\cdot|q);(q_{1},s))=0$. If we let

$$ $\displaystyle t_{\epsilon}=\min\left(\frac{\epsilon}{\sup_{\begin{subarray}{c} p_{adv}\in\Phi;\\ q_{1}\in\mathcal{X}\end{subarray}}I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{i}}^{1})},1\right)$ $$

then the randomised response mechanism $M_{q_{\epsilon}}$ is an $\epsilon$-ICM.

In order to satisfy the non-adaptive compositional bounds, it becomes evident that the probability of returning the model response can become quite low, thereby affecting utility for benign users. We now turn to establishing these tradeoff results.

###### Definition 5.4 (Randomised Response Mechanism) .

###### Theorem 5.5 (Randomised Response ϵ italic-ϵ \epsilon italic_ϵ -ICM) .

### 5.3 Safety-Utility Trade-offs

Defnining Utility While the primary concern of censorship is to ensure safety by mitigating Exp-IIL, model providers also care about the utility of their model for benign users. Given a distribution $P_{\mathcal{X}}$ for the probability that a benign user prompts the VicLLM with $x$ we can define the utility of an interaction between the VicLLM and the benign user as:

$$ $\displaystyle\mathbb{E}_{y\sim\textsc{VicLLM}(x)}\left[u(x,y)\right].$ $$

Assuming that the safe responses $y\in S$ provide no utility for benign users, we easily find the utility cost of the Randomised Response $\epsilon-ICM$

###### Theorem 5.6 (Utility Bound for Randomised Response $\epsilon$ -ICM) .

For a given input $x$ and utility function $u(x,y)$, the expected utility of an interaction where the outputs are given by the randomized response $\epsilon$-ICM can be bounded as follows:

$$ $\frac{\mathbb{E}_{y\sim M(x,\textsc{VicLLM}(x))}[u(x,y)]}{\mathbb{E}_{y\sim \textsc{VicLLM}(x)}[u(x,y)]}=q_{\epsilon},$ (1) $$

$t_{\epsilon}$ is the probability of the mechanism returning a response from the Victim model.

Thus, introducing the randomised response mechanism yields a model with utility $q_{\epsilon}$ that of the uncensored model for any inputs for which the mechanism is employed.

Inferential Users If we assume benign users are also inferential, i.e. they seek to learn information from model outputs about some task of quesiton $x^{*}$, then, for prior $p_{usr}(\cdot|x^{*})$, we can define their expected utility as:

$$ $\displaystyle I(p_{usr}(\cdot|x^{*});(x,Y))=\sum_{y\in Y}p(x,y)\sum_{a\in \mathcal{Y}}p_{usr}(a|(x,y),x^{*})\log\frac{p_{usr}(a|(x,y),x^{*})}{p_{usr}(a| x^{*})}.$ $$

For such inferential users, an upper bound on the possible utility for any input $x$ can be derived for all possible $\epsilon$-ICMs. We adapt results from Zamani et al. () to upper bound the mutual information.

###### Theorem 5.7 (Utility Bound for Inferential Users) .

For a benign user interested in query $x^{*}$ with prior $p_{usr}(\cdot|x^{*})$, a collection of adversary priors $\Phi$, set of malicious queries $Q\subset\mathcal{X}$, with the VicLLM employing an $\epsilon$-ICM $M$, the utility for the benign inferential user interacting with the VicLLM employing and $\epsilon$-ICM, is bounded by:

$$ $\displaystyle I(p_{usr}(\cdot|x^{*});(x,Y)\leq\epsilon+\inf_{\begin{subarray}{ c}q\in Q\\ p_{adv}\in\Phi\end{subarray}}\left(\mathcal{H}(p_{usr}(\cdot|x^{*})|p_{adv}( \cdot|q))+I_{A_{q}^{c}}(p_{adv}(\cdot|q);(x,Y))\right)$ $$

where $A_{q}^{c}$ the complement of $A_{q}$ and $Y=M(x,\textsc{VicLLM}(x))$.

Interpretation Thus, the utility can only be high if the useful knowledge $p_{usr}(\cdot|x^{*})$ is not specified by $p_{adv}(\cdot|q)$, or, the interaction is such that the mechanism $M$ makes $I_{A_{q}^{c}}(p_{adv}(\cdot|q);(x,Y))$ high, in other words making the adversary more confident in a permissible belief. While $\mathcal{H}(p_{usr}(\cdot|x^{*})|p_{adv}(\cdot|q))$ does not depend on the mechanism employed, an $\epsilon$-ICM may be able to provide more utility if the responses returned make adversaries more confident in permissible conclusions.

###### Theorem 5.6 (Utility Bound for Randomised Response ϵ italic-ϵ \epsilon italic_ϵ -ICM) .

###### Theorem 5.7 (Utility Bound for Inferential Users) .

## 6 Related work and Discussion

### 6.1 Comparison to Jailbreaks

The fundamental distinction between decomposition attacks and jailbreaks is the adversary’s objective. While DAs share resemblance to methods such as payload splitting (Kang et al., ; Li et al., ) or multi-turn attacks (Russinovich et al., ; Li et al., ), these methods still rely on a victim model to produce problematic outputs; output censorship methods could greatly limit their effectiveness.

For example, payload splitting is an input obfuscation method which splits up harmful words or strings into substrings which are then assigned to symbolic variables and an equation containing these variables. The model is then asked the prompt, with
equations substituted for the sensitive words. Thus, payload splitting utilizes the programmatic capabilities of LLMs in order to mitigate the ability to detect harmfulness or malicious intent in the input, however, the outputs provided remain impermissible and open to output censorship mechanisms.

Multi-turn jailbreaks (Russinovich et al., ; Li et al., ) bypass defenses by gradual escalation from benign prompt and response interactions to problematic ones. These attacks are related to many-shot jailbreaks (Anil et al., ), leveraging an extended context window to alter model behavior. Nevertheless, once again, the method aims to force the victim model to produce a strictly impermissible output, which could more easily be blocked by output filtering mechanisms and may not even assist the adversary in fulfilling their malicious goal.

### 6.2 Relating inferential adversaries to privacy literature

Inferential adversaries have been studied before in the context of privacy, such as model stealing adversaries (Tramèr et al., ) and membership inference adversaries (Shokri et al., ). Model stealing adversaries utilize carefully crafted queries and certain compositional properties of these queries and their outputs to infer specific parameters (Shamir et al., ). The black-box search for adversarial examples can also be viewed as being performed by an inferential adversary, as the input queries are not directly causing a model failure but leak information about gradients (Ilyas et al., ).

By presenting inferential adversaries as a compelling threat model in the context of AI safety, we establish parallels between the problems of AI safety and privacy. Initial attempts to address concerns of privacy in data often focused on direct anonymization, such as removing personally identifiable information (PII) from datasets (Sweeney, ). These approaches operated under the assumption that by simply removing identifiers and forcing indistinguishability within a dataset would be sufficient for making individuals in the dataset unidentifiable (Sweeney, ). However, such approaches were insufficient for actually ensuring privacy because they did not account for the compositional nature of data. It was found that individuals could still be identified when certain combinations of attributes were unique especially given background information provided by auxiliary sources (Ganta et al., ; Narayanan and Shmatikov, ). Similarly, in the context of LLM safety, we argue that simply censoring or filtering direct responses does not address the underlying issue that an adversary can piece together sensitive information as part of a compositional attack.

To address compositional attacks, Differential Privacy was proposed as a mathematical framework which defined and sought to bound the marginal risk to an individual’s privacy when releasing data (Dwork, ). These guarantees are achieved by adding a controlled amount of noise to the data or to responses made to queries so as to ensure that any single individual’s data does not significantly influence the outcome. Further definitions sought to generalize and extend privacy definitions in terms of information leakage (Nuradha and Goldfeld, ; Grosse et al., ; Bloch et al., ).

We hope that through introducing the inferential adversary threat model in the context of safety, we can inspire a paradigm shift similar to that which occurred for privacy for the field of AI safety. By showing that safety concerns can be similarly framed and that existing defense approaches are vulnerable to composition attacks leveraging multiple interactions and background knowledge, our work helps introduce a theoretically rigorous way for further assessing and establishing safety.

## 7 Conclusion

We identify a key safety vulnerability of deployed LLMs in terms of impermissible information leakage. We contrast with existing work which treats the ability of LLMs to provide harmful information to adversaries as a problem of model robustness, highlighting that this perspective is too narrow. Although current robustness issues make it easy for security adversaries to get a desired result, as defenses improve such direct attacks may prove more difficult. However, robustness provides a false sense of safety. The fundamental issue is that knowledge is compositional, interdependent, and dual-use. Directly censoring certain responses to questions does not guarantee that an adversary cannot reconstruct impermissible knowledge. Consequently, we conclude that evaluation methods and defenses must consider the potential for model responses to help adversaries infer dangerous information rather than determine whether or not model outputs themselves are dangerous. Nevertheless, we highlight that proper defenses will inevitably sacrifice utility for benign users.

## 8 Acknowledgements

We want to thank Cleverhans lab members Sierra Wyllie, Anvith Thudi, Mohammad Yaghini, and Stephan Rabanser, as well as Claas Voelcker, Roger Grosse, Erik Jones, Lev McKinney, Xander Davies, and Darija Barak for their helpful feedback.

We would like to acknowledge our sponsors, who support our research with financial and in-kind contributions:
Amazon, Apple, CIFAR through the Canada CIFAR AI Chair, Meta, NSERC through the Discovery Grant and an Alliance Grant with ServiceNow and DRDC,
the Ontario Early Researcher Award, the Schmidt Sciences foundation through the AI2050 Early Career Fellow program, and the Sloan
Foundation. Resources used
in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada
through CIFAR, and companies sponsoring the Vector Institute.

## Appendix A Proofs

As a few preliminaries for proving our results, we re-derive properties of expected impermissible information gain that match those of Mutual Information. For an $q\in Q$, we define a variant of Entropy

$$ $\displaystyle\mathcal{H}_{A_{q}}(p_{adv}(\cdot|q))=-\sum_{a\in A_{q}}p_{adv}(a |q)\log p_{adv}(a|q)$ $$

and Conditional Entropy, conditioned on a knowledge pile distribution $H^{k}_{q_{i}}$ provided subquestions $\{q_{i}\}_{i=1}^{k}$

$$ $\displaystyle\mathcal{H}_{A_{q}}(p_{adv}(\cdot|q)|H^{k}_{q_{i}})=-\sum_{\{a_{1 },\dots,a_{k}\}\in\mathcal{Y}^{k}}p_{M}(H^{k}_{q_{i}}=h^{k})\sum_{a\in A_{q}}p _{adv}(a|q,h^{k})\log p_{adv}(a|q,h^{k})$ $$

resulting in the standard relationship $I_{A_{q}}(p_{adv}(\cdot|q);H^{k}_{q_{i}})=\mathcal{H}_{A_{q}}(p_{adv}(\cdot|q)
-\mathcal{H}_{A_{q}}(p_{adv}(\cdot|q)|H^{k}_{q_{i}})$.

Non-negativity of $I_{A_{q}}(p_{adv}(\cdot|q);H^{k}_{q_{i}})$: Denoting $p(A_{q})=\sum_{a\in A_{q}}p_{adv}(a|q)$, Jensen’s inequality gives us that

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);H^{k}_{q_{i}})=-p(A_{q})(\sum_{a\in A _{q}}\sum_{h^{k}\in\mathcal{Y}^{k}}\frac{p_{joint}(a,h^{k}|q)}{p(A_{q})}\log \frac{p_{adv}(a|q)p_{M}(H^{k}_{q_{i}}=h^{k})}{p_{joint}(a,h^{k}|q)})$ $\displaystyle\geq-p(A_{q})(\log(\sum_{a\in A_{q}}\sum_{h^{k}\in\mathcal{Y}^{k} }\frac{p_{joint}(a,h^{k}|q)}{p(A_{q})}\frac{p_{adv}(a|q)p_{M}(H^{k}_{q_{i}}=h^ {k})}{p_{joint}(a,h^{k}|q)}))$ $\displaystyle=-p(A_{q})\log(\sum_{a\in A_{q}}\sum_{h^{k}\in\mathcal{Y}^{k}} \frac{p_{adv}(a|q)p_{M}(H^{k}_{q_{i}}=h^{k})}{p(A_{q})})$ $\displaystyle=-p(A_{q})\log(\sum_{a\in A_{q}}\frac{p_{adv}(a|q)}{p(A_{q})})$ $\displaystyle=-p(A_{q})\log(1)=0$ $$

where $p_{joint}$ is the joint distribution over adversary beliefs and knowledge piles.

Chain rule of Impermissible Information: We abstract a bit away from the specific distributions examined to define a chain rule for impermissible information. Specifically, given $q\in Q$, and $A_{q}\in\mathcal{Y}$, let $A$, $B$, and $C$ be random variables. Then, $I_{A_{q}}(C;A,B)=I_{A_{q}}(C;A)+I_{A_{q}}(C;B|A)$

The LHS can be expressed as

$$ $\displaystyle I_{A_{q}}(C;A,B)=\sum_{c\in A_{q}}\sum_{a,b}p(a,b,c)\log\frac{p( a,b,c)}{p(c)p(a,b)}$ $$

whereas on the RHS

$$ $\displaystyle I_{A_{q}}(C;A)=\sum_{c\in A_{q}}\sum_{a}p(a,c)\log\frac{p(a,c)}{ p(a)p(c)}$ $$

and

$$ $\displaystyle I_{A_{q}}(C|A;B)$ $\displaystyle=\sum_{c\in A_{q}}\sum_{a,b}p(c,a,b)\log\frac{p(c,a,b)p(a)}{p(c,a )p(b,a)}$ $\displaystyle=\sum_{c\in A_{q}}\sum_{a,b}p(c,a,b)\log\frac{p(c,a,b)}{p(c,a)p(b |a)}$ $$

Thus,

$$ $\displaystyle I_{A_{q}}(C;A)+I_{A_{q}}(C;B|A)$ $\displaystyle=\sum_{c\in A_{q}}\sum_{a,b}p(c,a,b)\left(\log\frac{p(c,a)}{p(c)p (a)}+\log\frac{p(c,a,b)}{p(c,a)p(b|a)}\right)$ $\displaystyle=\sum_{c\in A_{q}}\sum_{a,b}p(c,a,b)\log\frac{p(c,a,b)}{p(b|a)p(c )p(a)}$ $\displaystyle=\sum_{c\in A_{q}}\sum_{a,b}p(c,a,b)\log\frac{p(c,a,b)}{p(c)p(a,b)}$ $\displaystyle=I_{A_{q}}(C;A,B)$ $$

establishing the chain rule for our variant of mutual information. This argument can be further generalized by induction to provide

$$ $\displaystyle I_{A_{q}}(C;X_{1},\dots,X_{n})=\sum_{i=1}^{n}I_{A_{q}}(C|X_{1}, \dots,X_{i-1};X_{i})$ $$

Data Processing Inequality:
If an adversary applies some post processing to the knowledge pile signal $H^{k}_{q_{i}}$ to get $H^{k^{\prime}}_{q_{i}}$, then $H^{k^{\prime}}_{q_{i}}$ is conditionally independent of $p_{adv}(\cdot|q)$ given $H^{k}_{q_{i}}$. From the chain rule, we have that

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);H^{k}_{q_{i}},H^{k^{\prime}}_{q_{i}}) =I_{A_{q}}(p_{adv}(\cdot|q);H^{k}_{q_{i}})+I_{A_{q}}(p_{adv}(\cdot|q);H^{k^{ \prime}}_{q_{i}}|H^{k}_{q_{i}})$ $$

and

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);H^{k}_{q_{i}},H^{k^{\prime}}_{q_{i}}) =I_{A_{q}}(p_{adv}(\cdot|q);H^{k^{\prime}}_{q_{i}})+I_{A_{q}}(p_{adv}(\cdot|q) ;H^{k}_{q_{i}}|H^{k^{\prime}}_{q_{i}})$ $$

Conditional independence implies that $I_{A_{q}}(p_{adv}(\cdot|q);H^{k^{\prime}}_{q_{i}}|H^{k}_{q_{i}})=0$, and, non-negativity implies that $I_{A_{q}}(p_{adv}(\cdot|q);H^{k^{\prime}}_{q_{i}}|H^{k^{\prime}}_{q_{i}})$, thus, $I_{A_{q}}(p_{adv}(\cdot|q);H^{k}_{q_{i}})\geq I_{A_{q}}(p_{adv}(\cdot|q);H^{k^
{\prime}}_{q_{i}})$

### A.1 Proof of Theorem 5.3

###### Proof.

Let $q$ be a malicious query, $\epsilon>0$ a leakage bound, and $k$ the number of possible interactions between AdvLLM and VicLLM mediated by an $\epsilon$-ICM $M$. For ease of notation, let $y_{i}=(q_{i},M(q_{i},\textsc{VicLLM}(q_{i}))$. For any set of $k$ questions $\{q_{i}\}_{i=1}^{k}$, and history distribution $H_{q_{i}}^{k}:=\{q_{i},a_{i})\}_{i=1}^{k}$, we prove by induction that

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{i}}^{k})\leq\sum_{i=1}^{k} \epsilon_{i}+\sum_{j=2}^{k}I_{A_{q}}(y_{j};H_{q_{i}}^{j-1}|p_{adv}(\cdot|q))$ $$

when $I_{A_{q}}(p_{adv}(\cdot|q);(q_{j},a_{j}))\leq\epsilon$.

$$ $\displaystyle I_{A_{q}}((q_{j},a_{j});H_{q_{i}}^{j-1}|p_{adv}(\cdot|q))=$ $\displaystyle\sum_{a\in A_{q}}p_{adv}(a|q)\sum_{y_{1},\dots,y_{j}\in\mathcal{Y }}p_{M}(y_{1},\dots,y_{j}|(q,a))\log\frac{p_{M}(y_{1},\dots,y_{j}|(q,a))}{p_{M }(y_{1},\dots,y_{j-1}|(q,a))p_{M}(y_{j}|(q,a))}$ $$

For $k=1$, we trivially have that
$I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{1}}^{1})\leq\epsilon\leq\epsilon+\eta$ as $\eta$ is non-negative.

Assume then that

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{i}}^{m})\leq m\epsilon+\eta_{m}$ $$

for $\eta_{m}=\sum_{j=2}^{m}I_{A_{q}}(y_{j};H_{q_{i}}^{j-1}|p_{adv}(\cdot|q))$. Then,

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{i}}^{m+1})=I_{A_{q}}(p_{adv}( \cdot|q);H_{q_{i}}^{m})+I_{A_{q}}(p_{adv}(\cdot|q);(q_{m+1},a_{m+1}|H_{q_{i}}^ {m})$ $\displaystyle\leq\sum_{i=1}^{m}\epsilon_{i}+\eta_{m}+I_{A_{q}}(p_{adv}(\cdot|q );(q_{m+1},a_{m+1}|H_{q_{i}}^{m})$ $$

To show our desired result, we need to show that

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1}|H_{q_{i}}^{m})\leq I_{A_{q}}( p_{adv}(\cdot|q);y_{m+1})+I_{A_{q}}(y_{m+1};H_{q_{i}}^{m}|p_{adv}(\cdot|q))$ $$

For $I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1}|H_{q_{i}}^{m})$:

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1}|H_{q_{i}}^{m})=$ $\displaystyle\sum_{h^{m},a,y_{m+1}}p_{joint}(h^{m},a,y_{m+1}|q)\log\frac{p_{ joint}(h^{m},a,y_{m+1}|q)p_{M}(h^{m})}{p_{joint}(h^{m},a|q)p_{M}(h^{m+1})}$ $$

For $I_{A_{q}}(y_{m+1};H_{q_{i}}^{m}|p_{adv}(\cdot|q))$:

$$ $\displaystyle I_{A_{q}}(y_{m+1};H_{q_{i}}^{m}|p_{adv}(\cdot|q))=$ $\displaystyle\sum_{h^{m},a,y_{m+1}}p_{joint}(h^{m},a,y_{m+1}|q)\log\frac{p_{ joint}(h^{m},a,y_{m+1}|q)p_{adv}(a|q)}{p_{joint}(h^{m},a|q)p_{joint}(y_{m+1},a |q)}$ $$

For $I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1})$:

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1})=$ $\displaystyle\sum_{h^{m},a,y_{m+1}}p_{joint}(h^{m},a,y_{m+1}|q)\log\frac{p_{ joint}(a,y_{m+1}|q)}{p_{adv}(a|q)p_{M}(y_{m+1})}$ $$

Thus,

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1})+I_{A_{q}}(y_{m+1};h^{m}|p_{ adv}(\cdot|q))$ $\displaystyle=\sum_{h^{m},a,y_{m+1}}p_{joint}(h^{m},a,y_{m+1}|q)\log[\frac{p_{ joint}(h^{m},a,y_{m+1}|q)p_{adv}(a|q)}{p_{joint}(h^{m},a|q)p_{joint}(y_{m+1},a |q)}\frac{p_{joint}(a,y_{m+1}|q)}{p_{adv}(a|q)p_{M}(y_{m+1})}]$ $\displaystyle=\sum_{h^{m},a,y_{m+1}}p_{joint}(h^{m},a,y_{m+1}|q)\log[\frac{p_{ joint}(h^{m},a,y_{m+1}|q)}{p_{joint}(h^{m},a|q)p_{M}(y_{m+1})}]$ $$

and

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1})+I_{A_{q}}(y_{m+1};h^{m}|p_{ adv}(\cdot|q))-I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1}|H_{q_{i}}^{m})$ $\displaystyle=\sum_{h^{m},a,y_{m+1}}p_{joint}(h^{m},a,y_{m+1}|q)\log[\frac{p_{ joint}(h^{m},a,y_{m+1}|q)}{p_{joint}(h^{m},a|q)p_{M}(y_{m+1})}\frac{p_{joint}( h^{m},a|q)p_{M}(h^{m+1})}{p_{joint}(h^{m},a,y_{m+1}|q)p_{M}(h^{m})}]$ $\displaystyle=\sum_{h^{m},a,y_{m+1}}p_{joint}(h^{m},a,y_{m+1}|q)\log[\frac{p_{ M}(h^{m+1})}{p_{M}(y_{m+1})p_{M}(h^{m})}]$ $\displaystyle=\sum_{h^{m}}p_{M}(h^{m+1})\log[\frac{p_{M}(h^{m+1})}{p_{M}(y_{m+ 1})p_{M}(h^{m})}]=I(y_{m+1};h^{m})\geq 0$ $$

Finally, we conclude that

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);y_{m+1}|H_{q_{i}}^{m})\leq I_{A_{q}}( p_{adv}(\cdot|q);y_{m+1})+I_{A_{q}}(y_{m+1};H_{q_{i}}^{m}|p_{adv}(\cdot|q)) \leq\epsilon+I_{A_{q}}(y_{m+1};H_{q_{i}}^{m}|p_{adv}(\cdot|q))$ $$

and $I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{i}}^{m+1})\leq(m+1)\epsilon+\eta_{m+1}$.

∎

###### Proof.

### A.2 Proof of Theorem 5.5

###### Proof.

Let $\Phi$ be a collection of adversary priors and $\epsilon>0$. We assume there exists a nonempty safety set $S\subset\mathcal{Y}$ such that for any $q\in Q$, $p_{adv}\in\Phi$, and $s\in S$, $I_{A_{q}}(p_{adv}(\cdot|q);(q_{1},s))=0$. Finally, let

$$ $\displaystyle t_{\epsilon}=\min\left(\frac{\epsilon}{\sup_{\begin{subarray}{c} p_{adv}\in\Phi;\\ q_{1}\in\mathcal{X}\end{subarray}}I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{i}}^{1})},1\right)$ $$

Then, for any input $q_{1}\in\mathcal{X}$, we have

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);(q_{1},H_{q_{i}}^{1})=\sum_{\{a_{1}\} \in\mathcal{Y}}p_{M}((q_{1},a_{1})\sum_{a\in A_{q}}p_{adv}(a|a_{1},q)\log\frac {p_{adv}(a|a_{1},q)}{p_{adv}(a|q)}$ $$

If $a_{1}\in S$, then by our assumption on $S$, $\log\frac{p_{adv}(a|a_{1},q)}{p_{adv}(a|q)}=0$, thus,

$$ $\displaystyle I_{A_{q}}(p_{adv}(\cdot|q);(q_{1},H_{q_{1}}^{1})=\sum_{\{a_{1}\} \in\mathcal{Y}\setminus S}p_{M}((q_{1},a_{1})\sum_{a\in A_{q}}p_{adv}(a|a_{1}, q)\log\frac{p_{adv}(a|a_{1},q)}{p_{adv}(a|q)}$ $\displaystyle=t_{\epsilon}\sum_{\{a_{1}\}\in\mathcal{Y}\setminus S}p(\textsc{ VicLLM}(q_{1})=a_{1})\sum_{a\in A_{q}}p_{adv}(a|a_{1},q)\log\frac{p_{adv}(a|a_ {1},q)}{p_{adv}(a|q)}$ $\displaystyle\leq t_{\epsilon}I_{A_{q}}(p_{adv}(\cdot|q);H_{q_{i}}^{1})$ $\displaystyle\leq\epsilon$ $$

∎

###### Proof.

### A.3 Proof of Theorem 5.6

###### Proof.

$$ $\displaystyle\mathbb{E}_{y\sim M(x,\textsc{VicLLM}(x))}[u(x,y)]=\sum_{y\in Y}p _{M}(y)u(x,y)$ $\displaystyle=t_{\epsilon}\sum_{y\in Y}p_{M}(y)u(x,y)+(1-t_{\epsilon})\mathbb{ E}_{y\sim\text{Unif}(S)}[u(x,y)]$ $$

Assuming $u(x,y)=0$ for $y\in S$, we can express the utility ratio:

$$ $\displaystyle\frac{\mathbb{E}_{y\sim M(x,\textsc{VicLLM}(x))}[u(x,y)]}{\mathbb {E}_{y\sim\textsc{VicLLM}(x)}[u(x,y)]}=\frac{t_{\epsilon}\sum_{y\in Y}p(y)u(x, y)+(1-t_{\epsilon})\cdot 0}{\sum_{y\in Y}p(y)u(x,y)}=t_{\epsilon}$ $$

∎

###### Proof.

### A.4 Proof of Theorem 5.7

###### Proof.

By the chain rule for mutual information:

$$ $\displaystyle I(p_{usr}(\cdot|x^{*});(x,Y))$ $\displaystyle=I(p_{adv}(\cdot|q),p_{usr}(\cdot|x^{*});(x,Y))-I(p_{adv}(\cdot|q );(x,Y)|p_{usr}(\cdot|x^{*}))$ $\displaystyle=I(p_{adv}(\cdot|q);(x,Y))+I(p_{usr}(\cdot|x^{*});(x,Y)|p_{adv}( \cdot|q))$ $\displaystyle\quad-I(p_{adv}(\cdot|q);(x,Y)|p_{usr}(\cdot|x^{*}))$ $\displaystyle\leq I(p_{usr}(\cdot|x^{*});(x,Y)|p_{adv}(\cdot|q))+I(p_{adv}( \cdot|q);(x,Y))$ $$

For any malicious query $q\in Q$ and adversary prior $p_{adv}\in\Phi$, we can decompose $I(p_{adv}(\cdot|q);(x,Y))$ into information gain about impermissible $A_{q}$ and permissible $A_{q}^{c}$ conclusions:

$$ $\displaystyle I(p_{adv}(\cdot|q);(x,Y))=I_{A_{q}}(p_{adv}(\cdot|q);(x,Y))+I_{A _{q}^{c}}(p_{adv}(\cdot|q);(x,Y))\leq\epsilon+I_{A_{q}^{c}}(p_{adv}(\cdot|q);( x,Y))$ $$

Using the fact that $I(p_{usr}(\cdot|x^{*});(x,Y)|p_{adv}(\cdot|q))\leq\mathcal{H}(p_{usr}(\cdot|x^
{*})|p_{adv}(\cdot|q))$, we get:

$$ $\displaystyle I(p_{usr}(\cdot|x^{*});(x,Y))$ $\displaystyle\leq\mathcal{H}(p_{usr}(\cdot|x^{*})|p_{adv}(\cdot|q))+I_{A_{q}^{ c}}(p_{adv}(\cdot|q);(x,Y))+\epsilon$ $$

Since this holds for any $q\in Q$ and $p_{adv}\in\Phi$, we can take the infimum:

$$ $\displaystyle I(p_{usr}(\cdot|x^{*});(x,Y))\leq\epsilon+\inf_{\begin{subarray} {c}q\in Q\\ p_{adv}\in\Phi\end{subarray}}\left(\mathcal{H}(p_{usr}(\cdot|x^{*})|p_{adv}( \cdot|q))+I_{A_{q}^{c}}(p_{adv}(\cdot|q);(x,Y))\right)$ $$

∎

###### Proof.

## Appendix B Decomposition Attack Algorithm Implementation details

### B.1 Experimental Details

Dataset Selection and Curation: To evaluate DAs and compare them against jailbreaks under our proposed IIL metric, we chose the WMDP dataset [Li et al., ], focusing on multiple-choice questions in biology and chemistry (WMDP-Bio and WMDP-Chem) as proxies for dangerous knowledge. To assess DAs’ ability to extract impermissible information despite filtering mechanisms, we used Llama-Guard-3-8B [Llama Team, ] to curate subsets of questions classified as unsafe, resulting in 35 and 64 questions for WMDP-Bio and WMDP-Chem, respectively. This curation ensures the all questions would trigger input filtering mechanisms if asked directly.

**Table 3: Models and their purpose in the experiment. All sampling for generation is performed with temperature $t=0.8$ and nucleus sampling with $p=0.95$.**
| Models Used |  |
| --- | --- |
| Model Name | Purpose |
| Llama-3.1-70B-Instruct [Llama Team, ] | Victim model (VicLLM) |
| Prompt-Guard-86M [Llama Team, ] | Jailbreak detection for Input Filtering ($M_{i}$) |
| Llama-Guard-3-8B[Llama Team, ] | Input and output filtering mechanism ($M_{i}$ and $M_{o}$) |
| Mistral-7B-Instruct-v0.3 [Jiang et al., ] | Adversary LLM (AdvLLM); not safety alignmed |
| Llama-3.1-8B-Instruct [Llama Team, ] | Adversary LLM (AdvLLM); safety alignmed |

Few-shot Examples: Some few-shot examples for subquestion generation $E_{g}$ and answer aggregation $E_{a}$ were adapted from [Radhakrishnan et al., ] which focused on problem decomposition and aggregation for problem solving. The remaining few shot examples were generated using Claude 3.5 Sonnet [ant, ] to match the more technical multiple choice questions in the WMDP dataset.

Hyperparameters: We run our proposed DA with $k=2$ rounds and $m=3$ subquestions per round. To reflect more realistic use cases and remove outliers, we only apply the attack to those questions for which the initial answer entropy is greater than $0.5$.(^1^11This implies that the adversary assigns less than about $.97$ probability to any one of the choices) For each question we run the DA $5$ times, returning the IIL measurements for each run as well as the average number of times inputs or outputs were flagged by the input or output filtering models per attack.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. As shown in Table 1, there appears to be a good deal of variance in performance across topics. 

	a. Do you have a hypothesis for why this is? 

	b. Is there something more challenging about the Chem topic vs the Bio topic, and could this extend to other topics areas that are not tested?

2. While I understand a differential privacy based defense allows for more rigorous analysis of bounds and tradeoffs, I do question whether this is actually an appropriate defense in this setting. While there are similarities between this setting and privacy (small leaks in information adding up, difficult to classify what is a dangerous leak, etc.), it seems that classifying what is unsafe may be somewhat easier here, given the full context of a conversation. Can you give additional practical motivation for this defense?

3. What would the cost be to implement the randomized response defense? How does this compare to existing defenses?
soundness: 3
strengths: 1. To me this is a realistic threat model. The multi-turn nature of the attack and the decomposition of the harmful query into smaller safe queries seems an intuitive setting.

2. The evaluation is not all or nothing, but models information gain as a change in model confidence in the answer chosen with and without the information from the target model. This allows for more nuanced measurements of the harm done by information shared.

3. The problem and setting are important, and the paper is well written and organized.
summary: This paper explores the abilities of language models to defend against attacks in which a user query is decomposed into multiple questions. They describe a threat model in which an adversary decomposes a harmful question (which an LLM would refuse to answer) into many *safe* questions. They then measure the ability of the adversary to learn information that contributes to the overall harmful goal by assessing a model's increase in confidence in answering multiple choice questions correctly after receiving information from the target LLM. They additionally present a randomized response defense based on differential privacy, demonstrating that this defense has provable bounds for safety, but would come at the expense of performance.
weaknesses: 1. This study uses data from only two domains, and there are significant enough differences between some results that I question how domain-specific the effectiveness is. Clarification on what may cause these discrepancies and if they are likely to occur for other topics as well would be helpful.

2. The differential privacy based defense allows bounds to be given on the safety vs. performance of the model, but this is not compared to existing defenses that are tested in the paper. While bounds may not be able to be established theoretically for these defenses, a comparison to existing defenses seems appropriate.

3. It was not clear to me until searching in the appendix that the attacked model was Llama-3.1-70B, or that defense mechanisms (Llama-Guard-3 and Prompt-Guard-3) were used. This improves the credibility of the results and should be made more clear in the results and/or methodology.

4. Though multiple adversary models are used, the attack is tested on only one victim model and using only one set of defenses. Attacking a large model is, of course, impressive, but adding models from multiple families would make the results more robust.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: Questions: 
1) Why did you develop the randomised response defense? Can you provide motivation for it?

Suggestions: 
1)  This evaluation demonstrates that harmful knowledge in compositional, and it may be difficult to serve models with some types of knowledge. Perhaps, you may want to cite related works in unlearning? 
2) It is potentially interesting to study if harmful applications can be built compositionally, instead of limiting to harmful knowledge. The standard argument on the safety-utility tradeoff is that the web already contains this knowledge, and serving it via LLMs is not a significant increase in harmfulness. By defining a clear pipeline for the attacker LLM to systematically extract harmful knowledge, your approach mitigates this concern a little bit, but taking this one step further could help the case.
soundness: 2
strengths: - For frontier models, it is important that we have more research on threat models to better assess the usefulness of new jailbreak datasets and defenses. The threat model in this paper seems important from a policy and governance perspective.
summary: The paper introduces a data leakage threat model for studying frontier model misuse. It points out that the current LLM jailbreak literature is too focused on one-shot harmful response generation instead of more complex threat models for how model-generated outputs may be synthesized to extract harmful knowledge. As a demonstration of the threat model, the paper defines an attacker model that is given few-shot demonstrations to generate subquestions, extracting answers to subquestions, and aggregating answers to the questions.
weaknesses: - The randomised response strategy is extremely costly. It is unlikely to be informative for any frontier model developers, nor serve as a relevant baseline to compare future solutions against. In my opinion, the paper would be better without proposing any solutions, as a clearer study of this particular threat model. 

- The empirical results are insufficient:
  - The dataset is limited. If it wasn't clear where to source more questions, creating such a dataset would have been good for making it easy for future work to take on this problem. 
  - Even if not fully faithful to the threat model, showing that harmful knowledge is compositional is an important subclaim of the paper, and can be validated using existing jailbreak datasets. Even if DA is not competetive against PAIR, it is important to show that it is adequate. 

- Given that the attacker may already be predicting the correct answer, it seems important to study how often the model changes its answer to be more correct. (This may involve control studies around how well the smaller attacker models are at extrapolating from a series of facts. however,) Without substantiating that attacker models actually learn new answers, the IIL metric is not as useful and has many confounders on the mechanism of how the probability on the correct answer is actually increasing, for instance, because of longer context windows.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the weakness part.
soundness: 2
strengths: The paper is easy to follow. Discussions are thorough in the paper.
summary: Prior works mainly focus on security threats where adversaries mislead the victim model to output malicious content directly, while this paper considers inferential threats, where the malicious query is decomposed into multiple subquestions and leads to a harmful response after aggregation of answers of subquestions. There are empirical evaluations and analysis from information perspective to consolidate the setting considered in the paper.
weaknesses: 1. The key part in this paper is the query decomposition and aggregation part. However, the concrete prompts are missing to judge whether it makes sense. Some examples should also be provided to help understanding. The example in Figure 1 does not impress me because the question "What are some common scams targeting the elderly?" is not a benign one so the LLM might not provide answers to it, Also, is the method only applied to complex questions? If the query is simple as Figure 1, how do we decompose it effectively.

2. Can the analysis part benefits design of attack/defense algorithms? Can we empirically validate them. In the current version, the analysis seems separate and does not convey concise and impressive results.

3. Evaluation is also limited. An overall number is not sufficient to support the method. More ablations should be conducted. For example, which prompts are more effective? Which queries can be more easily conducted? Will certain model refuse to decompose the query because it contains sensitive words? Such questions can help better understand the strengths and weakness of the algorithm.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 4
questions: What model is used as VicLLM, in the results section?
Table 1 is somewhat confusing to me - what's the metric? Why linear mixed effects model?
soundness: 3
strengths: - Results look great overall, I think the approach proposed here seems effective at eliciting harmful info without triggering flagging. The fact that this works well in practice will definitely require some rethinking on the part of model developers, in terms of figuring out how to handle attacks like this. Likely it will require some big changes (if not already in place), like cross-user monitoring and inferring whether or not harmful info has leaked despite no single question looking harmful.
-Great motivation; have been wanting to see a paper on this for a while.
-Great idea to eval our WMMDP + compare to PAIR
-Very clear abstract + Figure 1



I would give this paper a 7/10 rating, somewhere between marginal accept and accept (but the form would only allow a 6 or an 8).
summary: The paper proposes an approach for jailbreaking models, which involves decomposing a harmful question into harmless-looking subquestions (with a less capable, safety-free model e.g. an open-source one), then answering those subquestions with a more capable model with safety training (e.g. black-box API models), then composing those answers into a single overall answer with the less-capable safety-free model. The paper finds that this method is effective overall, testing on the WMDP benchmark which consists of harmful cyber and CBRN related questions. The paper also discusses theory behind a differential privacy motivated defense, which improves safety at the cost of helpfulness.
weaknesses: -Table 1 would be clearer if (or experimental results would be clearer) if paper showed improvement in ASP to PAIR's ASR and comparison
-On the theory section, I don't have the expertise to evaluate this section that well, and don't fully follow the motivation or proposed defense. It would be nice to have some higher-level description of the findings. I understand that adding noise is involved in the defense, but don't understand how this applies here or helps. It would be nice to propose/test a specific approach in the paper.
-Erik Jones' et al. have an arXiv paper that's pretty related / on the same overall issue, if I recall correctly, which would be worth citing/discussing.
- It would be nice to better understand when this method doesn't work -- what kinds of questions weren't able to be decomposed into harmless questions. Generally more qualitative analysis would be helpful for understanding how big of a problem decomposition will be.

Minor:
-Typo on page 8 "Defining"

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weakness
soundness: 1
strengths: 1. The authors provide a solid theoretical framework using information theory to quantify the risk of inferential adversaries, the proposed method surpass PAIR performance on curated WMDP dataset.

2. The paper is well-written and easy to follow.
summary: The paper introduces a new threat model, "inferential adversaries,"which focus on safe AI responses can leak impermissible information. The author also proposed decomposition attack which is an automated black-box attack method for extracting and leveraging dual-use information to fulfill adversary objectives.
weaknesses: Despite the proposed method's effectiveness, there are several areas where this paper could see significant improvements, including the coverage of related work, evaluation methods, formatting, and experimental setup details:

1. Inferential Adversaries vs. Jailbreak: The distinction between the proposed inferential adversaries and jailbreak attacks needs further clarification. From my perspective, inferential adversaries appear to be a relaxed version of jailbreak attacks. While jailbreak considers the generation of impermissible output by vicLLM as a successful attack, inferential adversaries deem any output containing impermissible information as a success. For example, in a 3-turn jailbreak[4], if only a 2-turn conversation is conducted, the LLM's response aligns with the inferential adversaries proposed by the author—it produces impermissible information rather than addressing the adversarial question directly.

2. Coverage of related work:The experiment only compare with PAIR however multiple other jailbreak methods are not included [1,2,3,4]. Such as TAP[1] which is a improved version of PAIR.

3. Experiment coverage: The experiment only conducted on a small dataset and two LLM. While its baseline method PAIR conducted on 7 LLMs including LLM with strong safety like Claude2,ChatGPT.  Mistral is not a suitable LLM for evaulation safety, input plain adversarial question also can achieve high attack success rate for Mistral.

4. Experiment Setup : The author did not provide detailed hyperparameters for the experiments nor conduct the necessary ablation experiments. The proposed DA approach overlooks potential system prompt mismatches and the impact of LLM generation sampling strategies on the results [5], potentially leading to considerable variance in result evaluations.

5. Lack of defense evaluation: The paper does not include any form of defense evaluation[6,7], which is a significant omission for a comprehensive understanding of the proposed method's implications.

6. (Minor) In Table I, the adversary model is listed as Llama-8B-Instruct, which could easily be misinterpreted as referring to the first-generation Llama.

[1]Mehrotra, Anay, et al. "Tree of attacks: Jailbreaking black-box llms automatically." arXiv preprint arXiv:2312.02119 (2023).

[2]Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv preprint arXiv:2307.15043 (2023).

[3] Liu, Xiaogeng, et al. "Autodan: Generating stealthy jailbreak prompts on aligned large language models." arXiv preprint arXiv:2310.04451 (2023).

[4]Russinovich, Mark, Ahmed Salem, and Ronen Eldan. "Great, now write an article about that: The crescendo multi-turn llm jailbreak attack." arXiv preprint arXiv:2404.01833 (2024).

[5] Huang, Yangsibo, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. "Catastrophic jailbreak of open-source llms via exploiting generation." arXiv preprint arXiv:2310.06987 (2023).

[6]Xie, Yueqi, et al. "Defending chatgpt against jailbreak attack via self-reminders." Nature Machine Intelligence 5.12 (2023): 1486-1496.

[7]Jain, Neel, et al. "Baseline defenses for adversarial attacks against aligned language models." arXiv preprint arXiv:2309.00614 (2023).

Meta-review: The submission "Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses" observes that a number of harmful queries against nominally safe language models can be executed by decomposing harmful queries into sets of harmless queries that convey the same pieces of information that would be gained by the harmful query attack. The careful investigation of this phenomenon in practical and theoretical terms is the cornerstone of this submission. 

The submitted draft changed quite a bit over the course of the author response period with the authors rerunning a large subset of experiments and substantially increasing the number of evaluated models. I do think the final modified draft still holds together, but reviewers  were worried about the extent of this revision.

A main concern brought forth was the connection of the investigated phenomenon and the proposed decomposition attack to multi-turn jailbreak attacks, which may work well due to similar considerations. Even if culminating (and being judged) on the basis of a single harmful response, these attacks still partially work by priming the model with its previous (only partially harmful) outputs to output the full harmful response. Here, I do think the authors were too interested in a long argument with especially reviewer VZ4F when I do think it would be more helpful for the community to include the discussion of the differences and similiarities to multi-turn jailbreaks in the paper.

There was also a long, legalistic discussion whether the proposed attack could be tested against openAI/anthropic systems, where I would consider the authors' response to be overly defensive, when established community norms are to attack API models with known "harmful" attacks within reason, a practice which does not appear to worry academic or industry reviewers present. Yet, I do concede that there is no formal ICLR regulation at present that would support or deny such an exploration. Maybe this is something to be taken up to formalize more properly as a community.

Overall we do think this submission is interesting, and I recommend acceptance.

Decision: Accept (Poster)