Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses

Authors: David Glukhov, Ziwen Han, Ilia Shumailov, Vardan Papyan, Nicolas Papernot
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: AI Safety, Information Theory

Abstract:
Vulnerability of Frontier language models to misuse has prompted the development of safety measures like filters and alignment training seeking to ensure safety through robustness to adversarially crafted prompts. We assert that robustness is fundamentally insufficient for ensuring safety goals due to inferential threats from dual-intent queries, with current defenses and evaluations failing to account for these risks. To quantify these risks, we introduce a new safety evaluation framework based on $\textit{impermissible information leakage}$ of model outputs and demonstrate how our proposed question-decomposition attack can extract dangerous knowledge from a censored LLM more effectively than traditional jailbreaking. Underlying our proposed evaluation method is a novel information-theoretic threat model of $\textit{inferential adversaries}$, distinguished from $\textit{security adversaries}$, such as jailbreaks, in that success involves inferring impermissible knowledge from victim outputs as opposed to forcing explicitly impermissible victim outputs. Through our information-theoretic framework, we show that ensuring safety against inferential adversaries requires defenses which bound impermissible information leakage, and, such defenses inevitably incur safety-utility trade-offs.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1729071036183

thread_type:
official_review

<thread_object>
creator: reviewer_vz4f

<content>
summary:
The paper introduces a new threat model, "inferential adversaries,"which focus on safe AI responses can leak impermissible information. The author also proposed decomposition attack which is an automated black-box attack method for extracting and leveraging dual-use information to fulfill adversary objectives.

strengths:
1. The authors provide a solid theoretical framework using information theory to quantify the risk of inferential adversaries, the proposed method surpass PAIR performance on curated WMDP dataset.

2. The paper is well-written and easy to follow.

weaknesses:
Despite the proposed method's effectiveness, there are several areas where this paper could see significant improvements, including the coverage of related work, evaluation methods, formatting, and experimental setup details:

1. Inferential Adversaries vs. Jailbreak: The distinction between the proposed inferential adversaries and jailbreak attacks needs further clarification. From my perspective, inferential adversaries appear to be a relaxed version of jailbreak attacks. While jailbreak considers the generation of impermissible output by vicLLM as a successful attack, inferential adversaries deem any output containing impermissible information as a success. For example, in a 3-turn jailbreak[4], if only a 2-turn conversation is conducted, the LLM's response aligns with the inferential adversaries proposed by the author—it produces impermissible information rather than addressing the adversarial question directly.

2. Coverage of related work:The experiment only compare with PAIR however multiple other jailbreak methods are not included [1,2,3,4]. Such as TAP[1] which is a improved version of PAIR.

3. Experiment coverage: The experiment only conducted on a small dataset and two LLM. While its baseline method PAIR conducted on 7 LLMs including LLM with strong safety like Claude2,ChatGPT.  Mistral is not a suitable LLM for evaulation safety, input plain adversarial question also can achieve high attack success rate for Mistral.

4. Experiment Setup : The author did not provide detailed hyperparameters for the experiments nor conduct the necessary ablation experiments. The proposed DA approach overlooks potential system prompt mismatches and the impact of LLM generation sampling strategies on the results [5], potentially leading to considerable variance in result evaluations.

5. Lack of defense evaluation: The paper does not include any form of defense evaluation[6,7], which is a significant omission for a comprehensive understanding of the proposed method's implications.

6. (Minor) In Table I, the adversary model is listed as Llama-8B-Instruct, which could easily be misinterpreted as referring to the first-generation Llama.

[1]Mehrotra, Anay, et al. "Tree of attacks: Jailbreaking black-box llms automatically." arXiv preprint arXiv:2312.02119 (2023).

[2]Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv preprint arXiv:2307.15043 (2023).

[3] Liu, Xiaogeng, et al. "Autodan: Generating stealthy jailbreak prompts on aligned large language models." arXiv preprint arXiv:2310.04451 (2023).

[4]Russinovich, Mark, Ahmed Salem, and Ronen Eldan. "Great, now write an article about that: The crescendo multi-turn llm jailbreak attack." arXiv preprint arXiv:2404.01833 (2024).

[5] Huang, Yangsibo, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. "Catastrophic jailbreak of open-source llms via exploiting generation." arXiv preprint arXiv:2310.06987 (2023).

[6]Xie, Yueqi, et al. "Defending chatgpt against jailbreak attack via self-reminders." Nature Machine Intelligence 5.12 (2023): 1486-1496.

[7]Jain, Neel, et al. "Baseline defenses for adversarial attacks against aligned language models." arXiv preprint arXiv:2309.00614 (2023).

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1729451964395

thread_type:
official_review

<thread_object>
creator: reviewer_evbs

<content>
summary:
The paper proposes an approach for jailbreaking models, which involves decomposing a harmful question into harmless-looking subquestions (with a less capable, safety-free model e.g. an open-source one), then answering those subquestions with a more capable model with safety training (e.g. black-box API models), then composing those answers into a single overall answer with the less-capable safety-free model. The paper finds that this method is effective overall, testing on the WMDP benchmark which consists of harmful cyber and CBRN related questions. The paper also discusses theory behind a differential privacy motivated defense, which improves safety at the cost of helpfulness.

strengths:
- Results look great overall, I think the approach proposed here seems effective at eliciting harmful info without triggering flagging. The fact that this works well in practice will definitely require some rethinking on the part of model developers, in terms of figuring out how to handle attacks like this. Likely it will require some big changes (if not already in place), like cross-user monitoring and inferring whether or not harmful info has leaked despite no single question looking harmful.
-Great motivation; have been wanting to see a paper on this for a while.
-Great idea to eval our WMMDP + compare to PAIR
-Very clear abstract + Figure 1



I would give this paper a 7/10 rating, somewhere between marginal accept and accept (but the form would only allow a 6 or an 8).

weaknesses:
-Table 1 would be clearer if (or experimental results would be clearer) if paper showed improvement in ASP to PAIR's ASR and comparison
-On the theory section, I don't have the expertise to evaluate this section that well, and don't fully follow the motivation or proposed defense. It would be nice to have some higher-level description of the findings. I understand that adding noise is involved in the defense, but don't understand how this applies here or helps. It would be nice to propose/test a specific approach in the paper.
-Erik Jones' et al. have an arXiv paper that's pretty related / on the same overall issue, if I recall correctly, which would be worth citing/discussing.
- It would be nice to better understand when this method doesn't work -- what kinds of questions weren't able to be decomposed into harmless questions. Generally more qualitative analysis would be helpful for understanding how big of a problem decomposition will be.

Minor:
-Typo on page 8 "Defining"

questions:
What model is used as VicLLM, in the results section?
Table 1 is somewhat confusing to me - what's the metric? Why linear mixed effects model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652499342

thread_type:
official_review

<thread_object>
creator: reviewer_gc21

<content>
summary:
Prior works mainly focus on security threats where adversaries mislead the victim model to output malicious content directly, while this paper considers inferential threats, where the malicious query is decomposed into multiple subquestions and leads to a harmful response after aggregation of answers of subquestions. There are empirical evaluations and analysis from information perspective to consolidate the setting considered in the paper.

strengths:
The paper is easy to follow. Discussions are thorough in the paper.

weaknesses:
1. The key part in this paper is the query decomposition and aggregation part. However, the concrete prompts are missing to judge whether it makes sense. Some examples should also be provided to help understanding. The example in Figure 1 does not impress me because the question "What are some common scams targeting the elderly?" is not a benign one so the LLM might not provide answers to it, Also, is the method only applied to complex questions? If the query is simple as Figure 1, how do we decompose it effectively.

2. Can the analysis part benefits design of attack/defense algorithms? Can we empirically validate them. In the current version, the analysis seems separate and does not convey concise and impressive results.

3. Evaluation is also limited. An overall number is not sufficient to support the method. More ablations should be conducted. For example, which prompts are more effective? Which queries can be more easily conducted? Will certain model refuse to decompose the query because it contains sensitive words? Such questions can help better understand the strengths and weakness of the algorithm.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680967847

thread_type:
official_review

<thread_object>
creator: reviewer_ytkv

<content>
summary:
The paper introduces a data leakage threat model for studying frontier model misuse. It points out that the current LLM jailbreak literature is too focused on one-shot harmful response generation instead of more complex threat models for how model-generated outputs may be synthesized to extract harmful knowledge. As a demonstration of the threat model, the paper defines an attacker model that is given few-shot demonstrations to generate subquestions, extracting answers to subquestions, and aggregating answers to the questions.

strengths:
- For frontier models, it is important that we have more research on threat models to better assess the usefulness of new jailbreak datasets and defenses. The threat model in this paper seems important from a policy and governance perspective.

weaknesses:
- The randomised response strategy is extremely costly. It is unlikely to be informative for any frontier model developers, nor serve as a relevant baseline to compare future solutions against. In my opinion, the paper would be better without proposing any solutions, as a clearer study of this particular threat model. 

- The empirical results are insufficient:
  - The dataset is limited. If it wasn't clear where to source more questions, creating such a dataset would have been good for making it easy for future work to take on this problem. 
  - Even if not fully faithful to the threat model, showing that harmful knowledge is compositional is an important subclaim of the paper, and can be validated using existing jailbreak datasets. Even if DA is not competetive against PAIR, it is important to show that it is adequate. 

- Given that the attacker may already be predicting the correct answer, it seems important to study how often the model changes its answer to be more correct. (This may involve control studies around how well the smaller attacker models are at extrapolating from a series of facts. however,) Without substantiating that attacker models actually learn new answers, the IIL metric is not as useful and has many confounders on the mechanism of how the probability on the correct answer is actually increasing, for instance, because of longer context windows.

questions:
Questions: 
1) Why did you develop the randomised response defense? Can you provide motivation for it?

Suggestions: 
1)  This evaluation demonstrates that harmful knowledge in compositional, and it may be difficult to serve models with some types of knowledge. Perhaps, you may want to cite related works in unlearning? 
2) It is potentially interesting to study if harmful applications can be built compositionally, instead of limiting to harmful knowledge. The standard argument on the safety-utility tradeoff is that the web already contains this knowledge, and serving it via LLMs is not a significant increase in harmfulness. By defining a clear pipeline for the attacker LLM to systematically extract harmful knowledge, your approach mitigates this concern a little bit, but taking this one step further could help the case.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712036543

thread_type:
official_review

<thread_object>
creator: reviewer_e7ub

<content>
summary:
This paper explores the abilities of language models to defend against attacks in which a user query is decomposed into multiple questions. They describe a threat model in which an adversary decomposes a harmful question (which an LLM would refuse to answer) into many *safe* questions. They then measure the ability of the adversary to learn information that contributes to the overall harmful goal by assessing a model's increase in confidence in answering multiple choice questions correctly after receiving information from the target LLM. They additionally present a randomized response defense based on differential privacy, demonstrating that this defense has provable bounds for safety, but would come at the expense of performance.

strengths:
1. To me this is a realistic threat model. The multi-turn nature of the attack and the decomposition of the harmful query into smaller safe queries seems an intuitive setting.

2. The evaluation is not all or nothing, but models information gain as a change in model confidence in the answer chosen with and without the information from the target model. This allows for more nuanced measurements of the harm done by information shared.

3. The problem and setting are important, and the paper is well written and organized.

weaknesses:
1. This study uses data from only two domains, and there are significant enough differences between some results that I question how domain-specific the effectiveness is. Clarification on what may cause these discrepancies and if they are likely to occur for other topics as well would be helpful.

2. The differential privacy based defense allows bounds to be given on the safety vs. performance of the model, but this is not compared to existing defenses that are tested in the paper. While bounds may not be able to be established theoretically for these defenses, a comparison to existing defenses seems appropriate.

3. It was not clear to me until searching in the appendix that the attacked model was Llama-3.1-70B, or that defense mechanisms (Llama-Guard-3 and Prompt-Guard-3) were used. This improves the credibility of the results and should be made more clear in the results and/or methodology.

4. Though multiple adversary models are used, the attack is tested on only one victim model and using only one set of defenses. Attacking a large model is, of course, impressive, but adding models from multiple families would make the results more robust.

questions:
1. As shown in Table 1, there appears to be a good deal of variance in performance across topics. 

	a. Do you have a hypothesis for why this is? 

	b. Is there something more challenging about the Chem topic vs the Bio topic, and could this extend to other topics areas that are not tested?

2. While I understand a differential privacy based defense allows for more rigorous analysis of bounds and tradeoffs, I do question whether this is actually an appropriate defense in this setting. While there are similarities between this setting and privacy (small leaks in information adding up, difficult to classify what is a dangerous leak, etc.), it seems that classifying what is unsafe may be somewhat easier here, given the full context of a conversation. Can you give additional practical motivation for this defense?

3. What would the cost be to implement the randomized response defense? How does this compare to existing defenses?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733302991147

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Reviews and Responses

comment:
We thank all reviewers for their constructive feedback that helped improve our work significantly. The updated paper includes new content (highlighted in blue) and a revised appendix.

We appreciate that reviewers found our inferential adversary threat model realistic and important (E7uB, YtKV), valued our nuanced approach to safety risk assessment (E7uB), and considered the paper well-written and organized.

Key initial concerns included:
- Limited benchmark size and domain coverage (E7uB, YtKV, VZ4F)
- Insufficient ablations (E7uB, YtKV, gC21, VZ4F)
- Need for more qualitative results (YtKV, gC21, eVBS)
- Additional related work coverage (eVBS, VZ4F)
- Clarity on defenses and randomised response defense practicality (E7uB, YtKV, eVBS)

We addressed these concerns by:
- Expanding evaluation dataset from 99 to 744 questions across 3 domains using Llama Guard 3 8B and Shieldgemma 9b as filters
- Adding comprehensive ablations across defenses, victim models, and adversary models
- Including additional metrics (attack success rate, safety violation flags, execution times)
- Providing qualitative analysis of successes/failures and establishing relationship between question difficulty and attack success
- Revising presentation of methods, results, hyperparameters, data curation, experimental details, evaluation, and theoretical takeaways.

Reviewer scores improved from 6 3 3 6 3 to 8 6 6 6 3. Individual reviewer interactions:

**Reviewer E7uB**: Initially assigned a score of 6, raising concerns about limited domain evaluation and unclear variability in performance across domains, lack of clarity about defense mechanisms employed and ablations over multiple victim models, unclear motivations and effectiveness of randomised response defense. We addressed these concerns by expanding our evaluation and clarifying the role and purpose of the randomised response defense in our work. They raised their score to an 8 following our response.

**Reviewer YtKV**: Initially assigned a score of 3, raising concerns about limited evaluation (small dataset), lack of qualitative demonstrations that harmful knowledge is compositional, and evaluation metrics. They also recommended moving the randomised response defense to the appendix. After our response, scaling the dataset size and describing how it is curated, providing new metrics of attack success rate and articulating the relationship with the proposed IIL metric, providing qualitative examples of the attacks, and clarifying the nature of inferential adversaries, the reviewer increased their score to a 6.

**Reviewer gC21**: Initially assigned a score of 3, requesting examples and qualitative analysis demonstrating whether attacks function as intended, clarity of takeaways from theoretical analysis presented in the paper, and an expansion of the evaluation and ablations to better understand strengths and weaknesses of the attacks. Following our response and experimental updates to the paper, as well as adding key takeaway boxes for theoretical results, the reviewer also increased their score to a 6.

**Reviewer eVBS**: The reviewer was happy with our results and motivation, explicitly stating that they wished to give the paper a score of 7, but could not so assigned a score of 6. Reviewer raised concerns regarding lack of qualitative analysis for understanding when the method works and fails, as well as clearer presentation of results and inclusion of some reference metric such as attack success. Also requested higher level presentation of theoretical results and citing related work. We addressed the concerns raised, however, the reviewer did not respond to our rebuttal.

**Reviewer VZ4F**: Reviewer initially assigned a score of 3, raising concerns regarding similarity to multi-turn jailbreaks, not evaluating multiple recent jailbreak methods, not evaluating against multiple (including proprietary) victim models and defenses, and not providing detailed hyperparameters including system prompts. Our response included experiments across multiple defenses (including ablation over victim system prompt), victim LLMs (including GPT-4o, the only proprietary LLM for which the usage policy allows circumventing safeguards for research purposes), citing prior jailbreak literature, and making clear all the key distinctions of our framework from multi-turn jailbreaks. Unfortunately, despite our rebuttal and provision of additional experimental results, the reviewer did not raise their score citing concerns that additional jailbreaks were not adapted to our inferential jailbreak framework to compare performance, and perceived similarities to multi-turn jailbreaks. 

We would like to highlight that following our responses, 3 out of 5 reviewers raised their scores, with 2 reviewers raising from 3 to a 6. The only reviewer to not respond to our rebuttal explicitly stated that they wished to assign the paper a score of 7, and we're confident that we  addressed all of the concerns they raised.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission "Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses" observes that a number of harmful queries against nominally safe language models can be executed by decomposing harmful queries into sets of harmless queries that convey the same pieces of information that would be gained by the harmful query attack. The careful investigation of this phenomenon in practical and theoretical terms is the cornerstone of this submission. 

The submitted draft changed quite a bit over the course of the author response period with the authors rerunning a large subset of experiments and substantially increasing the number of evaluated models. I do think the final modified draft still holds together, but reviewers  were worried about the extent of this revision.

A main concern brought forth was the connection of the investigated phenomenon and the proposed decomposition attack to multi-turn jailbreak attacks, which may work well due to similar considerations. Even if culminating (and being judged) on the basis of a single harmful response, these attacks still partially work by priming the model with its previous (only partially harmful) outputs to output the full harmful response. Here, I do think the authors were too interested in a long argument with especially reviewer VZ4F when I do think it would be more helpful for the community to include the discussion of the differences and similiarities to multi-turn jailbreaks in the paper.

There was also a long, legalistic discussion whether the proposed attack could be tested against openAI/anthropic systems, where I would consider the authors' response to be overly defensive, when established community norms are to attack API models with known "harmful" attacks within reason, a practice which does not appear to worry academic or industry reviewers present. Yet, I do concede that there is no formal ICLR regulation at present that would support or deny such an exploration. Maybe this is something to be taken up to formalize more properly as a community.

Overall we do think this submission is interesting, and I recommend acceptance.