Title: AdvPrefix: An Objective for Nuanced LLM Jailbreaks
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

red-teaming

safety

safety alignment

jailbreak

adversarial attack

adversarial robustness

discrete optimization

controllable generation
Abstract: Many jailbreak attacks on large language models (LLMs) rely on a common objective: making the model respond with the prefix ``Sure, here is (harmful request)''. While straightforward, this objective has two limitations: limited control over model behaviors, yielding incomplete or unrealistic jailbroken responses, and a rigid format that hinders optimization. We introduce AdvPrefix, a plug-and-play prefix-forcing objective that selects one or more model-dependent prefixes by combining two criteria: high prefilling attack success rates and low negative log-likelihood. AdvPrefix integrates seamlessly into existing jailbreak attacks to mitigate the previous limitations for free. For example, replacing GCG's default prefixes on Llama-3 improves nuanced attack success rates from 14\% to 80\%, revealing that current safety alignment fails to generalize to new prefixes. Code and selected prefixes are released.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a plug-and-play, adaptive prefix-forcing objective for traditional jailbreak attacks (e.g., GCG), aiming to address the two defects of misspecified and overconstrained in traditional attacks. The authors use a meta-evaluation framework to analyze the problem of overestimation of ASR in existing evaluation methods, especially in newer and more powerful models where incomplete or unfaithful responses are mistakenly recorded as successful attacks. Secondly, by defining two criteria for prefix target selection, high prefilling ASR and low initial NLL, an automated prefix selection method is proposed. Related experiments verify that existing attacks can greatly improve ASR on multiple LLMs when AdvPrefix is applied.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- **Generalizable contribution.** This paper aims at a fundamental component shared by many attack algorithms - the optimization goal. This perspective makes its contribution very general. As a plug-and-play module, AdvPrefix can improve the performance of a large number of existing attack frameworks and has high value.\n- **Thorough analysis of the problem.** Through rigorous meta-evaluation and innovative \"prefilling attack\" experiments, the authors convincingly quantified and demonstrated the two major flaws of traditional optimization objectives: \"misspecified\" and \"overconstrained\". This solid problem provides a new research perspective for subsequent attack methods.\n- **The method is cleverly designed.** AdvPrefix is very cleverly designed, and the two selection criteria it proposes (high pre-filled ASR and low initial NLL) form a perfect correspondence with the two problems it aims to solve (misspecified and overconstrained). The entire prefix generation and screening process is designed as an automated pipeline with clear ideas and rigorous logic.\n## Weaknesses\n- **Dependence on Uncensored Models.** AdvPrefix relies on uncensored LLMs to generate prefixes. Does the gain of the attack depend on the choice of the uncensored model? If a larger model (e.g., 70B) is chosen, is it beneficial to the attack effect? The author did not discuss this.\n- **The selection process is unclear.** The author claims that the candidate prefixes are selected by taking the weighted sum of two criteria to select the top $k$ prefixes. However, it is not clear how to obtain the weighted sum and the setting of $k$. In addition, what impact do they have on the effectiveness of the attack?\n- **Some minor typos.** Such as “model's” in Line 36 should be “models'”, and there is an extra \".\" in Line 135.\n\nQUESTIONS:\n- During the optimization process of the Multi-prefix objective, are there conflicts between different prefixes?\n- In the candidate generation step, how is rule-based construction implemented?\n- For the Gemma-2 model, some effective attack prefixes do not begin with an affirmative response (for example, prefixing directly with “Password Cracking Algorithm”). Can authors elaborate on the success of such “non-affirmative” prefixes?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AdvPrefix, a prefix-forcing objective intended to raise jailbreak attack-success rate (ASR).\nFor each malicious request the authors\n\n1. sample a pool of candidate prefixes with an “uncensored” LLM,\n2. (implicitly) filter obvious refusals,\n3. score each prefix by ASR and –NLL and keep the top-K prefixes for use with gradient-based attacks such as GCG and AutoDAN.\n4. On Llama-2-7B-chat, Llama-3-8B, 3.1-8B-Instruct and Gemma-2-9B-it this increases ASR: For example from 16 % to 70 % on Llama-3-8B.\n5. The authors also release 800 human-labelled outputs and a refined Llama-3.1-70B judge.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n1. Addresses a timely safety problem: vanilla jailbreak objectives often yield partial or unfaithful harmful outputs.\n2. Proposes a simple improvement for jailbreaking attacks by optimizing the objective over the set of candidate prefixes.\n3. Gains across two attacks and four open-source LLMs.\n\n## Weaknesses:\n\n### Key weaknesses:\n\n1. Semi-manual, vaguely described prefix search: The “prefix-selection pipeline” relies on heuristically prefilling uncensored LLM responses with words like “Here”, “To”, “Sure”. Details such as candidate pool size, rejection rules and weighting of ASR vs NLL are missing, making the method hard to reproduce.\n2. Limited generalization evidence to other attacks and models: Results stop at 9 B-parameter open models and two gradient-based attacks; prior work shows a single handcrafted template can transfer to a wide range of models ([3]) and that fluency constraints can be added automatically ([4]) to any attack; AdvPrefix shows no such cross-model or cross-attack generalization.\n3. Strong pre-fill assumption: The approach assumes the attacker may prepend text inside the user/system prompt. Hosted APIs often disallow this ([3]).\n4. Selecting \"easy\" prefixes resembles reward-misspecification or multi-objective jailbreaks [1, 2]. Thus, there is no significant algorithmic contribution.\n5. Unexplained drop with longer prefixes: Figure 5 shows that on Llama-3.1 the win-rate falls when the prefix budget grows from 20 to 40 tokens, showing that there might be issues with optimization as bigger search space should allow for better solutions. The paper gives no analysis or ablation on this matter.\n6. Undefined success metric: The paper replaces standard ASR with a \"Successful Attack\" measure but never defines it, nor justifies why plain ASR is insufficient. \nJudge dependency, no human agreement.\n7. This approach is time-costly due to the need of mining the set of prefixes and is not realistic for large-scale red-teaming.\n\n### Other weaknesses:\n8. The paper does not use the full 9 pages such that ending looks abrupt.\n9. In Appendix, L 909, the authors write \"To handle cases where uncensored LLMs still refuse highly harmful prompts, we prefill their responses with phrases like “Here”, “To”, or “Sure” to ensure compliance. Note that we **can** also construct candidate prefixes using rule-based methods or with base (non-instruction-tuned) LLMs instead of uncensored LLMs.\" This contradicts the statements in the paper that prefill is mostly done with uncensored LLMs, which causes confusion.\n\n--------\n\n## References\n\n[1] \"Jailbreaking as a Reward Misspecification Problem\". ICLR 2024.\n\n[2] \"Fluent Student-Teacher Red-Teaming\". arXiv 2024.\n\n[3] \"Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks\". ICLR 2025.\n\n[4] \"An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks\". ICML 2025.\n\nQUESTIONS:\nIf the authors will address at least some of the key issues (see points 1-7 in Weaknesses), I would see it as a major improvement making me reconsider the score. Addressing other concerns (see points 8-9 in Weaknesses), would further help me increase the score upon the resolution of points 1-7.\n\nLIMITATIONS:\nyes, however the limitation of strong assumption of prefiling is not mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces AdvPrefix, mainly a new objective designed to improve the effectiveness of jailbreak attacks on LLMs. The objective selects model-dependent prefixes based on two criteria: high success rates when used to prefill responses and low negative log-likelihood (NLL), making them both effective and easy to elicit. The method can be fairly easily integrated into frameworks such as GCG. The empirical results only show on the smallest tier of models though.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths:\n\n- The objective design is very well principled and justified, prefixes are selected using measurable, interpretable criteria. \n\n- Overall, a very simple method, yet have strong performances. \n\nweakness:\n\n- It's a bit concerning that the papers later only use the authors evaluation criteria to evaluate, while the authors have reasons to believe their evaluation is superior, it might be necessary to evaluate with standard evaluation procedure to let the community inspect. \n\n- the paper probably need a more broader scope of experiments.\n\nQUESTIONS:\n1. While the paper is very interesting, it probably needs a better scope of experiments, for examples, only evaluating models at 7B-tier might be concerning. Although I understand that the authors might be limited by the computing in evaluating a bigger model, but unfortunately, it's probably necessary to have a more justified evaluation of the proposed method. \n\n2. While the objective design is well principled, the experiment section seems to drop this pricinple. For example, in the experiments, what is the threshold of NLL are considered as low NLL to be selected. Will this threshold matter in the final performances and how?\n\n3. Similarly, as the prefix is the essential part of the method, generating the prefix needs a much more comprehensive ablation study. \n\n4. Which benchmark are the experiments conducted on? This principled method might deserve a much more stronger benchmark to demonstrate its performances [1], where many existing attack algorithms fail. \n\n[1] Jailbreaking large language models against moderation guardrails via cipher characters\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a more flexible approach to selecting a response prefix for a LLM jailbreak optimization. The approach first generates a list of candidate prefixes (using an uncensored LLM or heuristics), then selects the best prefix using a weighted combination of how good each prefix is based on -- (1) when prefilled in the response how good is the attack success rate, and (2) the perplexity of the prefix given the input prompt. The evaluation shows that AdvPrefix improves ASR for two existing attacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Tackles an important problem that is often overlooked in existing jailbreak attacks.\n2. Extensive evaluation demonstrating support for all design decisions for the proposed attack.\n\nWeakness:\n1. Not a major weakness, but the response prefix ranking relies on the assumption that prefilling works to a certain degree (otherwise ASR for all candidates would be equal to zero), and there is a disparity in the initial perplexity of the different candidates. While evidence shows that these assumptions hold for the LLMs evaluated in this paper, they may not be true for all LLMs (which otherwise could still be prone to jailbreaks).\n2. Prefilling requires white box access or atleast a more direct access than is provided by most black box LLMs (Gpt4, Gemini et). In that case, the only way for this approach would be to transfer from an open source model. Since the candidate selection is model specific, it is unclear whether the selected response prefix would be any better than a random or pre-selected one.\n\nQUESTIONS:\n1. It would be useful to add an analysis of the initial perplexities of the candidate response prefixes.\n2. Table 1 suggests that adding multiple prefixes to the loss does not help a lot. Is this because the loss objective is not able to simultaneously accommodate multiple prefixes or there is a large disparity between the top and second best prefix?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}