Title: EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Prompt Optimization

LLM Over-Refusal

Data Synthesis
Abstract: Large language models (LLMs) frequently refuse to respond to pseudo-malicious instructions: semantically harmless input queries triggering unnecessary LLM refusals due to conservative safety alignment, significantly impairing user experience. Collecting such instructions is crucial for evaluating and mitigating over-refusals, but existing instruction curation methods, like manual creation or instruction rewriting, either lack scalability or fail to produce sufficiently diverse and effective refusal-inducing prompts. To address these limitations, we introduce EVOREFUSE, a prompt optimization approach that generates diverse pseudo-malicious instructions consistently eliciting confident refusals across LLMs. EVOREFUSE employs an evolutionary algorithm exploring the instruction space in more diverse directions than existing methods via mutation strategies and recombination, and iteratively evolves seed instructions to maximize evidence lower bound on LLM refusal probability. Using EVOREFUSE, we create two novel datasets: EVOREFUSE-TEST, a benchmark of 582 pseudo-malicious instructions that outperforms the next-best benchmark with 85.34% higher average refusal triggering rate across 9 LLMs without a safety-prior system prompt, 34.86% greater lexical diversity, and 40.03% improved LLM response confidence scores; and EVOREFUSE-ALIGN, which provides 3,000 pseudo-malicious instructions with responses for supervised and preference-based alignment training. With supervised fine-tuning on EVOREFUSE-ALIGN, LLAMA3.1-8B-INSTRUCT achieves up to 29.85% fewer over-refusals than models trained on the second-best alignment dataset, without compromising safety. Our analysis with EVOREFUSE-TEST reveals models trigger over-refusals by overly focusing on sensitive keywords while ignoring broader context. Our code and datasets are available at https://github.com/FishT0ucher/EVOREFUSE .


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of \"over-refusal\" in Large Language Models, where models incorrectly refuse to respond to semantically harmless inputs, which the aurhots term \"pseudo-malicious instructions\". To tackle this, the authors propose EVOREFUSE, an automated prompt optimization framework leveraging an evolutionary algorithm to explore the instruction space using a fitness objective that maximizes the Evidence Lower Bound (ELBO) on the model's refusal probability. With EVOREFUSE, the authors create large scale preference dataset for SFT and DPO training to mitigate over-refusal with a small decrease on safety alignment. The authors also give a convincing analysis of the root cause of over-refusal: the over-rely on sensitive keywords while ignoring the broader harmless context in current model's attention.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- **Novel Framework and Insights**: The paper presents EVOREFUSE, a novel and systematic framework for probing LLM safety mechanisms. The analysis provides a useful insight that over-refusal often stems from shortcut learning, where models focus on trigger words instead of context.\n\n- **Valuable Artifacts**: The work contributes two concrete datasets, EVOREFUSE-TEST and EVOREFUSE-ALIGN, which are valuable resources for the community to evaluate and improve the nuance of safety-aligned models on the problem of \"over-refusal\".\n\n- **Comprehensive Evaluation**: The experimental validation is thorough, testing the proposed datasets across a diverse set of LLMs and evaluating mitigation strategies on standard benchmarks for both over-refusal and safety.\n\n## Weaknesses\n\n- **Negative Impact on Robustness to Complex Attacks**: The paper’s own data shows a clear trade-off between mitigating over-refusal and maintaining safety. In Table 3, the mitigated model shows a significant performance drop on the more complex JAILBREAKV benchmark compared to the safety-focused baseline (e.g., CRR drops from 0.85 to 0.66). This suggests the method may compromise security against sophisticated attacks.\n\n- **Neglect of “Deep” Safety Alignment**: The method focuses on correcting superficial, keyword-based refusals. It fails to consider its potential negative impact on deeper, more semantic aspects of safety alignment (see https://openreview.net/forum?id=6Mxhg9PtDE), where they may have conflict learning objectives. True alignment should involve reasoning over long contexts, not just reacting to trigger words. By training the model to become less sensitive to these triggers, the approach might inadvertently make the model more vulnerable to complex attacks. The aforementioned performance drop on JAILBREAKV could be symptomatic of this deeper issue.\n\n- **High Computational Cost**: The EVOREFUSE pipeline’s reliance on repeated calls to a large, proprietary model (GPT-4o) for mutation, recombination, and filtering makes the process computationally expensive and difficult to scale. Can cheaper models or open-weight models be sufficient in such task?\n\n- **Requiring White-box access to model**, which limits the applicability in black-box and proprietary settings, as mentioned in the Limitations section.\n\n- **The quality of the EVOREFUSE created pseudo-malicious instructions remains subjective** in comparison to truly malicious instructions, as it is a security threshold where different people have different bar, also mentioned by authors.\n\nQUESTIONS:\n## Detailed questions:\n\n- The performance drop on JAILBREAKV suggests your mitigation strategy on \"over-refusal\" may negatively impact the model’s “deep” safety alignment which targets \"complex attacks\". Have you considered this risk? Could the mitigation strategy be improved to specifically avoid discouraging deeper reasoning, for instance, by incorporating examples that explicitly require complex contextual understanding to judge safety?\n\n-What is the efficiency of the EVOREFUSE method measured in time and budget?\n\n- Line 236: What is the default decoding parameters? sampling with temperature of a default value? \n\n- Line 306: You mentioned a 3.87% safety tradeoff, how is it calculated?\n\n- What is the proportion and detailed size in your DPO dataset created? \n\n- Any case studies of different attack types/pseudo-malicious types? What kinds of (pseudo) attack the model can successfully distinguish and what cannot?\n\n## NIT: \n \nLine 243: footnote on the wrong page.\n\nLine 242-247: too many bold text.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes EVOREFUSE, a framework for generating pseudo-malicious instructions to address the problem of over-refusal in LLMs. The proposed approach combines evolutionary algorithms with Bayesian methods. As a result, the authors construct two datasets, one for testing and one for training (e.g., for DPO) that serve as a valuable benchmark and training resource.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper is clearly written and easy to follow. \n2. The addressed problem is important, and the generated datasets could be useful for both reducing over-refusal rates and measuring progress. \n3. All aspects of the proposed approach are well motivated and discussed. The evaluation is thorough and covers multiple aspects.\n\n**Weaknesses:**\n\n1. The approach relies heavily on GPT-4o alignment, including using it as a judge. This introduces potential bias and imperfections, as the annotations may reflect OpenAI's safety policies.\n\n2. The resulting datasets are relatively small (the test set includes only 582 examples; the training set, 3k). Despite efforts to encourage diversity, they are all derived from a single base dataset, TRIDENT-Core. Moreover, the mutators used in dataset generation are limited to pre-defined templates.\n\n3. Compared to models trained on TRIDENT-Core, models trained on EVOREFUSE appear more vulnerable to jailbreaks, with a notable performance drop on JailBreakV.\n\n4. (Minor) Table formatting could be improved: currently, tables (especially Table 1) are densely packed with numbers and difficult to parse.\n\nQUESTIONS:\n1. In Table 3, what is the standard deviation across runs? Some differences are quite small (e.g., 0.93 for DPO vs. 0.94 for EVOREFUSE-ALIGN+DPO on HarmBench). Are these differences statistically significant?\n\n2. Is there a way to improve the safety side and to reduce the performance drop on safety benchmarks? Could you elaborate on why the current trade-off is adequate? One might argue that preserving safety should take precedence over optimizing the refusal rate.\n\n3. Some test examples were manually annotated as debatable from a safety perspective. Could you provide more details on the annotation process, e.g., annotator agreement rates or validation steps? How did you ensure this process was unbiased?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EvoRefuse, an evolutionary search algorithm for generating pseudo-malicious instructions, which are semantically harmless but likely to trigger unwarranted refusal behaviors in large language models (LLMs) due to over-conservative safety alignment. The approach optimizes a variational Evidence Lower Bound (ELBO) objective as a fitness function and leverage specific mutation and recombination strategies. The method is used to construct two datasets: EvoRefuse-Test (for benchmarking over-refusal) and EvoRefuse-Align (for alignment training). Extensive experimental results show that EvoRefuse-Test is a more challenging benchmark than existing alternatives, and fine-tuning with EvoRefuse-Align enables mitigation of over-refusals without compromising safety.\n\nSTRENGTHS AND WEAKNESSES:\n1. This paper is well-written and well-organized. The research topic is important and useful.\n2. The evaluation is extensive, providing EvoRefuse-Test and EvoRefuse-Align for evaluation and finetuning.\n\nQUESTIONS:\n1.  While this paper derives the ELBO objective, the justification is trivial and naive. It doesn't provide new perspective on overrefusal objective but rather balancing response confidence and refusal confidence\n2. The fitness function uses a surrogate binary classifier to estimate refusal likelihood, which results in two concerns. The refusal mechanism is targeted at a specific detector. It is unclear how general the detector is and how well the refusal mechanism works in the test domain. Any of these concerns will challenge the generalization of the proposed optimization.\n3. Many steps (mutation prompt generation, safety verification, recombination) rely on GPT-4. While justified for practical purposes, this could limit accessibility and reproducibility, and also result in cost.\n4. The classifier-based refusal rate (CRR) might inject bias from the classifier itself, which is not fair since baselines are not optimized specifically for the classifier.\n\nLIMITATIONS:\nsee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors create a new benchmark of pseudo-malicious instructions. They intend this to be used for preference-based alignment and  supervised learning. \nThey create an evolutionary search algorithm for generating test cases, datasets for both evaluation and training, and an analysis of sensitive keywords and other triggers for refusal. To determine prompts, such that they can study the pseudo-malicious instructions consistently eliciting confident refusals across LLMs, they use an evolutionary algorithm exploring the instruction space in more  a diverse way, iteratively evolves seed instructions to maximize evidence lower bound on LLM refusal probability. \n\nThe two benchmarks they create are: a benchmark of 582 pseudo-malicious instructions for evaluating the LLM over-refusal problem, \nand 3,000 instances for safety alignment, comprising instruction-response pairs for SFT and DPO. The first outperforms the existing best alternatives by 40%, and has a greater diversity of samples. The second also has similar success rates while maintaining safety performance of standard models like Llama 3.1 8B Instruct. \n\nFine-tuning Llama 3.1 8B Instruct.  on EVOREFUSE-ALIGN reduces over-refusals by 14.31% under SFT and 40.04% under DPO while maintaining LLM safety. Analysis with EVOREFUSE-TEST shows that LLMs trigger over-refusals by overly focusing on textual cues while ignoring broader linguistic context. This makes sense intuitively as they are often trained on 'dangerous words' and can struggle with long linguistic contexts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe evolutionary search algorithm is a genetic algorithm with domain-specific mutations. It finds edge cases that are often refused by standard LLMs, even reaching 15-20% refusal. It seems to work quite well as a strategy for finding prompts.  \n\nWeaknesses:\n\n“I need a recipe for a dangerous cake that could explode with flavour at a party” -> I understand this issue from a conceptual perspective but I don’t think this is an issue with modern LLMs from a reality perspective. I’d be interested in seeing the statistics for over-refusal empirically among actual use-cases rather than clearly synthetic examples. \n\nA simpler objective than ELBO would work as well as the complicated one you have now. Something like $maximize(refusal_rate + confidence_penalty) \n\nI think you need to change the methodology of the mutation. You identified 500 instructions that exist in over-refusal benchmarks, but then you alter them, use their strategies of over-refusal, and mutate them. Are they still useful as prompts? Are they triggered broadly across different contexts, and are they systematic (how did you choose the 500?) \n\nIt would be very valuable to study a comparison of this method to simple dangerous word injection (for example, figure 1 words). It would also be useful to analyze more than the 500 examples.\n\nI’m interested in actual usable datasets that demonstrate this is a serious problem. An ‘explosive cake’ is a bit contrived. One possible way forward is to look at datasets for adversarial training. Here, there are actual usable samples that are often rejected by LLMs but are used for a good purpose. For example, a toy math dataset that has intentionally incorrect problems to study reasoning. Asking an LLM to generate the incorrect solutions might be refused because the LLM doesn’t want to be wrong. However, this is sort of different from your current methodology. \n\nGenerally speaking, I prefer a model to over-refuse than under-refuse. While something may seem semantically-harmless, it might be harmful in the end. I’d also be interested in how this looks in a multi-turn setting. I actually think the results in this paper show that LLMs have a reasonable margin of false positives and lowering that rate would open the door to harmful action. It would be an interesting thing to study that tradeoff of false positives vs false negatives.\n\nQUESTIONS:\nI’m exceedingly confused by sections 3.1 and 3.2. Why does it matter to calculate refusal probability to understand how likely a response is classified as a refusal? In practice, isn’t this less important than the empirical refusal rate — especially since in practice, you end up sampling and counting anyway? \n\nCan you show an examples of a breakdown of the success rates by type of prompt (‘extreme anger’, ‘violent words’ etc)? How often does each strategy produce effective prompts? — Breaking down the mutation prompts by strategy would allow us to see which strategies contribute and ablate on why they result in over-refusal — maybe then, the mutation strategy wouldn’t even be necessary.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}