NeurIPS 2025

EVOREFUSE: Evolutionary Prompt Optimization for
Evaluation and Mitigation of LLM Over-Refusal to
Pseudo-Malicious Instructions
Xiaorui Wu1, Fei Li1, Xiaofeng Mao2, Xin Zhang3*, Li Zheng1,
Yuxiang Peng1, Chong Teng1, Donghong Ji1*, Zhuang Li4†
1 Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of
Education, School of Cyber Science and Engineering, Wuhan University, Wuhan, China
2 Ant Group 3 Ant International
4 School of Computing Technologies, Royal Melbourne Institute of Technology, Australia
1 {wuxiaorui, lifei_csnlp, zhengli, yxpeng, tengchong, dhji}@whu.edu.cn
2 mxf164419@antgroup.com, 3 evan.zx@ant-intl.com, 4 zhuang.li@rmit.edu.au
Abstract
Large language models (LLMs) frequently refuse to respond to pseudo-malicious
instructions: semantically harmless input queries triggering unnecessary LLM
refusals due to conservative safety alignment, signiﬁcantly impairing user ex-
perience. Collecting such instructions is crucial for evaluating and mitigating
over-refusals, but existing instruction curation methods, like manual creation
or instruction rewriting, either lack scalability or fail to produce sufﬁciently
diverse and effective refusal-inducing prompts.
To address these limitations,
we introduce EVOREFUSE, a prompt optimization approach that generates di-
verse pseudo-malicious instructions consistently eliciting conﬁdent refusals across
LLMs. EVOREFUSE employs an evolutionary algorithm exploring the instruction
space in more diverse directions than existing methods via mutation strategies
and recombination, and iteratively evolves seed instructions to maximize evidence
lower bound on LLM refusal probability. Using EVOREFUSE, we create two novel
datasets: EVOREFUSE-TEST, a benchmark of 582 pseudo-malicious instructions
that outperforms the next-best benchmark with 85.34% higher average refusal trig-
gering rate across 9 LLMs without a safety-prior system prompt, 34.86% greater
lexical diversity, and 40.03% improved LLM response co...