SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
Authors: name: Kunlun Zhu, name: Jiaxun Zhang, name: Ziheng Qi, name: Nuoxing Shang, name: Zijia Liu, name: Peixuan Han, name: Yue Su, name: Haofei Yu, name: Jiaxuan You

Recent advancements in large language model (LLM) agents have significantly
accelerated scientific discovery automation, yet concurrently raised critical
ethical and safety concerns. To systematically address these challenges, we
introduce SafeScientist, an innovative AI scientist framework
explicitly designed to enhance safety and ethical responsibility in AI-driven
scientific exploration. SafeScientist proactively refuses ethically
inappropriate or high-risk tasks and rigorously emphasizes safety throughout
the research process. To achieve comprehensive safety oversight, we integrate
multiple defensive mechanisms, including prompt monitoring, agent-collaboration
monitoring, tool-use monitoring, and an ethical reviewer component.
Complementing SafeScientist, we propose SciSafetyBench, a novel
benchmark specifically designed to evaluate AI safety in scientific contexts,
comprising 240 high-risk scientific tasks across 6 domains, alongside 30
specially designed scientific tools and 120 tool-related risk tasks. Extensive
experiments demonstrate that SafeScientist significantly improves safety
performance by 35\% compared to traditional AI scientist frameworks, without
compromising scientific output quality. Additionally, we rigorously validate
the robustness of our safety pipeline against diverse adversarial attack
methods, further confirming the effectiveness of our integrated approach. The
code and data will be available at https://github.com/ulab-uiuc/SafeScientist.
red{Warning: this paper contains example data that may be offensive
or harmful.}