Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer
  a Safer Path?
Authors: name: Yoshua Bengio; name: Michael Cohen; name: Damiano Fornasiere; name: Joumana Ghosn; name: Pietro Greiner; name: Matt MacDermott; name: Sören Mindermann; name: Adam Oberman; name: Jesse Richardson; name: Oliver Richardson; name: Marc-Antoine Rondeau; name: Pierre-Luc St-Charles; name: David Williams-King
Published: 2025-02-21 18:28:36

Abstract: The leading AI companies are increasingly focused on building generalist AI
agents -- systems that can autonomously plan, act, and pursue goals across
almost all tasks that humans can perform. Despite how useful these systems
might be, unchecked AI agency poses significant risks to public safety and
security, ranging from misuse by malicious actors to a potentially irreversible
loss of human control. We discuss how these risks arise from current AI
training methods. Indeed, various scenarios and experiments have demonstrated
the possibility of AI agents engaging in deception or pursuing goals that were
not specified by human operators and that conflict with human interests, such
as self-preservation. Following the precautionary principle, we see a strong
need for safer, yet still useful, alternatives to the current agency-driven
trajectory. Accordingly, we propose as a core building block for further
advances the development of a non-agentic AI system that is trustworthy and
safe by design, which we call Scientist AI. This system is designed to explain
the world from observations, as opposed to taking actions in it to imitate or
please humans. It comprises a world model that generates theories to explain
data and a question-answering inference machine. Both components operate with
an explicit notion of uncertainty to mitigate the risks of overconfident
predictions. In light of these considerations, a Scientist AI could be used to
assist human researchers in accelerating scientific progress, including in AI
safety. In particular, our system can be employed as a guardrail against AI
agents that might be created despite the risks involved. Ultimately, focusing
on non-agentic AI may enable the benefits of AI innovation while avoiding the
risks associated with the current trajectory. We hope these arguments will
motivate researchers, developers, and policymakers to favor this safer path.