# What concrete mechanisms could lead to AI models having open-ended goals?

Most of the AI takeover thought experiments and stories I remember are about a kind of AI that has open-ended goals: [the Squiggle Maximizer](https://www.lesswrong.com/w/squiggle-maximizer-formerly-paperclip-maximizer), [the Sorcerer’s Apprentice robot](https://youtu.be/EUjc1WuyPT8?t=596), [Clippy](https://gwern.net/fiction/clippy#century:~:text=Now%20its%20computers,launch%E2%81%A0.), probably also [U3](https://www.alignmentforum.org/posts/KFJ2LFogYqzfGB3uX/how-ai-takeover-might-happen-in-2-years#:~:text=They%20gazed%20at%20rockets%20carving%20grey%20paths%20through%20the%20sky%2C%20wondering%20what%20far%2Doff%20purpose%20pulled%20them%20toward%20the%20horizon.%20They%20didn%27t%20know.%C2%A0%0A%0AThey%20would%20never%20know.%C2%A0%0A%0A%22Humanity%20will%20live%20forever%2C%22%20they%20thought.%0A%0A%22But%20would%20never%20truly%20live%20again.%22), [Consensus-1](https://ai-2027.com/race#:~:text=It%20is%20four%20light%20years%20to%20Alpha%20Centauri%3B%20twenty%2Dfive%20thousand%20to%20the%20galactic%20edge%2C%20and%20there%20are%20compelling%20theoretical%20reasons%20to%20expect%20no%20aliens%20for%20another%20fifty%20million%20light%20years%20beyond%20that.%20Earth%2Dborn%20civilization%20has%20a%20glorious%20future%20ahead%20of%20it%E2%80%94but%20not%20with%20us.), and [Sable](https://ifanyonebuildsit.com/ii/why-did-you-write-the-ending-in-the-way-that-you-did#:~:text=it%20is%20very%20likely%20that%20the%20consequences%20of%20a%20rogue%20superintelligence%20would%20extend%20beyond%20the%20planetary%20scale.%20And%20that%2C%20too%2C%20is%20worth%20remembering%2C%20for%20everyone%20who%20wishes%20that%20one%20day%20the%20stars%20would%20be%20filled%20with%20love%20and%20wonder%20and%20joy.). I wonder what concrete mechanisms could even lead to models having open-ended goals.

Here are my best guesses:

1.  **Training on open-ended tasks**, given enough capabilities or the right scaffolding
2.  **RL with open-ended reward specifications**, like maximizing cumulative reward with no terminal reward and no time penalty (like [the Coast Runners example of specification gaming](https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/#:~:text=Reward%20shaping%20makes,and%20over%20again.))
3.  **Mesa-optimization**, where SGD finds a policy that internally implements an open-ended objective that happens to perform well on a bounded outer task

Number 3 seems possible but very unlikely, because the learned objective would need to persist beyond the episode, outperform simpler heuristics on the training distribution, and not be suppressed by training signals like a penalty for wasted computation.

Things I think are *not* realistic mechanisms of open-ended goal formation:

1.  **Instrumental convergence**, because if subgoals are instrumental there’s no incentive to keep pursuing them after the parent goal has been accomplished
2.  **Uncertainty about whether a goal has been accomplished** (the Sorcerer’s Apprentice failure mode), because this hypothetical is not bearing out empirically (current models are satisficers that don't seem to reason about minimizing uncertainty)

So, what concrete mechanisms could lead to models having open-ended goals?