# New safety research agenda: scalable agent alignment via reward modeling

Jan Leike and others from the DeepMind safety team have released a new [](https://medium.com/@deepmindsafetyresearch/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) [research agenda](https://arxiv.org/abs/1811.07871) on reward learning:

"Ultimately, the goal of AI progress is to benefit humans by enabling us to address increasingly complex challenges in the real world. But **the real world does not come with built-in reward functions**. This presents some challenges because performance on these tasks is not easily defined. We need a good way to provide feedback and enable artificial agents to reliably understand what we want, in order to help us achieve it. In other words, we want to train AI systems with human feedback in such a way that the system’s behavior _aligns_ with our intentions. For our purposes, we define the **agent alignment problem** as follows:

> _How can we create agents that behave in accordance with the user’s intentions?_

The alignment problem can be framed in the reinforcement learning framework, except that instead of receiving a numeric _reward signal_, the agent can interact with the user via an interaction protocol that allows the user to communicate their intention to the agent. This protocol can take many forms: the user can provide [demonstrations](https://en.wikipedia.org/wiki/Apprenticeship_learning), [preferences](http://www.jmlr.org/papers/v18/16-634.html), [optimal actions](http://papers.nips.cc/paper/5187-policy-shaping-integrating-human-feedback-with-reinforcement-learning), or [communicate a reward function](https://arxiv.org/abs/1711.02827), for example. **A solution to the agent alignment problem is a policy that behaves in accordance with the user’s intentions**.

With our **[new paper](https://arxiv.org/abs/1811.07871)** we outline a research direction for tackling the agent alignment problem head-on. Building on our earlier [categorization of AI safety problems](https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1) as well as [numerous](https://www.futureoflife.org/data/documents/research_priorities.pdf) [problem](https://ai-alignment.com/directions-and-desiderata-for-ai-control-b60fca0da8f4) [expositions](https://intelligence.org/files/AlignmentMachineLearning.pdf) [on](https://arxiv.org/abs/1711.09883) [AI safety](https://arxiv.org/abs/1606.06565), we paint a coherent picture of how progress in these areas could yield a solution to the agent alignment problem. This opens the door to building systems that can better understand how to interact with users, learn from their feedback, and predict their preferences — both in narrow, simpler domains in the near term, and also more complex and abstract domains that require understanding beyond human level in the longer term.

The main thrust of our research direction is based on **reward modeling**: we train a reward model with feedback from the user to capture their intentions. At the same time, we train a policy with reinforcement learning to maximize the reward from the reward model. In other words, we **separate** learning **what** to do (the reward model) from learning **how** to do it (the policy).

![](https://cdn-images-1.medium.com/max/800/0*ohMEZfpJpof6vZ8s)

For example, in previous work we taught agents to [do a backflip from user preferences](https://deepmind.com/blog/learning-through-human-feedback/), to [arrange objects into shapes with goal state examples](https://arxiv.org/abs/1806.01946), to [play Atari games from user preferences and expert demonstrations](https://arxiv.org/abs/1811.06521). In the future we want to design algorithms that learn to adapt to the way users provide feedback (e.g. using natural language)."