Reinforcement Learning with Reward Modeling
By: Matthew Williamson
Tags: Reinforcement Learning

As research and development in data science has progressed, Data Science has expanded to include a wide variety of different machine learning techniques with various strengths, weaknesses, and applications. Some of the techniques I find the most fascinating are described by the sub-category ‘deep reinforcement learning’. In this post I’ll dive into a research paper produced as a collaboration between deep-mind and openAI in 2017 titled “Deep Reinforcement Learning from Human Preferences”. First, I’ll explain the problem that reinforcement learning aims to solve and how the general structure for these algorithms is set up, then jump into how the model described in this paper improves upon this concept.

Reinforcement learning is a strategy for producing a model that can produce intelligent actions within a given environment. For example, if you wanted to teach a model how to play a game, you would likely want to use a reinforcement learning model. Early reinforcement learning algorithms like q-learning could only handle discrete environments and action spaces. Evolutionary reinforcement learning algorithms could handle continuous environments and action spaces, but they require large amounts of compute power to train implementations that don’t get stuck in local minima. A more recently developed algorithm called the actor-critic method is an advancement of q-learning that can handle continuous environments and action spaces without the same exponential increase in compute power associated with evolution-based algorithms. Since the algorithm implemented in this paper is a variation on the actor-critic method I’ll explain a little bit about how the Actor-Critic method actually works, but it is worth noting that the concepts in this paper can be applied to any reinforcement learning algorithm that is capable of handling continuous environments and action spaces.

The Actor-Critic method works by implementing an ‘agent’ that contains two models, one that generates actions given a specific environment state, and one that evaluates an action given the action, the state that action was taken in, and the reward produced by the environment. The reward is calculated by the environment and is typically a mathematical formula that calculates how good a particular state is based on the environment variables.

basic Actor-Critic flow of information

Where this algorithm starts to struggle (and most other reinforcement algorithms), is when the reward function is difficult to define. The critic needs to be told how good or bad the current state is to be able to evaluate how good or bad the move that produced that state is, but if the quality of the environment state is difficult or impossible to quantify, then what? If the task was to say, teach a robot how to properly fry an egg, how would you write a mathematical formula using the robot’s sensors to determine how well it’s doing?

This is where the research in this paper comes in. The paper suggests the use of a supervised learning model that produces a reward given a particular state. The data that the reward model is trained on are short segments of environment states and the actions the agent took in those states. Then the model will pair these clips up and show them side by side to a human and the human determines which one looks more like the task that is trying to be achieved.

Actor-Critic flow with reward modeling

This way the agent can keep training in the environment and receiving rewards from the reward model while the human can evaluate the clips at their own pace. As the human evaluates more and more clips, the reward model keeps getting updated and getting more accurate at giving a reward that represents the human’s objective. As the reward model gets more accurate, the agent also becomes better at taking actions that the human prefers. The results the researchers got in this paper indicated that a minimal amount of input by the human resulted in very accurate model performance.

accuracy of reward model, synthetic queries are generated by choosing the clip that the real reward function rated higher

This widens the array of problems reinforcement learning can produce and reduces the number of human hours required to produce significant results. It also reduces the level of expertise needed to train the model. An example would be if you wanted to teach a model how to do a backflip in a game, before you would either have to write an intricate formula for evaluating a backflip attempt or you would have to give the model examples of a human controlling the character to do a backflip, both of which would require some expertise to some degree. On the other hand, if you just need to be able to identify which of two examples looks more like a backflip, just about anybody is going to be able to make that distinction. This opens the door for crowd sourcing the dataset for the reward model if the problem is complex enough. For the relatively simple challenges the models in the paper where learning, only about 1 hour of non-expert human evaluation produced results that were as good or better than using a pre-defined reward function that took approximately 2 hours for an expert to write. Additionally, the researchers were able to train some of the models to exhibit novel behaviors for which writing a reward function would be extraordinarily difficult, and yet it still only took about 1 hour of non-expert evaluation to produce that functionality. This implies that this concept should scale well with more complex tasks with hard to define evaluation. If the task you want to train an agent to perform can be described by saying “I’m not sure exactly how to define it but I’d know it when I see it” then this modeling technique is probably the answer.