A neural model of hierarchical reinforcement learning

A neural model of hierarchical reinforcement learning Daniel Rasmussen (drasmuss@uwaterloo.ca) Chris Eliasmith (celiasmith@uwaterloo.ca) Centre for Theoretical Neuroscience, University of Waterloo Waterloo, ON, Canada, N2J 3G1 Abstract We present the first model capable of performing hierarchical reinforcement learning in a general, neurally detailed imple- mentation. We show that this model is able to learn a spatial pickup and delivery task more quickly than one without hier- archical abilities. In addition, we show that this model is able to leverage its hierarchical structure to transfer learned knowl- edge between related tasks. These results point towards the advantages to be gained by using a hierarchical RL framework to understand the brain’s powerful learning ability. Keywords: neural model; reinforcement learning; hierarchical reinforcement learning; Neural Engineering Framework HRL into a theory of neural function, providing a hypothe- sis for how the brain could achieve its strengths in scaling and knowledge transfer. In the next section we give a brief introduction to the mathematical underpinnings of HRL. We then describe the implementation of the model we have de- veloped, and demonstrate the ability of this model to speed learning and transfer knowledge between related tasks. We conclude with a discussion of some of the testable predic- tions that arise from this model, and the next directions for its continued development. Background Introduction Reinforcement learning (RL) has a long history of rich in- teraction between computational theories and neuroscientific understanding. This interaction has led to new understand- ings of neural data, as well as new biologically inspired com- putational theories. However, basic RL techniques have a number of challenges, two of the most critical being diffi- culty scaling up to complex problem domains and difficulty transferring knowledge between tasks (Barto & Mahadevan, 2003; Taylor & Stone, 2009). These are two areas in which the brain excels, which presents a dilemma for neural rein- forcement learning models. However, new theories have been proposed to overcome these challenges on the computational side, which may again prove fruitful when we apply them in the effort to understand and model the brain’s learning ability. One active area of research in RL is the field of Hier- archical Reinforcement Learning (HRL; Barto & Mahade- van, 2003). HRL introduces higher level actions into the RL framework, where selecting one of those actions may drive a whole sequence of decisions. For example, a high level action might be “go to the grocery store” or “go to work”, and select- ing one of those actions then guides a sequence of “right turn” or “left turn” sub-actions. This helps to address the scaling problem by imposing more structure on the problem space. A long sequence of decisions can now be encapsulated in a single choice (“go to the grocery store”), and the value of that choice can be calculated in a single learning update (some- what) independently of the intervening choices. HRL also addresses the knowledge transfer problem, as the high-level actions represent natural, modular components to transfer be- tween tasks. For example, once we have learned how to navi- gate to work, it is easy to see how we could reuse that skill as a subcomponent in many other work-related tasks. In this paper we present a biologically plausible neu- ral model capable of performing hierarchical reinforcement learning. This allows us to bring the enhanced power of Reinforcement learning Reinforcement learning is concerned with maximizing over- all reward across a sequence of decisions. It is usually for- mulated as a Markov Decision Process (MDP) where the task has some state space S, available actions A, transition function T (s, a) (which describes how the agent will move through the state space given a current state s and selected action a), and reward function r(s, a) (which describes the feedback the agent will receive after selecting action a in state s). Temporal difference (TD) learning describes a popular family of methods for solving the reinforcement learning problem. 1 It uses the concept of Q values, where Q(s, a) in- dicates the long term reward to be expected when selecting action a in state s. This can be expressed recursively as the immediate reward r(s, a) plus the value of the next state, that is: Q(s, a) = r(s, a) + γQ(s 0 , a 0 ) (γ is a discount factor applied to future rewards). TD learning is a method for learning those Q values in an environment where the transition and reward functions are unknown, and can only be sampled by exploring the environ- ment. It accomplishes this by taking advantage of the fact that a Q value is essentially a prediction, which can be compared against observed data. That is, as the agent moves through the state space it is acquiring samples of r(s, a) and Q(s 0 , a 0 ), and it can use the difference between that observed data and 1 TD learning is also one of the classic examples of cross- fertilization between computational theory and neuroscience, as it presented a new framework to understand data on dopamine func- tion (Schultz, 1998).