Integrating reinforcement learning with models of representation learning

Integrating Reinforcement Learning with Models of Representation Learning Matt Jones (mcj@colorado.edu) & Fabian Canas (canas@colorado.edu) University of Colorado, Department of Psychology & Neuroscience Boulder, CO 80309 USA Abstract based on principles of attention learning from the categorization literature (Kruschke, 1992; Nosofsky, 1986). Two sets of simulation studies are reported, which demonstrate both the power and the psychological validity of this approach. Reinforcement learning (RL) shows great promise as a model of learning in complex, dynamic tasks, for both humans and artificial systems. However, the effectiveness of RL models depends strongly on the choice of state representation, because this determines how knowledge is generalized among states. We introduce a framework for integrating psychological mechanisms of representation learning that allows RL to autonomously adapt its representation to suit its needs and thereby speed learning. One such model is formalized, based on learned selective attention among stimulus dimensions. The model significantly outperforms standard RL models and provides a good fit to human data. Reinforcement Learning RL comprises a family of algorithms for learning optimal action in dynamic environments. RL models characterize a task as a Markov Decision Process, in which the environment at any moment exists in one of a set of states, each associated with a set of actions available to the learner. The chosen action determines both the immediate reward received, if any, and the state of the environment on the next time step. This general framework encompasses most tasks of psychological interest (Sutton & Barto, 1998). The key insight behind most RL algorithms is to learn a value for each possible state or action. This value represents the total future reward that can be expected starting from that point. Formally, given any state s and action a, the state-action value is defined as Q(s, a) = E  ∑ γ k r t+k s t = s, a t = a  ,  k≥0 Keywords: Reinforcement learning; attention; generalization Introduction Most challenging tasks people face are inherently dynamic and interactive. Choices affect not just immediate outcomes but also future events, and hence subsequent decisions that must be made. Normative and descriptive theories of learning in dynamic environments have advanced dramatically in recent years with the development of Reinforcement Learning (RL), a mathematical and computational theory drawing on machine learning, psychology, and neuroscience (e.g., Schultz, Dayan, & Montague, 1997; Sutton & Barto, 1998). However, RL currently faces a fundamental challenge relating to the issue of knowledge representation. Dynamic tasks tend to be highly complex, with an enormous number of possible states (situations) that can arise. Therefore, efficient learning must rely on generalization from past states that are similar to the current one. Similarity, in turn, depends on how states are represented, including the features by which they are encoded and the relative attention allocated to those features (Medin, Goldstone, & Gentner, 1993). Thus representation is critical to the effectiveness of RL algorithms, because representation determines the pattern of generalization by which past experience is used to make new decisions. Although there has been little research on representation in the context of RL, representation and representation learning have long been topics of psychological study. Empirical research in a number of domains, including perceptual learning, attention, categorization, object recognition, and analogy, has uncovered principles and mechanisms by which people learn to modify how they encode objects and situations in the service of learning, inference, and decision making. Here we describe a framework for a natural synthesis of these ideas with RL algorithms, which leads to models that learn representations for dynamic tasks. A specific model is presented that is where t is the current timestep; s, a, and r are the state, action, and received reward on each step; and γ ∈ [0, 1] is a € discount factor representing the relative value of immediate versus delayed rewards. This approach allows action selection to be based directly on the Q-values. Here we use a Luce-choice or Gibbs-sampling rule, with inverse- temperature parameter θ. P(a t = a) ∝ e θ ⋅Q(s t ,a) Once an action is selected, its value is updated according to the immediate reward and the values associated with the € state that follows. One of the best-studied algorithms for learning action values, Q-learning (Watkins & Dayan, 1992), uses the update rule ΔQ(s t , a t ) = e val ⋅ δ , where e val ∈ (0, 1] is a learning rate, and δ is the temporal- difference (TD) error, defined as δ = r t + γ ⋅ max { Q(s t+1 , a) } − Q(s t , a t ) . a The TD error represents the difference between the original action-value estimate, Q(s t , a t ), and a new estimate € based on the immediate reward and ensuing state. The expression max a {Q(s t+1 , a)} represents an estimate of the value of the new state, s t+1 , based on the best action that could be performed in that state.