Learning to selectively attend

Learning to Selectively Attend Samuel J. Gershman (sjgershm@princeton.edu) Jonathan D. Cohen (jdc@princeton.edu) Yael Niv (yael@princeton.edu) Department of Psychology and Neuroscience Institute, Princeton University Princeton, NJ 08540 USA Abstract bias: Restricting attention to only a few dimensions is akin to the belief that only those dimensions are relevant for earning reward. This has the effect of reducing the space of possible value functions to a much smaller subspace. While Bayesian probability theory stipulates the ideal learner, in general it may not be computationally tractable to perform the necessary calculations exactly (Kruschke, 2006; Daw & Courville, 2008). We therefore consider a “hybrid” model that combines the computational efficiency of model- free RL with the statistical efficiency of Bayesian inference. We compare the Bayesian and hybrid models to naive RL (no sparsity bias) and show that models that exploit structured knowledge better capture choice behavior in our experiment. How is reinforcement learning possible in a high-dimensional world? Without making any assumptions about the struc- ture of the state space, the amount of data required to effec- tively learn a value function grows exponentially with the state space’s dimensionality. However, humans learn to solve high- dimensional problems much more rapidly than would be ex- pected under this scenario. This suggests that humans em- ploy inductive biases to guide (and accelerate) their learning. Here we propose one particular bias—sparsity—that amelio- rates the computational challenges posed by high-dimensional state spaces, and present experimental evidence that humans can exploit sparsity information when it is available. Keywords: reinforcement learning; attention; Bayes. Introduction Reinforcement learning (RL) in high-dimensional state spaces is a notoriously difficult problem in machine learn- ing (Sutton & Barto, 1998), primarily because of the curse of dimensionality: The number of states grows exponentially with dimensionality (Bellman, 1957), and thus if one were naively to represent a separate value (expected reward) for each state, one would require astronomical amounts of data to effectively learn the value function (and thereby behave adap- tively). Nonetheless, humans appear to learn rapidly from small amounts of data. Thus, while substantial evidence has accumulated that human behavior follows the predictions of RL models (Dayan & Niv, 2008), these models may funda- mentally underestimate the learning capabilities of humans. Following work in other areas of cognition (Braun, Mehring, & Wolpert, 2009; Kemp & Tenenbaum, 2009), we suggest that rapid learning arises from the exploitation of structured knowledge in the form of inductive biases. In par- ticular, our proposal is that humans employ a sparsity bias: the assumption that only one (or a small number) of dimen- sions (input features) is relevant at any given time for predict- ing reward. For example, when you are at a stoplight, only the color of the light matters, not its shape, size, etc. In other do- mains (such as ordering food in a restaurant), you may know that dimensional relevance is sparse, but not which particu- lar dimensions are relevant (does it matter which restaurant it is? which table I am sitting at? who the chef is? who the waiter is? ); for this purpose, one requires a learning al- gorithm that can exploit sparsity. We formalize this idea in terms of rational statistical inference, and present new exper- imental evidence that human behavior is consistent with such a model. Central to our analysis is the idea that selective attention is a direct consequence of Bayesian inference with a sparsity The Computational Problem For concreteness, we consider one particular example of the general class of reinforcement learning problems for which the sparsity assumption holds. This example is meant to cap- ture the abstract structure of many problems facing humans in the real world, where they must make choices between sev- eral multidimensional stimuli under conditions where most dimensions are unpredictive of reward. This example will also serve as a formal description of the task that we asked human subjects to perform, the results of which we report in a later section. The subject plays N trials, and observes M stimuli simulta- neously on each trial. The ith stimulus on trial n is denoted by a D-dimensional vector x ni , where each integer-valued com- ponent x ni j indicates the property of the jth stimulus dimen- sion (for instance, [color = green, shape = triangle, texture = dots]). Each set of trials has a target dimension d (e.g., ‘shape’) and target property f on that dimension (e.g., ‘cir- cle’). The subject chooses a stimulus c n on each trial and observes a binary reward r n . The probability of reward given choice and target is θ 1 if x nc n d = f P(r n = 1|c n , d, f , X n ) = θ 0 otherwise, In other words, the subject receives a reward with probability θ 1 if the chosen stimulus posseses the target property on the target dimension, and with probability θ 0 otherwise. Bayesian Model Given uncertainty about the target dimension and property, a Bayesian model would use Bayes’ rule to infer the posterior over the target dimensions and property and then calculate the