A Comparison of Human and Agent Reinforcement Learning in Partially Observable Domains

A Comparison of Human and Agent Reinforcement Learning in Partially Observable Domains Zoubin Ghahramani University of Cambridge United Kingdom zoubin@eng.cam. ac.uk Finale Doshi-Velez Massachusetts Institute of Technology Cambridge, Massachusetts finale@alum.mit. edu Abstract It is commonly stated that reinforcement learning (RL) algo- rithms require more samples to learn than humans. In this work, we investigate this claim using two standard problems from the RL literature. We compare the performance of human subjects to RL techniques. We find that context—the meaning- fulness of the observations—plays a significant role in the rate of human RL. Moreover, without contextual information, hu- mans often fare much worse than classic algorithms. Compar- ing the detailed responses of humans and RL algorithms, we also find that humans appear to employ rather different strate- gies from standard algorithms, even in cases where they had indistinguishable performance to them. Keywords: sequential decision-making; reinforcement learn- ing; computational models; machine learning The ability of humans to make sequential decisions under uncertainty has been widely studied in psychology and neu- roscience. The field of reinforcement learning (RL) studies the theoretical formulation and algorithmic implementation of artificial agents that make sequential decisions to maxi- mize their expected reward (Sutton & Barto, 1998). While RL algorithms often provide theoretical guarantees on the qual- ity of the agent’s long-term behaviour, the common lore in the RL community (Singh, 2009; Peters, Bagnell, & Schaal, 2006; Morimoto & Doya, 2005) is these approaches are painfully slow, requiring thousands of trials to learn to act in, what seem to humans, relatively simple domains. While RL has been applied as a theoretical tool for un- derstanding human decision making behaviour (Samejima & Doya, 2007; Daw, O’Doherty, Dayan, Seymour, & Dolan, 2006; Kakade & Dayan, 2002; Daw, Courville, & Tourtezky, 2006; Yoshida & Ishii, 2003; Acu˜na & Schrater, 2008; Dayan & Daw, 2008), the supposed “slowness” of RL methods has not been experimentally tested against human learning per- formance. Are these RL algorithms actually slower to learn than humans? To what extent is this lore biased by the fact that humans bring structural knowledge from previous expe- riences to new problems? For example, when entering a new building, a human will probably assume that he cannot walk through its walls, whereas RL problems would typically have to relearn this fact for each new location. Humans also tend to assume near-deterministic worlds, whereas RL algorithms are often initialized as believing all possible outcomes are equally likely. In this paper, we focus on the approaches humans take on problems where aspects of the environment cannot be fully- observed (formally partially observable Markov decision pro- cesses (POMDPs)). POMDPs offer a more realistic scenario for decision making under uncertainty than the simpler (fully- observable) Markov decision processes, since they assume that the state of the world is known, but inferred from noisy observations. In this setting, we show that, surprisingly, when put in an identical setup on standard decision making prob- lems, RL methods often learn faster and achieve better solu- tions than humans. Even more surprisingly, while human per- formance does improve when subjects are given contextual information about the problem, their average performance of- ten still does not match RL methods. Our work has inter- esting implications for our understanding of both human and machine decision making. Without contextual information, humans may require more experience than RL algorithms to perform well even on simple problems. However, making use of context is one of the important open problems for machine learning. Experiment We tested two hypotheses: first, that human subjects would perform significantly better if given contextual observations, and second, that human subjects would outperform RL algo- rithms. Performance was evaluated as the sum of rewards ob- tained during the last tenth of a learning trial. We also exam- ined which RL algorithms’ behaviour most closely matched human behaviour. Task Descriptions The tasks consisted of two common problems in the RL lit- erature, both formulated as POMDPs. Playing the role of the agent, the human subject—who had no initial knowledge about the structure of the problem—selected actions to take. The problem returned an observation, displayed on a com- puter screen, which depended on the underlying state of the environment, and an immediate reward. The subject’s goal was to maximize their cumulative rewards. Each task could be presented to the subject in two differ- ent versions. In the with-context version C+, the domain’s observations had meaning in the context of the task. In the context-free version C−, observations had no meaning; the C− version of the problem was meant to simulate what a RL algorithm might “see,” as a computer system cannot attribute meaning or significance to particular observations. In the first problem, the tigerworld task (Kaelbling, Littman, & Cassandra, 1995), players were confronted with two doors (see figure ? for an illustration). Behind one door was a tiger (reward = −100); behind the other was a prize (reward = +10). At every iteration, players had three options: