Goal-Proximity Decision Making: Who needs reward anyway?

Goal-Proximity Decision Making: Who needs reward anyway? Vladislav D. Veksler Wayne D. Gray Michael J. Schoelles (vekslv@rpi.edu) (grayw@rpi.edu) Cognitive Science Department Rensselaer Polytechnic Institute 110 8th Street Troy, NY 12180 USA (schoem@rpi.edu) Abstract Reinforcement learning (RL) models of decision-making cannot account for human decisions in the absence of prior reward or punishment. We propose a mechanism for choosing among available options based on goal-option association strengths, where association strengths between objects represent object proximity. The proposed mechanism, Goal- Proximity Decision-making (GPD), is implemented within the ACT-R cognitive framework. A one-choice navigation experiment is presented. GPD captures human performance in the early trials of the experiment, where RL cannot. Keywords: RL, GPD, reinforcement learning, associative learning, latent learning, ACT-R, information scent, decision- making, seeking behavior, navigation, model-tracing. Introduction How does a cognitive agent choose a path of actions from an infinitely large decision-space? Reinforcement learning (RL) models, which are models of human trial-and-error behavior, explain how an agent may reduce its decision- space over time by attending to the reward structure of the task-environment. However, as goals change, so does the reward structure of the agent’s world. Relearning the reward structure for every possible goal may take an extremely long time. For greater efficiency, a cognitive agent should be able to learn more about its environment than just the reward structure, and to exploit this knowledge for achieving new goals in the absence of prior reward/punishment. For example, a person may see a hardware store on their way to the mall, and incidentally learn its location. Some time later, if they need to go to a hardware store, the person can find their way to that store, because they know its location. There had been no reward or punishment for the actions leading to this hardware store, and so the ability to find its location cannot be explained solely through the principles of reinforcement learning. We propose a mechanism for making decisions in the absence of prior reward or punishment, and provide initial tests of its fidelity and efficiency as compared to RL. Given multiple possible paths of action, the proposed mechanism chooses the path most strongly associated with the current goal, regardless of prior reward. Strength of association between any two items, in turn, depends on experienced temporal proximity of those items. From here forth we refer to the proposed mechanism as GPD (goal-proximity decision-making). The rest of this paper describes a key theoretical problem for RL models of decision-making (the 2-goal problem), briefly summarizes classic evidence in psychological literature for reward-independent decision-making in humans and animals, and presents two computational models that exemplify non-RL-based decision-making. We then outline the implementation of the GPD mechanism within the ACT-R cognitive framework. Finally, we describe a single-choice navigation experiment, and provide fits of GPD and RL decision mechanisms to human data. We conclude that GPD can account for human performance where RL cannot – prior to any reward or punishment. What this paper is not about Because everything in cognition is so closely knit, the GPD theory may evoke topics that are outside of the scope of current work. The following topics are important to cognitive science but tangential to the focus of this paper. First, GPD is not meant to replace RL, but rather to complement it. How GPD and RL may interact is a topic for further research. Second, GPD does not address planning. GPD is a theory of immediate behavior; how this behavior may be used in complex planning procedures is a tangential topic. Third, GPD partially addresses episodic memory and associative learning. However, associative learning is not the focus of this paper. Rather, the focus here is on the goal- oriented decision-making that can emerge from a simple associative learning mechanism. The topic of associative learning should comprise other lines of research (e.g. sequence recall, free association, priming) in addition to this one, and is too extensive to address here. Fourth, GPD describes how an agent may choose which option to approach given multiple possible paths. Although avoidance behavior is just as important as approach behavior, and should eventually become part of the GPD theory, it is assumed here to be a separate topic. The 2-goal Problem Consider a scenario where an agent has to achieve goal A, then goal B, in the same environment. To increase efficiency humans and animals would learn the environment during task A, and perform faster on task B (the Experiment below provides evidence for this phenomenon). That is, we do not just learn the positive utility for the actions that