When Things Get Worse before they Get Better: Regulatory Fit and Average-Reward Learning in a Dynamic Decision-Making Environment

When Things Get Worse before they Get Better: Regulatory Fit and Average- Reward Learning in a Dynamic Decision-Making Environment A. Ross Otto, Arthur B. Markman, and Bradley C. Love Department of Psychology, University of Texas, Austin, TX 78712 USA [rotto@mail.utexas. edu, markman@psy.utexas. edu, love@psy.utexas. edu] Todd M. Gureckis Department of Psychology, New York University, 6 Washington Place, New York, NY 10003 [todd.gureckis@nyu.edu] Abstract This work explores the influence of motivation on choice behavior in a dynamic decision-making environment, where the payoffs from each choice depend on one’s recent choice history. Previous research reveals increased levels of exploratory choice among participants in a regulatory fit. The present study placed promotion and prevention-focused participants in a dynamic environment for which optimal performance requires that participants sustain a single choice strategy in the face of temporary payoff decreases. These participants either gained or lost points with each choice. Our behavioral results and model-based analysis, using the average-reward reinforcement learning framework, revealed differential levels of reactivity to local changes in payoffs— specifically, participants in a regulatory fit were less reactive to local perturbations in payoffs than participants in a regulatory mismatch and performed more optimally as a result. Keywords: Decision making; motivation; reinforcement learning Introduction Motivation is essential to action (e.g., Carver & Scheier, 1998; Yerkes & Dodson, 1908). Social psychology makes the distinction between two general motivational orientations (or regulatory foci), a promotion focus and a prevention focus, which accentuate potential gains and losses in the environment, respectively (Higgins, 1997). Recent research reveals that an interaction occurs between one’s regulatory focus and the reward structure of the task being performed, affecting peoples’ use of flexible strategies in a number of tasks. In one study (Worthy, Maddox, & Markman, 2007) utilizing a two-armed bandit task for which optimal choice behavior required exploratory choices as opposed to exploitative choices (c.f. Daw et al., 2006), participants attempting to earn a prize (inducing a promotion focus) exhibited more optimal choice behavior when the task environment had a gains reward structure (i.e., participants were maximizing gains of points) than when the task environment had a losses reward structure (i.e. participants were minimizing loss of points). Likewise, participants attempting to avoid losing a prize (a prevention focus) performed more optimally when the task involved a losses reward structure than when the task involved a gains reward structure. In n-armed bandit tasks where which the decision-maker learns to maximize his or her payoffs by making choices and experiencing the consequences of those choices (e.g. Daw et al., 2006; Bechara, A.R. Damasio, H. Damasio, & Anderson, 1994), optimal performance depends on balancing the demands of gathering and exploiting information about choice payoffs. Worthy et al.’s (2007) study demonstrated that participants in a regulatory fit, for whom their situational regulatory focus matches the reward structure of the task environment, exhibit more exploratory choice strategies than do participants in a regulatory mismatch, who exhibit more exploitative choice strategies. Through continued exploration of choices—with the consequence of occasionally taking decreases in payoffs— participants in a regulatory fit display behavior that is adaptive for the overall long-term pursuit of rewards. Further, their model-based analyses suggested that participants in a regulatory fit place less weight on outcomes from recent choices than do participants in a regulatory mismatch. While this class of tasks is well suited for investigating exploratory versus exploitative choice behavior, we seek to understand how motivational factors affect the degree to which recent changes in payoffs drive choice behavior. In this report, we examine the effects of regulatory fit in a two-option, repeated-choice decision making task in which payoff-maximizing, long-term optimal behavior requires that participants persevere with one choice strategy, sustaining temporary decreases in payoffs in order to maximize long-term gain. Our experiment placed participants in a version of the “rising optimum” task, previously used to investigate temporal-difference accounts of learning (Eagelman, Person, & Berns, 1998; Montague & Berns, 2002) and the problem of temporal credit assignment in human sequential decision-making (Bogacz, McClure, Li, Cohen, & Montague, 2007). Unlike other bandit tasks in which payoff contingencies remain invariant to participants’ behavior (e.g., Bechara et al., 1994; Daw et al., 2006; Worthy et al., 2007; Yechaim & Busemeyer; 2005), in this task, the state of the task environment changes as a function of a participant’s recent choices, which in turn governs the payoffs associated with each action. Consider the two payoff curves depicted in Figure 1, which correspond to the possible payoffs for two choices A and B in Egelman et al.’s (1998) “rising optimum” task. The payoff received from a choice depends on the proportion of A choices made over the last 20 trials, represented by the