Separating Skills from Preference: Using Learning to Program by Reward

Developers of artificial agents commonly take the view that we can only specify agent behavior via the expensive process of implementing new skills. This paper offers an alternative expressed by the separation hypothesis: that the behavioral differences among individuals are due to the action of distinct preferences over the same set of skills. We test this hypothesis in a simulated automotive domain by using a reinforcement learning algorithm to induce vehicle control policies, given a structured skill for driving that contains options, and a user-supplied reward function. We show that qualitatively distinct reward functions produce agents with qualitatively distinct behavior over the same set of skills. This leads to a new development metaphor we call Ôprogramming by rewardÕ. 1. Motivation and Background In many domains, humans exhibit complex physical behaviors that let them accomplish sophisticated tasks. Researchers have explored two main approaches to learning such behaviors, each associated with a different class of representational formalisms. One paradigm encodes control knowledge as rules or similar structures (e.g., Laird & Rosenbloom, 1990; Sammut, 1996) that state conditions under which to execute actions. An alternative framework instead specifies some function that maps state-action pairs onto a numeric utility (e.g., Watkins & Dayan, 1992), which is then used to select among actions. Both approaches have repeatedly demonstrated their ability to learn useful control policies across a broad range of domains, yet each lends itself most naturally to different aspects of intelligent behavior. This idea is best illustrated by work on game playing, where developers regularly use rules or other logical constraints to specify which moves are legal but invoke numeric evaluation functions to select among them. We claim that a similar division of labor will prove useful in research on policies for reactive control, including learning such policies from agent experience. In this paper, we assume that an agent already has access to a set of logical rules that constrain the allowable actions, but that it must learn the value of its remaining options from delayed reward. Elsewhere (Shapiro et al., 2001), we have shown that this use of background knowledge can greatly speed the process of learning control policies. Here we focus on a different claim: that providing a learning agent with different reward signals can lead to a great variety of behaviors that still share the same overall structure. This approach to learning — which we call programming by reward -should prove useful in constructing simulated agents for computer games, in supporting personalized services that must operate within certain constraints, and many other tasks. In the following pages, we report one instance of this general framework, which we have cast in an architecture for physical agents called Icarus. We begin by describing the architecture’s logical formalism for encoding hierarchical skills, taking examples from the task of driving an automobile. We then turn to the value functions that Icarus uses to select among applicable skills and its algorithm for using delayed rewards to update these functions. After this, we present experimental studies designed to test our hypothesis that providing such a system with different rewards can produce distinctive yet still viable policies. Finally, we examine some other approaches to learning complex skills and suggest directions for additional research on this topic. 2. The Icarus Language Icarus is a language for specifying the behavior of artificial agents that learn. Its structure is dually motivated by the desire to build practical agent applications and the desire to support policy learning in a computationally efficient way. We responded to the first goal by providing Icarus with powerful representations. However, the desire for rapid learning suggests a simpler format that offers a clear mapping into the Markov decision process (MDP) model, since MDPs provide a conceptual framework for developing learning algorithms,