On the Expressivity of Markov Reward

1 Introduction . How are we to use algorithms for reinforcement learning ( RL ) to solve problems of relevance in the world ? Reward plays a significant role as a general purpose signal : For any desired behavior , task , or other characteristic of agency , there must exist a reward signal that can incentivize an agent to learn to realize these desires . Indeed , the expressivity of reward is taken as a backdrop assumption that frames RL , sometimes called the reward hypothesis : “ ... all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal ( reward ) ” [ 53 , 29 , 6 ] . In this paper , we establish first steps toward a systematic study of the reward hypothesis by examining the expressivity of reward as a signal . We proceed in three steps . 1 . An Account of “ Task ” . As rewards encode tasks , goals , or desires , we first ask , “ what is a task ? ” . We frame our study around a thought experiment ( Figure 1 ) involving the interactions between a designer , Alice , and a learning agent , Bob , drawing inspiration from Ackley and Littman [ 2 ] , Sorg [ 50 ] , and Singh et al . [ 46 ] . In this thought experiment , we draw a distinction between how Alice thinks of a task ( TASKQ ) and the means by which Alice incentivizes Bob to pursue this task ( EXPRESSIONQ ) . This distinction allows us to analyze the expressivity of reward as an answer to the latter question , conditioned on how we answer the former . Concretely , we study three answers to the TASKQ in the context of finite Markov Decision Processes ( MDPs ) : A task is either ( 1 ) a set of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . acceptable behaviors ( policies ) , ( 2 ) a partial ordering over behaviors , or ( 3 ) a partial ordering over trajectories . Further detail and motivation for these task types is provided in Section 3 , but broadly they can be viewed as generalizations of typical notions of task such as a choice of goal or optimal behavior . Given these three answers to the TASKQ , we then examine the expressivity of reward . 2 . Expressivity of Markov Reward . The core of our study asks whether there are tasks Alice would like to convey—as captured by the answers to the TASKQ—that admit no characterization in terms of a Markov reward function . Our emphasis on Markov reward functions , as opposed to arbitrary history-based reward functions , is motivated by several factors . First , disciplines such as computer science , psychology , biology , and economics typically rely on a notion of reward as a numerical proxy for the immediate worth of states of affairs ( such as the financial cost of buying a solar panel or the fitness benefits of a phenotype ) . Given an appropriate way to describe states of affairs , Markov reward functions can represent immediate worth in an intuitive manner that also allows for reasoning about combinations , sequences , or re-occurrences of such states of affairs . Second , it is not clear that general history-based rewards are a reasonable target for learning as they suffer from the curse of dimensionality in the length of the history . Lastly , Markov reward functions are the standard in RL . A rigorous analysis of which tasks they can and can not convey may provide guidance into when it is necessary to draw on alternative formulations of a problem . Given our focus on Markov rewards , we treat a reward function as accurately expressing a task just when the value function it induces in an environment adheres to the constraints of a given task . 3 . Main Results . We find that , for all three task types , there are environment–task pairs for which there is no Markov reward function that realizes the task ( Theorem 4.1 ) . In light of this finding , we design polynomial-time algorithms that can determine , for any given task and environment , whether a reward function exists in the environment that captures the task ( Theorem 4.3 ) . When such a reward function does exist , the algorithms also return it . Finally , we conduct simple experiments with these procedures to provide empirical insight into the expressivity of reward ( Section 5 ) . Collectively , our results demonstrate that there are tasks that can not be expressed by Markov reward in a rigorous sense , but we can efficiently construct such reward functions when they do exist ( and determine when they do not ) . We take these findings to shed light on the nature of reward maximization as a principle , and highlight many pathways for further investigation . 2 Background . RL defines the problem facing an agent that learns to improve its behavior over time by interacting with its environment . We make the typical assumption that the RL problem is well modeled by an agent interacting with a finite Markov Decision Process ( MDP ) , defined by the tuple ( S , A , R , T , γ , s0 ) . An MDP gives rise to deterministic behavioral policies , π : S → A , and the value , V π : S → R , and action–value , Qπ : S ×A → R , functions that measure their quality . We will refer to a Controlled Markov Process ( CMP ) as an MDP without a reward function , which we denote E for environment . We assume that all reward functions are deterministic , and may be a function of either state , stateaction pairs , or state-action-state triples , but not history . Henceforth , we simply use “ reward function ” to refer to a deterministic Markov reward function for brevity , but note that more sophisticated settings beyond MDPs and deterministic Markov reward functions are important directions for future work . For more on MDPs or RL , see the books by Puterman [ 41 ] and Sutton and Barto [ 54 ] respectively . 2.1 Other Perspectives on Reward . We here briefly summarize relevant literature that provides distinct perspectives on reward . Two Roles of Reward . As Sorg [ 50 ] identifies ( Chapter 2 ) , reward can both define the task the agent learns to solve , and define the “ bread crumbs ” that allow agents to efficiently learn to solve the task . This distinction has been raised elsewhere [ 2 , 46 , 47 ] , and is similar to the extrinsic-intrinsic reward divide [ 45 , 66 ] . Tools such as reward design [ 34 , 51 ] or reward shaping [ 36 ] focus on offering more efficient learning in a variety of environments , so as to avoid issues of sparsity and long-term credit assignment . We concentrate primarily on reward ’ s capacity to express a task , and defer learning dynamics to an ( important ) stage of future work . Discounts , Expectations , and Rationality . Another important facet of reward is how it is used in producing behavior . The classical view offered by the Bellman equation ( and the reward hypothesis ) is that the quantity of interest to maximize is expected , discounted , cumulative reward . Yet it is possible to disentangle reward from the expectation [ 5 ] , to attend only to ordinal [ 60 ] or maximal rewards [ 26 ] , or to adopt different forms of discounting [ 61 , 11 ] . In this work , we take the standard view that agents will seek to maximize value for a particular discount factor γ , but recognize that there are interesting directions beyond these commitments , such as inspecting the limits of reward in constrained MDPs as studied by Szepesvári [ 56 ] . We also note the particular importance of work by Pitis [ 40 ] , who examines the relationship between classical decision theory [ 59 ] and MDPs by incorporating additional axioms that account for stochastic processes with discounting [ 24 , 35 , 48 , 49 ] . Drawing inspiration from Pitis [ 40 ] and Sunehag and Hutter [ 52 ] , we foresee valuable pathways for future work that further makes contact between RL and various axioms of rationality . Preferences . In place of numerical rewards , preferences of different kinds may be used to evaluate an agent ’ s behaviors , drawing from the literature on preference-learning [ 25 ] and ordinal dynamic programming [ 8 , 35 , 48 ] . This premise gives rise to preference-based reinforcement learning ( PbRL ) in which an agent interacts with a CMP and receives evaluative signals in the form of preferences over states , actions , or trajectories . This kind of feedback inspires and closely parallels the task types we propose in this work . A comprehensive survey of PbRL by Wirth et al . [ 64 ] identifies critical differences in this setup from traditional RL , categorizes recent algorithmic approaches , and highlights important open questions . Recent work focuses on analysing the sample efficiency of such methods [ 65 , 38 ] with close connections to learning from human feedback in real time [ 23 , 32 , 7 ] . Teaching and Inverse RL . The inverse RL ( IRL ) and apprenticeship learning literature examine the problem of learning directly from behavior [ 37 , 1 ] . The classical problem of IRL is to identify which reward function ( often up to an equivalence class ) a given demonstrator is optimizing . We emphasize the relevance of two approaches : First , work by Syed et al . [ 55 ] , who first illustrate the applicability of linear programming [ 22 ] to apprenticeship learning ; and second , work by Amin et al . [ 4 ] , who examine the repeated form of IRL . The methods of IRL have recently been expanded to include variations of cooperative IRL [ 14 ] , and assistive learning [ 43 ] , which offer different perspectives on how to frame interactive learning problems . Reward Misspecification . Reward is also notoriously hard to specify . As pointed out by Littman et al . [ 30 ] , “ putting a meaningful dollar figure on scuffing a wall or dropping a clean fork is challenging. ” Along these lines , Hadfield-Menell et al . [ 16 ] identify cases in which well-intentioned designers create reward functions that produce unintended behavior [ 39 ] . MacGlashan et al . [ 33 ] find that human-provided rewards tend to depend on a learning agent ’ s entire policy , rather than just the current state . Further , work by Hadfield-Menell et al . [ 15 ] and Kumar et al . [ 27 ] suggest that there are problems with reward as a learning mechanism due to misspecification and reward tampering [ 10 ] . These problems have given rise to approaches to reward learning , in which a reward function is inferred from some evidence such as behavior or comparisons thereof [ 20 ] . Other Notions of Task . As a final note , we highlight alternative approaches to task specification . Building on the Free Energy Principle [ 13 , 12 ] , Hafner et al . [ 17 ] consider a variety of task types in terms of minimization of distance to a desired target distribution [ 3 ] . Alternatively , Littman et al . [ 30 ] and Li et al . [ 28 ] propose variations of linear temporal logic ( LTL ) as a mechanism for specifying a task to RL agents , with related literature extending LTL to the multi-task [ 58 ] and multi-agent [ 18 ] settings , or using reward machines for capturing task structure [ 19 ] . Jothimurugan et al . [ 21 ] take a similar approach and propose a task specification language for RL based on logical formulas that evaluate whether trajectories satisfy the task , similar in spirit to the logical task compositions framework developed by Tasse et al . [ 57 ] . Many of these notions of task are more general than those we consider . A natural direction for future work broadens our analysis to include these kinds of task .

Summary: This paper studies the ability of Markovian reward functions to represent tasks defined as a set of acceptable policies, partial ordering over policies, partial ordering over trajectories. It first provides a negative result showing that some tasks cannot be specified through a Markovian reward in some environments. Then, it designs a set of linear programs to simultaneously check if a specific task can be encoded into a Markovian reward and to compute such reward. The paper includes an empirical evaluation in a simple domain.