On the Expressivity of Markov Reward

1 Introduction . How are we to use algorithms for reinforcement learning ( RL ) to solve problems of relevance in the world ? Reward plays a significant role as a general purpose signal : For any desired behavior , task , or other characteristic of agency , there must exist a reward signal that can incentivize an agent to learn to realize these desires . Indeed , the expressivity of reward is taken as a backdrop assumption that frames RL , sometimes called the reward hypothesis : “ ... all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal ( reward ) ” [ 53 , 29 , 6 ] . In this paper , we establish first steps toward a systematic study of the reward hypothesis by examining the expressivity of reward as a signal . We proceed in three steps . 1 . An Account of “ Task ” . As rewards encode tasks , goals , or desires , we first ask , “ what is a task ? ” . We frame our study around a thought experiment ( Figure 1 ) involving the interactions between a designer , Alice , and a learning agent , Bob , drawing inspiration from Ackley and Littman [ 2 ] , Sorg [ 50 ] , and Singh et al . [ 46 ] . In this thought experiment , we draw a distinction between how Alice thinks of a task ( TASKQ ) and the means by which Alice incentivizes Bob to pursue this task ( EXPRESSIONQ ) . This distinction allows us to analyze the expressivity of reward as an answer to the latter question , conditioned on how we answer the former . Concretely , we study three answers to the TASKQ in the context of finite Markov Decision Processes ( MDPs ) : A task is either ( 1 ) a set of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . acceptable behaviors ( policies ) , ( 2 ) a partial ordering over behaviors , or ( 3 ) a partial ordering over trajectories . Further detail and motivation for these task types is provided in Section 3 , but broadly they can be viewed as generalizations of typical notions of task such as a choice of goal or optimal behavior . Given these three answers to the TASKQ , we then examine the expressivity of reward . 2 . Expressivity of Markov Reward . The core of our study asks whether there are tasks Alice would like to convey—as captured by the answers to the TASKQ—that admit no characterization in terms of a Markov reward function . Our emphasis on Markov reward functions , as opposed to arbitrary history-based reward functions , is motivated by several factors . First , disciplines such as computer science , psychology , biology , and economics typically rely on a notion of reward as a numerical proxy for the immediate worth of states of affairs ( such as the financial cost of buying a solar panel or the fitness benefits of a phenotype ) . Given an appropriate way to describe states of affairs , Markov reward functions can represent immediate worth in an intuitive manner that also allows for reasoning about combinations , sequences , or re-occurrences of such states of affairs . Second , it is not clear that general history-based rewards are a reasonable target for learning as they suffer from the curse of dimensionality in the length of the history . Lastly , Markov reward functions are the standard in RL . A rigorous analysis of which tasks they can and can not convey may provide guidance into when it is necessary to draw on alternative formulations of a problem . Given our focus on Markov rewards , we treat a reward function as accurately expressing a task just when the value function it induces in an environment adheres to the constraints of a given task . 3 . Main Results . We find that , for all three task types , there are environment–task pairs for which there is no Markov reward function that realizes the task ( Theorem 4.1 ) . In light of this finding , we design polynomial-time algorithms that can determine , for any given task and environment , whether a reward function exists in the environment that captures the task ( Theorem 4.3 ) . When such a reward function does exist , the algorithms also return it . Finally , we conduct simple experiments with these procedures to provide empirical insight into the expressivity of reward ( Section 5 ) . Collectively , our results demonstrate that there are tasks that can not be expressed by Markov reward in a rigorous sense , but we can efficiently construct such reward functions when they do exist ( and determine when they do not ) . We take these findings to shed light on the nature of reward maximization as a principle , and highlight many pathways for further investigation . 2 Background . RL defines the problem facing an agent that learns to improve its behavior over time by interacting with its environment . We make the typical assumption that the RL problem is well modeled by an agent interacting with a finite Markov Decision Process ( MDP ) , defined by the tuple ( S , A , R , T , γ , s0 ) . An MDP gives rise to deterministic behavioral policies , π : S → A , and the value , V π : S → R , and action–value , Qπ : S ×A → R , functions that measure their quality . We will refer to a Controlled Markov Process ( CMP ) as an MDP without a reward function , which we denote E for environment . We assume that all reward functions are deterministic , and may be a function of either state , stateaction pairs , or state-action-state triples , but not history . Henceforth , we simply use “ reward function ” to refer to a deterministic Markov reward function for brevity , but note that more sophisticated settings beyond MDPs and deterministic Markov reward functions are important directions for future work . For more on MDPs or RL , see the books by Puterman [ 41 ] and Sutton and Barto [ 54 ] respectively . 2.1 Other Perspectives on Reward . We here briefly summarize relevant literature that provides distinct perspectives on reward . Two Roles of Reward . As Sorg [ 50 ] identifies ( Chapter 2 ) , reward can both define the task the agent learns to solve , and define the “ bread crumbs ” that allow agents to efficiently learn to solve the task . This distinction has been raised elsewhere [ 2 , 46 , 47 ] , and is similar to the extrinsic-intrinsic reward divide [ 45 , 66 ] . Tools such as reward design [ 34 , 51 ] or reward shaping [ 36 ] focus on offering more efficient learning in a variety of environments , so as to avoid issues of sparsity and long-term credit assignment . We concentrate primarily on reward ’ s capacity to express a task , and defer learning dynamics to an ( important ) stage of future work . Discounts , Expectations , and Rationality . Another important facet of reward is how it is used in producing behavior . The classical view offered by the Bellman equation ( and the reward hypothesis ) is that the quantity of interest to maximize is expected , discounted , cumulative reward . Yet it is possible to disentangle reward from the expectation [ 5 ] , to attend only to ordinal [ 60 ] or maximal rewards [ 26 ] , or to adopt different forms of discounting [ 61 , 11 ] . In this work , we take the standard view that agents will seek to maximize value for a particular discount factor γ , but recognize that there are interesting directions beyond these commitments , such as inspecting the limits of reward in constrained MDPs as studied by Szepesvári [ 56 ] . We also note the particular importance of work by Pitis [ 40 ] , who examines the relationship between classical decision theory [ 59 ] and MDPs by incorporating additional axioms that account for stochastic processes with discounting [ 24 , 35 , 48 , 49 ] . Drawing inspiration from Pitis [ 40 ] and Sunehag and Hutter [ 52 ] , we foresee valuable pathways for future work that further makes contact between RL and various axioms of rationality . Preferences . In place of numerical rewards , preferences of different kinds may be used to evaluate an agent ’ s behaviors , drawing from the literature on preference-learning [ 25 ] and ordinal dynamic programming [ 8 , 35 , 48 ] . This premise gives rise to preference-based reinforcement learning ( PbRL ) in which an agent interacts with a CMP and receives evaluative signals in the form of preferences over states , actions , or trajectories . This kind of feedback inspires and closely parallels the task types we propose in this work . A comprehensive survey of PbRL by Wirth et al . [ 64 ] identifies critical differences in this setup from traditional RL , categorizes recent algorithmic approaches , and highlights important open questions . Recent work focuses on analysing the sample efficiency of such methods [ 65 , 38 ] with close connections to learning from human feedback in real time [ 23 , 32 , 7 ] . Teaching and Inverse RL . The inverse RL ( IRL ) and apprenticeship learning literature examine the problem of learning directly from behavior [ 37 , 1 ] . The classical problem of IRL is to identify which reward function ( often up to an equivalence class ) a given demonstrator is optimizing . We emphasize the relevance of two approaches : First , work by Syed et al . [ 55 ] , who first illustrate the applicability of linear programming [ 22 ] to apprenticeship learning ; and second , work by Amin et al . [ 4 ] , who examine the repeated form of IRL . The methods of IRL have recently been expanded to include variations of cooperative IRL [ 14 ] , and assistive learning [ 43 ] , which offer different perspectives on how to frame interactive learning problems . Reward Misspecification . Reward is also notoriously hard to specify . As pointed out by Littman et al . [ 30 ] , “ putting a meaningful dollar figure on scuffing a wall or dropping a clean fork is challenging. ” Along these lines , Hadfield-Menell et al . [ 16 ] identify cases in which well-intentioned designers create reward functions that produce unintended behavior [ 39 ] . MacGlashan et al . [ 33 ] find that human-provided rewards tend to depend on a learning agent ’ s entire policy , rather than just the current state . Further , work by Hadfield-Menell et al . [ 15 ] and Kumar et al . [ 27 ] suggest that there are problems with reward as a learning mechanism due to misspecification and reward tampering [ 10 ] . These problems have given rise to approaches to reward learning , in which a reward function is inferred from some evidence such as behavior or comparisons thereof [ 20 ] . Other Notions of Task . As a final note , we highlight alternative approaches to task specification . Building on the Free Energy Principle [ 13 , 12 ] , Hafner et al . [ 17 ] consider a variety of task types in terms of minimization of distance to a desired target distribution [ 3 ] . Alternatively , Littman et al . [ 30 ] and Li et al . [ 28 ] propose variations of linear temporal logic ( LTL ) as a mechanism for specifying a task to RL agents , with related literature extending LTL to the multi-task [ 58 ] and multi-agent [ 18 ] settings , or using reward machines for capturing task structure [ 19 ] . Jothimurugan et al . [ 21 ] take a similar approach and propose a task specification language for RL based on logical formulas that evaluate whether trajectories satisfy the task , similar in spirit to the logical task compositions framework developed by Tasse et al . [ 57 ] . Many of these notions of task are more general than those we consider . A natural direction for future work broadens our analysis to include these kinds of task .

Summary: - An exploration and a concrete answer to the question of what is a task, and a critical examination of the reward hypothesis. The approach that this paper proposes is to define a set of all possible tasks in a Controlled Markov Process (CMP), and then ask the question, does there exist a Markov reward that realizes the given task.  - The paper provides answers the above question in the negative through a counter-example. In summary, the authors provide three modes of expressing the task intent: set of accetable policies (SOAP), partial ordering over policies (PO), partial ordering over trajectories (TO). The key result is that there are tasks defined as per all three methods that are not realizable as a Markov reward optimization problem  - The authors finally propose a linear-programming based polynomial time reward design algorithm that generates a Markov reward for the SOAP task specification and returns Null if such a reward does not exist.