Explicable Reward Design for Reinforcement Learning Agents

1 Introduction . A reward function plays the central role during the learning/training process of a reinforcement learning ( RL ) agent . Given a “ task ” the agent is expected to perform ( i.e. , the desired learning outcome ) , there are typically many different reward specifications under which an optimal policy has the same performance guarantees on the task . This freedom in choosing the reward function , in turn , leads to the fundamental question of reward design : What are different criteria that one should consider in designing a reward function for the agent , apart from the agent ’ s final output policy ? [ 1–3 ] . One of the important criteria is informativeness , capturing that the rewards should speed up the agent ’ s convergence [ 1–6 ] . For instance , a major challenge faced by an RL agent is because of delayed rewards during training ; in the worst-case , the agent ’ s convergence is slowed down exponentially w.r.t . the time horizon of delay [ 7 ] . In this case , we seek to design a new reward function that reduces this time horizon of delay while guaranteeing that any optimal policy induced by the designed function is also optimal under the original reward function [ 3 ] . The classical technique of potential-based reward shaping ( when applied with appropriate state potentials ) indeed allows us to reduce this time horizon of delay to 1 ; see [ 3 , 8 ] and Section 2 . With 1 , it means that globally optimal actions for any state are also myopically optimal , thereby making the agent ’ s learning process trivial . While informativeness is an important criterion , it is not the only criterion to consider when designing rewards for many practical applications . Another natural criterion to consider is sparseness as a proxy for ease of interpretability of the rewards . There are several practical settings where sparseness and interpretability of rewards are important , as discussed next . The first motivating application is when 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rewards are designed for human learners who are learning to perform sequential tasks , for instance , in pedagogical applications such as educational games [ 9 ] , virtual reality-based training simulators [ 10 , 11 ] , and solving open-ended problems ( e.g. , block-based visual programming [ 12 ] ) . In this context , tasks can be challenging for novice learners and a teacher agent can assist these learners by designing explicable rewards associated with these tasks . The second motivating application is when rewards are designed for complex compositional tasks in the robotics domain that involve reward specifications in terms of logic , automata , or subgoals [ 13 , 14 ] —these specifications induce a form of sparsity structure on the underlying reward function . The third motivating application is related to defense against reward-poisoning attacks in RL ( see [ 15–19 ] ) by designing structured and sparse reward functions that are easy to debug/verify . Beyond these practical settings , many naturally occurring reward functions in real-life tasks are inherently sparse and interpretable , further motivating the need to distill these properties in the automated reward design process . The key challenge is that higher informativeness typically requires dense rewards for many learning tasks – for instance , the above-mentioned potential-based shaped rewards that achieve a time horizon of 1 would require most of the states be associated with some real-valued reward ( see Sections 2 and 4 ) . To this end , an important research question that we seek to address is : How to balance these two criteria of informativeness and sparseness in the reward design process while guaranteeing an optimality criterion on policies induced by the reward function ? In this paper , we formalize the problem of designing explicable reward functions , focusing on the criteria of informativeness and sparseness . We investigate this problem from an expert/teacher ’ s point of view who has full domain knowledge ( in this case , an original reward function along with optimal policies induced by the original function ) , and seeks to design a new reward function for the agent— see Figure 1 and further discussion in Section 5 on expert-driven vs. agent-driven reward design . We tackle the problem from the perspective of discrete optimization and introduce a novel framework , EXPRD , to design reward functions . EXPRD allows us to appropriately balance informativeness and sparseness while guaranteeing that an optimal policy induced by the function belongs to a set of target policies . EXPRD builds upon an informativeness criterion that captures the ( sub- ) optimality of target policies at different time horizons from any given starting state . Our main contributions are:1 I . We formulate the problem of explicable reward functions to balance the two important criteria of informativeness and sparseness in the reward design process . ( Sections 2 and 3.1 ) II . We propose a novel optimization framework , EXPRD , to design reward functions . As part of this framework , we introduce a new criterion capturing informativeness of reward functions that is amenable to optimization techniques and is of independent interest . ( Sections 3.2 and 3.3 ) III . We provide a detailed mathematical analysis of EXPRD and show its connections to popular techniques , including potential-based reward shaping . ( Sections 3.3 and 3.4 ) IV . We provide a practical extension to apply our framework to large state spaces . We perform extensive experiments on two navigation tasks to demonstrate the effectiveness of EXPRD in designing explicable reward functions . ( Sections 3.5 and 4 ) 2 Problem Setup . Environment . An environment is defined as a Markov Decision Process ( MDP ) M : = ( S , A , T , γ , R ) , where the set of states and actions are denoted by S and A respectively . T : S ×S ×A → [ 0 , 1 ] captures the state transition dynamics , i.e. , T ( s′ | s , a ) denotes the probability of landing in state s′ by taking action a from state s. Here , γ is the discounting factor . The underlying reward function is given by R : S ×A → [ −Rmax , Rmax ] , for some Rmax > 0 . We interchangeably represent the reward function by a vector R ∈ R|S|·|A| , whose ( s |A|+ a ) -th entry is given by R ( s , a ) . We define the support of R as supp ( R ) : = { s : s ∈ S , R ( s , a ) 6= 0 for some a ∈ A } , and the ` 0-norm of R as ‖R‖0 : = |supp ( R ) | . Preliminaries and definitions . We denote a stochastic policy π : S → ∆ ( A ) as a mapping from a state to a probability distribution over actions , and a deterministic policy π : S → A as a mapping from a state to an action . For any policy π , the state value function V π∞ and the action value function Qπ∞ in the MDPM are defined as follows respectively : V π ∞ ( s ) = E [ ∑∞ t=0 γ tR ( st , at ) |s0 = s , T , π ] and Qπ∞ ( s , a ) = E [ ∑∞ t=0 γ trt|s0 = s , a0 = a , T , π ] . Further , the optimal value functions are given by V ∗∞ ( s ) = supπ V π ∞ ( s ) and Q ∗ ∞ ( s , a ) = supπ Q π ∞ ( s , a ) . There always exists a 1Github repo : https : //github.com/adishs/neurips2021_explicable-reward-design_code . deterministic stationary policy π that achieves the optimal value function simultaneously for all s ∈ S [ 7 , 20 ] , and we denote all such deterministic optimal policies by the set Π∗ : = { π : S → A s.t . V π∞ ( s ) = V ∗∞ ( s ) , ∀s ∈ S } . From here onwards , we focus on deterministic policies unless stated otherwise . For any π and R , we define the following quantities that capture the∞-step ( global ) optimality gap and the 0-step ( myopic ) optimality gap of action a at state s , respectively : δπ∞ ( s , a ) : = Q π ∞ ( s , π ( s ) ) −Qπ∞ ( s , a ) , and δπ0 ( s , a ) : = Qπ0 ( s , π ( s ) ) −Qπ0 ( s , a ) , ∀s ∈ S , a ∈ A , where Qπ0 ( s , a ) = R ( s , a ) is the 0-step action value function of policy π . The δ π ∞ ( s , a ) values are same for all π ∈ Π∗ , and we denote it by δ∗∞ ( s , a ) = V ∗∞ ( s ) −Q∗∞ ( s , a ) ; however , this is not the case with δπ0 ( s , a ) values in general . For any state s ∈ S and a set of policies Π , we define Πs : = { a : a = π ( s ) , π ∈ Π } . Then , we have that δ∗∞ ( s , a ) = 0 , ∀s ∈ S , a ∈ Π∗s . Explicable reward design . Figure 1 presents an illustration of the explicable reward design problem that we formalize below . A task is specified as an MDP M with a given goal-based reward function R where R has non-zero rewards only on goal states G ⊆ S , i.e. , R ( s , a ) = 0 , ∀s ∈ S\G , a ∈ A . Many naturally occurring tasks ( see Section 1 for motivating applications ) are goal-based and challenging for learning an optimal policy when the state space S is very large . In this paper , we study the following explicable reward design problem from an expert/teacher ’ s point of view : Given R and the corresponding optimal policy set Π ∗ w.r.t . R as the input , the teacher designs a new reward function R̂ with criteria of informativeness and sparseness while guaranteeing an invariance requirement ( these properties are formalized in Section 3 ) . Informally , the invariance requirement is that any optimal policy learned using the new reward R̂ belongs to the optimal policy set Π ∗ induced by R.2 Typical techniques for reward design and issues . Given a set of important states ( subgoals ) in the environment , one could design a handcrafted reward function R̂CRAFT by assigning non-zero reward values only to these states . Even though this simple approach produces a reward function with a specified sparsity level , it often fails to satisfy the invariance requirement . In particular , there are some well-known “ reward bugs ” that can arise in this approach and mislead the agent into learning sub-optimal policies ( see [ 2 , 3 ] ) . In the seminal work [ 3 ] , the authors introduced the potential-based reward shaping ( PBRS ) method to alleviate this issue . The reward function produced by the PBRS method with optimal value function V ∗ ∞ under R as the potential function is defined as follows : R̂PBRS ( s , a ) : = R ( s , a ) + γ ∑ s′∈S T ( s′ | s , a ) · V ∗∞ ( s′ ) − V ∗ ∞ ( s ) . ( 1 ) The set of optimal policies Π̂∗ induced by R̂PBRS is exactly equal to the set of optimal policies Π ∗ induced by R since δ̂π∞ ( s , a ) = δ ∗ ∞ ( s , a ) for all π ∈ Π ∗ [ 3 ] . In addition , for any state s ∈ S , globally optimal actions Π ∗ s ⊆ A underR are also myopically optimal under R̂PBRS since δ̂π0 ( s , a ) = δ ∗ ∞ ( s , a ) for all π ∈ Π∗ [ 3 , 8 ] – this leads to a dramatic speed-up in the learning process . However , the potentialbased reward shaping produces dense reward function which is less interpretable ( see Section 4 ) . 2In the rest of the paper , the quantities defined corresponding to R : = R are denoted by an overline , e.g. , the optimal policy set by Π ∗ and the∞-step optimality gaps by δ∗∞ ; the quantities defined corresponding to R : = R̂ are denoted by a widehat , e.g. , the optimal policy set by Π̂∗ .

Summary: This paper proposes an optimization-based framework for designing reward functions that balance sparsity/interpretability with informativeness. A novel optimization framework is proposed to directly learn these new reward signals given an initial reward signal and the corresponding optimal policy set. Three optimization problems are proposed that take into consideration subgoals, as well as learning subgoals on-the-fly given a budget on their number. An informativeness criterion is proposed based on Bellman residuals for optimization from which the new reward can be recovered. An iterative algorithm is proposed for approximately solving these problems, and state abstractions are also used to scale the method to larger state spaces. Theoretical analysis is provided to ensure feasibility of the optimization objective, and a near-optimal bound is provided for the iterative algorithm under reasonable assumptions on the informativeness criterion used. Finally experiments on simple domains are used to visualize the learned rewards in terms of their interpretability and informativeness.