Orchestrated Value Mapping for Reinforcement Learning

1 INTRODUCTION . The chief goal of reinforcement learning ( RL ) algorithms is to maximize the expected return — the value function — from each state ( Sutton & Barto , 2018 ; Szepesvári , 2010 ) . For decades , many algorithms have been proposed to compute the value function either as their main goal ( critic-only algorithms ) or as the means to help the policy search process ( actor-critic algorithms ) . However , when the environment comprises certain properties , learning the induced value function may become very challenging ( van Seijen et al. , 2019 ) . Examples include environments where rewards are dense in some parts of the state-space but very sparse in other parts , or when the size of rewards may vary dramatically . In the Atari 2600 game of Ms. Pac-Man , for instance , the reward can vary from 10 for small pellets to as large as 5000 for moving bananas . In other games like Tennis , as the agent acts randomly , it easily loses the game , causing negative rewards with high frequency . However , when the agent learns to capture the ball ( hence no negative reward anymore ) , it may still take a very long time before the agent gains a score and see a positive reward . Such learning scenarios , for one reason or another , have proved challenging for the conventional RL algorithms . One issue that may rise due to these environmental challenges is for the action gap — the optimal value difference between the best and second-best actions ( Farahmand , 2011 ) — to become significantly non-uniform . A recent study ( van Seijen et al. , 2019 ) showed promising results by simply mapping the value function to a logarithmic space and add important algorithmic steps to guarantee the convergence under generic conditions . While this construction addresses the problem of nonuniform action-gap and facilitates the use of smaller discount factors , it further opens a new direction to improve the learning : learn the value function in a different space that admits better properties compared to the original space . This interesting view naturally raises theoretical questions about the required properties of the functions that map the value to the learning space , and whether the algorithm still guarantees to converge to the optimal value function under this new construction . A loosely related topic is to presume a nonlinear Bellman equation . Originally , the Bellman equation of optimality ( Bellman , 1954 ; 1957 ) is a direct result of defining return as a discounted sum of future rewards . However , one may ignore this basis , and re-define Bellman equation in a generic non-linear way . In particular , van Hasselt et al . ( 2019 ) have shown that the new Bellman operator is still a contraction mapping and therefore the resulting algorithms may be well-behaved . The application of such algorithms is still unclear since the fixed-point does not have direct connection to the concept of return . In this paper , we do not consider nonlinear Bellman equations . Continuing with the first line of thought , a natural extension is to use different mappings concurrently as an ensemble , each of which adding different benefits , just like having various expertise in the board that runs a company . This may also be seen as a way of separation of concerns ( van Seijen et al. , 2016 ) . Ideally , we may wish to dynamically modify the influence of different mappings as the learning advances . For example , the agent may start with mappings that facilitate learning from sparse rewards , then as the agent learns to collect more rewards and they get denser , it gradually switches to mappings that simplifies learning on dense rewards . Of note , if there are known sources of rewards with known , specific characteristics ( say sparse positive rewards , but dense negative ones ) , same approach may prove beneficial . This has been the backbone for reward channeling architectures such as ( van Seijen et al. , 2017 ) . In this paper , building upon the aforementioned ideas , we present two orthogonal directions : generic value mapping and reward decomposition . We then combine the two and build a broad algorithm class , which can be instantiated to generate various interesting agents . We present the convergence results for the complete version of the algorithm . It is worth noting that this algorithm may be best seen as a blueprint for constructing other convergent algorithm classes . 2 VALUE MAPPING . We consider a general reinforcement learning problem , modeled as a Markov Decision Process ( MDP ) ( Puterman , 1994 ) M = ( S , A , P , r , γ ) , where S and A are the discrete sets of states and actions ; P : S ×A× S → [ 0 , 1 ] is a transition function that defines the transition probability from state s to s′ if action a is taken ; r : S × A × S → [ rmin , rmax ] is a reward function and γ ∈ [ 0 , 1 ] denotes a scalar discount factor . A policy π ( s , a ) = P [ At = a | St = s ] defines how an action is selected in a given state . Therefore , selecting actions according to a stationary policy in general results in a stochastic trajectory . The discounted sum of rewards over the trajectory induces a random variable , called return . We assume that all returns are finite and bounded . A trajectory is therefore called optimal if it induces maximum return . A state-action value function Qπ ( s , a ) is defined in conjunction with a policy π. Qπ ( s , a ) induces the expected return of taking action a at state s and following π thereafter . The optimal value function is defined as Q∗ ( s , a ) = maxπ Qπ ( s , a ) , which is the maximum expected return of all trajectories starting from ( s , a ) . Similarly , an optimal policy is defined as π∗ ( s , a ) = arg maxπ Qπ ( s , a ) . The optimal value function is unique ( Bertsekas & Tsitsiklis , 1996 ) and can be found for example as the fixed-point of the Q-Learning algorithm ( Watkins , 1989 ; Watkins & Dayan , 1992 ) , given by the following update rule : Qt+1 ( st , at ) ←− ( 1− αt ) Qt ( s , a ) + αt ( r ( st , at , st ) ) + γmax a′ Qt ( st+1 , a ′ ) ) ( 1 ) where αt is the learning rate ( step size ) parameter defined at time t. Our goal is to first map Q to a different space , and then perform the update , so that the learning process can benefit from the properties of the mapped space . We define a function f that maps the value function to some new space . In particular , we consider the following assumptions : Assumption 1 The function f ( x ) is a bijection ( either strictly increasing or strictly decreasing ) for all x in the given domain D = [ c1 , c2 ] ⊆ R. Assumption 2 The function f ( x ) holds the following properties for all x in the given domain D = [ c1 , c2 ] ⊆ R : 1. f is continuous on [ c1 , c2 ] and differentiable on ( c1 , c2 ) ; 2 . |f ′ ( x ) | ∈ [ δ1 , δ2 ] for x ∈ ( c1 , c2 ) , with 0 < δ1 < δ2 < ∞ ; 3. f is either of semi-convex or semi-concave . We next use f to map the value function , Q ( s , a ) , to its transformed version , namely Q̃ ( s , a ) = f ( Q ( s , a ) ) ( 2 ) Assumption 1 induces that f ( x ) is invertible , which we denote by f−1 ( x ) . Additionally , having known either of Q ( s , a ) or Q̃ ( s , a ) , the other one is uniquely computable . Of note , this assumption also implies that f preserves ordering ; however , it will change the direction of ordering if f is decreasing . Assumption 2 imposes more restriction on f ; nevertheless , it is still a broad class of functions to consider . Throughout the paper , we use tilde notation to refer to the “ mapped ” function or variable , while the mapping f is understandable from the context ( otherwise it is explicitly said ) . 2.1 BASE ALGORITHM . If the mapped value is naively placed in a Q-Learning-like algorithm , it will not converge to the optimal values in stochastic environments . More formally , for a tabular MDP , an update of the following form : ( compare it with equation 1 ) Q̃t+1 ( st , at ) : = ( 1− α ) Q̃t ( st , at ) + αf ( rt + γmax a′ f−1 ( Q̃t ( st+1 , a ′ ) ) ) ( 3 ) converges1 to the fixed point Q̃ ( s , a ) that satisfies Q̃ ( s , a ) = Es′∼P ( s , a , · ) [ f ( R ( s , a , s′ ) + γmax a′ f−1 ( Q̃ ( s′ , a′ ) ) ) ] , ( 4 ) where s′ denotes the next state . Let us define the notation Q ( s , a ) .= f−1 ( Q̃ ( s , a ) ) . If f is a semi-convex bijection , f−1 will be semi-concave and equation 4 deduces Q ( s , a ) . = f−1 ( Q̃ ( s , a ) ) = f−1 ( Es′∼P ( s , a , · ) [ f ( R ( s , a , s′ ) + γmax a′ Q ( s′ , a′ ) ) ] ) ≥ Es′∼P ( s , a , · ) [ f−1 ( f ( R ( s , a , s′ ) + γmax a′ Q ( s′ , a′ ) ) ) ] = Es′∼P ( s , a , · ) [ R ( s , a , s′ ) + γmax a′ Q ( s′ , a′ ) ] , ( 5 ) where the third line follows Jensen ’ s inequality . Comparing ( 5 ) with the Bellman ’ s optimality equation in the regular space , i.e . Q∗ ( s , a ) = Es′∼P ( s , a , · ) [ R ( s , a , s′ ) + γmaxa′ Q∗ ( s′ , a′ ) ] ; we conclude that the value function to which ( 3 ) converges over-estimates Bellman ’ s backup . Similarly , if f is a semi-concave function , then Q ( s , a ) under-estimates Bellman ’ s backup . Either way , it follows that the learned value function deviates from the optimal one . Further , the Jensen ’ s gap at a given state s — the difference between left- and right-hand-side of ( 5 ) — depends on the action a because the expectation operator depends on a . That is , at a given state s , the deviation of Q ( s , a ) from Q∗ ( s , a ) is not a fixed-value shift and can vary for various actions . Hence , the greedy policy w.r.t . Q ( s , · ) may not preserve ordering and it may not be an optimal policy either . In an effort to address this problem in the spacial case of f being a logarithmic function , van Seijen et al . ( 2019 ) observed that in the algorithm described by equation 3 , the step-size α generally conflates two forms of averaging : ( i ) averaging of stochastic update targets due to environment stochasticity ( happens in the regular space ) , and ( ii ) averaging over different states/actions ( happens in the f space ) . Moreover , they proposed to algorithmically separate the two and showed that such a separation will lift the Jensen gap between Q ( s , a ) and Q∗ ( s , a ) if the learning rate of the regular space decays to zero fast enough . Building from Log Q-Learning ( van Seijen et al. , 2019 ) , we define the base algorithm as the following : at time t it receives Q̃t ( s , a ) and a transition tuple ( s , a , r , s′ ) as defined previously , and outputs Q̃t+1 ( s , a ) , which then yields Qt+1 ( s , a ) = f−1 ( Q̃t+1 ( s , a ) ) . The steps are listed below : 1The convergence follows from stochastic approximation theory with the additional steps to show by induction that Q̃ remains bounded and then the corresponding operator is a contraction mapping . Qt ( s , a ) : = f −1 ( Q̃t ( s , a ) ) ( 6 ) ã′ : = arg max a′ ( Qt ( s ′ , a′ ) ) ( 7 ) Ut : = r + γf −1 ( Q̃t ( s ′ , ã′ ) ) ( 8 ) Ût : = f −1 ( Q̃t ( s , a ) ) + βreg , t ( Ut − f−1 ( Q̃t ( s , a ) ) ) ( 9 ) Q̃t+1 ( s , a ) : = Q̃t ( s , a ) + βf , t ( f ( Ût ) − Q̃t ( s , a ) ) ( 10 ) Here , the mapping f is any function that satisfies Assumptions 1 and 2 . Remark that similarly to the Log Q-Learning algorithm , ( 9 ) and ( 10 ) have decoupled averaging of stochastic update targets from that of different state-actions .

Summary: The paper proposes a framework for estimating the Q-value function by decomposing the reward into a linear combination of reward signals or channels. These individual channels are then mapped into a new space, similarly to the log Q-learning approach of van Seijen et al (2019). However, here the framework is extended to a more general class of function (convex, etc). Additionally, some of the theoretical assumptions in the original log Q-learning paper are softened. The method (with a particular instantiation of reward mapping) is tested on the Atari benchmark, and demonstrates improved performance compared to DQN and log Q-learning.