Learning Goal-Conditioned Value Functions with one-step Path rewards rather than Goal-Rewards

1 INTRODUCTION . Many tasks in robotics require the specification of a goal for every trial . For example , a robotic arm can be tasked to move an object to an arbitrary goal position on a table ( Gu et al. , 2017 ) ; a mobile robot can be tasked to navigate to an arbitrary goal landmark on a map ( Zhu et al. , 2017 ) . The adaptation of reinforcement learning to such goal-conditioned tasks where goal locations can change is called Multi-Goal Reinforcement Learning ( MGRL ) ( Plappert et al. , 2018 ) . State-of-theart MGRL algorithms ( Andrychowicz et al. , 2017 ; Pong et al. , 2018 ) work by estimating goalconditioned value functions ( GCVF ) which are defined as expected cumulative rewards from start states with specified goals . GCVFs , in turn , are used to compute policies that determine the actions to take at every state . To learn GCVFs , MGRL algorithms use goal-reward , defined as the relatively higher reward recieved on reaching the desired goal state . This makes the reward function dependent on the desired goal . For example , in the Fetch-Push task ( Plappert et al. , 2018 ) of moving a block to a given location on a table , every movement incurs a “ -1 ” reward while reaching the desired goal returns a “ 0 ” goal-reward . This dependence introduces additional reward resampling steps in algorithms like Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) , where trials in which the agent failed to reach the goal are reused by recomputing rewards as if the reached states were pseudodesired goals . Due to the dependence of the reward function on the goal , the relabelling of every pseudo-goal requires an independent reward-recomputation step , which can be expensive . In this paper , we demonstrate that goal-rewards are not needed to learn GCVFs . For the Fetch-Push example , the “ 0 ” goal-reward does not need to be achieved to learn its GCVF . Specifically , the agent continues to receive “ -1 ” reward even when the block is in the given goal location . This reward formulation is atypical in conventional RL because high reward is used to specify the desired goal location . However , this goal-reward is not necessary in goal-conditioned RL because the goal is already specified at the start of every episode . We use this idea , to propose a goal-conditioned RL algorithm which learns to reach goals without goal-rewards . This is a counter-intuitive result which is important for understanding goal-conditioned RL . Let us consider another example to motivate the redundancy of goal-rewards . Consider a student who has moved to a new campus . To learn about the campus , the student explores it randomly with no specific goal in mind . The key intuition here is that the student is not incentivized to find specific goal locations ( i.e . no goal-rewards ) but is aware of the effort required to travel between points around the university . When tasked with finding a goal classroom , the student can chain together these path efforts to find the least-effort path to the classroom . Based on this intuition of least-effort paths , we redefine GCVFs to be the expected path-reward that is learned for all possible start-goal pairs . We introduce a one-step loss that assumes one-step paths to be the paths of maximum reward between pairs wherein the state and goal are adjacent . Under this interpretation , the Bellman equation chooses and chains together one-step paths to find longer maximum reward paths . Experimentally , we show how this simple reinterpretation , which does not use goal rewards , performs as well as HER while outperforming it in terms of reward computation . We also extend a closely related algorithm , Floyd-Warshall Reinforcement Learning ( FWRL ) ( Kaelbling , 1993 ) ( also called Dynamic Goal Reinforcement learning ) to use parametric function approximators instead of tabular functions . Similar to our re-definition of GCVFs , FWRL learns a goal-conditioned Floyd-Warshall function that represents path-rewards instead of future-rewards . We translate FWRL ’ s compositionality constraints in the space of GCVFs to introduce additional loss terms to the objective . However , these additional loss terms do not show improvement over the baseline . We conjecture that the compositionality constraints are already captured by other loss terms . In summary , the contributions of this work are twofold . Firstly , we reinterpret goal-conditioned value functions as expected path-rewards and introduce one-step loss , thereby removing the dependency of GCVFs on goal-rewards and reward resampling . We showcase our algorithm ’ s improved sample efficiency ( in terms of reward computation ) . We thus extend algorithms like HER to domains where reward recomputation is expensive or infeasible . Secondly , we extend the tabular Floyd-Warshal Reinforcement Learning to use deep neural networks . 2 RELATED WORK . Goal-conditioned tasks in reinforcement learning have been approached in two ways , depending upon whether the algorithm explicitly separates state and goal representations . The first approach is to use vanilla reinforcement learning algorithms that do not explicitly make this separation ( Mirowski et al. , 2016 ; Dosovitskiy & Koltun , 2016 ; Gupta et al. , 2017 ; Parisotto & Salakhutdinov , 2017 ; Mirowski et al. , 2018 ) . These algorithms depend upon neural network architectures to carry the burden of learning the separated representations . The second approach makes this separation explicit via the use of goal-conditioned value functions ( Foster & Dayan , 2002 ; Sutton et al. , 2011 ) . Universal Value Function Appoximators ( Schaul et al. , 2015 ) propose a network architecture and a factorization technique that separately encodes states and goals , taking advantage of correlations in their representations . Temporal Difference Models combine model-free and model-based RL to gain advantages from both realms by defining and learning a horizon-dependent GCVF . All these works require the use of goal-dependent reward functions and define GCVFs as future-rewards instead of path-rewards , contrasting them from our contribution . Unlike our approach , Andrychowicz et al . ( 2017 ) propose Hindsight Experience Replay , a technique for resampling state-goal pairs from failed experiences ; which leads to faster learning in the presence of sparse rewards . In addition to depending on goal rewards , HER also requires the repeated recomputation of the reward function . In contrast , we show how removing goal-rewards removes the need for such recomputations . We utilize HER as a baseline in our work . Kaelbling ( 1993 ) also use the structure of the space of GCVFs to learn . This work employs compositionality constraints in the space of these functions to accelerate learning in a tabular domain . While their definition of GCVFs is similar to ours , the terminal condition is different . We describe this difference in Section 4 . We also extend their tabular formulation to deep neural networks and evaluate it against the baselines . 3 BACKGROUND . A reinforcement learning ( RL ) problem is formalized as a Markov Decision Process ( MDP ) ( Sutton et al. , 1998 ) . A MDP is defined by a five tuple ( S , A , T , R , γ ) , that governs a sequence of stateaction-reward triples [ ( s0 , a0 , r0 ) , . . . , ( sT , aT , rT ) ] . S is the state space , A is the action space , T ( s , a ) : S ×A → S is the system dynamics , R ( s , a ) : S ×A → R is the reward function and γ is the discount factor . In a typical RL problem the transition function T is not given but is known to be static . In RL , the objective is to find a policy π ( s ) : S → A that maximizes the expected cumulative reward over time , Rt = ∑∞ k=t γ k−trk , called the return . The discount factor , γ < 1 , forces the return to be finite for infinite horizons . Reinforcement learning is typically formulated in single-goal contexts . More recently there has been interest in multi-goal problems ( Andrychowicz et al. , 2017 ; Pong et al. , 2018 ; Plappert et al. , 2018 ) , which is the focus of this work . 3.1 DEEP REINFORCEMENT LEARNING . A number of reinforcement learning algorithms use parametric function approximators to estimate the return in the form of an action-value function , Q ( s , a ) : Qπ ( s , a ) = Eπ [ T∑ k=t γk−tR ( sk , ak ) ∣∣∣∣∣st = s , at = a ] , ( 1 ) where T is the episode length . When the policy π is optimal , the Q-function satisfies the Bellman equation ( Bellman , 1954 ) . Q∗ ( st , at ) = { R ( st , at ) + γmaxa∈AQ∗ ( st+1 , a ) if t < T R ( sT , aT ) if t = T . ( 2 ) TheQ∗ ( ) function can be learned using Q-learning algorithm ( Watkins & Dayan , 1992 ) . The optimal policy can be computed from Q∗ greedily , π∗ ( st ) = argmaxa∈AQ∗ ( st , a ) . In Deep Q-Networks ( DQN ) , Mnih et al . ( 2013 ) formulate a loss function based on the Bellman equation to approximate the optimal Q∗-function using a deep neural network , Qm : L ( θQm ) = Eat∼π ( st ; θπm ) [ ( Qm ( st , at ; θQm ) − yt ) 2 ] , ( 3 ) where yt = R ( st , at ) +maxa γQtgt ( st+1 , a ; θQtgt ) , is the target andQtgt is the target network ( Mnih et al. , 2015a ) . The target network is a slower-changing copy of the main network that stabilizes learning . Mnih et al . ( 2015a ) also employ replay buffers ( Lin , 1993 ) that store transitions from episodes . During training , these transitions are sampled out of order to train the networks in an off-policy manner , avoiding correlation in the samples and thus leading to faster learning . In this work , we use an extension of DQN to continuous action spaces called deep deterministic policy-gradients ( DDPG ) ( Lillicrap et al. , 2015 ) . DDPG approximates the policy with a policy network πtgt ( s ; θπ ) that replaces the max operator in yt . The target thus becomes yt = R ( st , at ) + γQtgt ( st+1 , πtgt ( st+1 ; θπ ) ; θQtgt ) and the loss function changes accordingly : L ( θQ , θπ ) = Eat∼π ( st ; θπ ) [ ( Qm ( st , at ; θQ ) − yt ) 2 ] . ( 4 ) 3.2 MULTI-GOAL REINFORCEMENT LEARNING . Multi-Goal Reinforcement Learning ( Plappert et al. , 2018 ) focuses on problems where the desired goal state can change for every episode . State-of-the-art MGRL algorithms learn a goal-conditioned value function ( GCVF ) , Q ( s , a , g ) , which is defined similar to the Q-function ( 5 ) , but with an additional dependence on the desired goal specification g ∈ G : Qπ ( s , a , g ) = Eπ [ T∑ k=t γk−tR ( sk , ak , g ) ∣∣∣∣∣st = s , at = a ] . ( 5 ) The structure of the goal specification , g ∈ G , can be arbitrary . For example , in a robotic arm experiment , possible goal specifications include the desired position of the end-effector and the desired joint angles of the robot . The states and achieved goals are assumed to be an observable part of the Goal-MDP to enable the agent to learn the correspondences between them , [ ( s0 , a0 , g0 , r0 ) , . . . , ( sT , aT , gT , rT ) ] . Consequently , this Goal-MDP is fully governed by the six tuple ( S , A , G , T , R , γ ) . The reward , R ( s , a , g ) : S ×A×G → R , and policy π ( s , g ) : S ×G → A are also in turn conditioned on goal information . Hindsight Experience Replay HER ( Andrychowicz et al. , 2017 ) builds upon this definition of GCVFs ( 5 ) . The main insight of HER is that there is no valuable feedback from the environment when the agent does not reach the goal . This is further exacerbated when goals are sparse in the state-space . HER solves this problem by reusing these failed experiences for learning . It recomputes a reward for each reached state by relabeling them as pseudo-goals . In our experiments , we employ HER ’ s future strategy for pseudo-goal sampling . More specifically , two transitions from the same episode in the replay buffer for times t and t + f are sampled . The achieved goal gt+f is then assumed to be the pseudo-goal . The algorithm generates a new transition for the time step t with the reward re-computed as if gt+f was the desired goal , ( st , at , st+1 , R ( st , at , gt+f ) ) . HER uses this new transition as a sample .

Summary: This paper aims to improve on Hindsight Experience Replay by removing the need to compute rewards for reaching a goal. The idea is to frame goal-reaching as a shortest path problem where all rewards are -1 until the goal is reached, removing the need to compute rewards. While similar ideas were explored in a recent arxiv tech report, this paper claims to build on these ideas with new loss functions. The experimental results do not seem to be any better compared to baselines when measured in terms of data efficiency, but the proposed method requires fewer “reward computations”.