Bi-linear Value Networks for Multi-goal Reinforcement Learning

1 INTRODUCTION ( a ) Move towards goals ( b ) Bilinear Decomposition Figure 1 : ( a ) The value towards a goal depends on both the current state s and the goal g which is far away . ( b ) we can capture this relationship by decomposing the value function into a bi-linear product between a local vector field ~f ( s , a ) and a global vector field , ~φ ( s , g ) . Gray arrows indicate the vector field ~φ ( s , g ) evaluated at all states . The black arrow indicates the best alignment with ~f ( s , a ) . Constructing agents capable of accomplishing a diverse set of goals is a long standing challenge in reinforcement learning ( Kaelbling , 1993 ) . A major issue is the high sample complexity resulting from interference caused by learning a single policy or value function for a multitude of goals . This issue can be partially mitigated by making efficient reuse of past interaction data , such as employing off-policy learning algorithms , along with hindsight experience re-labeling ( HER , see Andrychowicz et al . 2017 ) . A complementary approach improves data efficiency by encoding inductive biases in the neural network , such as the graph structure ( Li et al. , 2019 ) . Such a structure enables sharing information across goals and thereby mitigates interference and results in better generalization . In multi-goal reinforcement learning , a single , goal-conditioned universal value function ( UVFA Schaul et al . 2015 ) approximates the long-term return for a variety of goals . One possibility is to model the UVFA as a monolithic neural network , Qθ ( s , a , g ) , where state s , action a and goal g ( or the embedding of which ) are simply concatenated together as input . In such form , there is no explicit inductive bias for the network to discover shared structures between goals that could otherwise greatly improve the performance when the agent is tasked with a new goal ( Sutton et al. , 2011 ) . This ability to transfer to new goals when learning online , as the ability for the policy to perform given limited data has a significant compounding effect through the active collection of interactive data . Separating the local knowledge of the dynamics , and the global topology of the domain is a natural way to disentangle the problem . Inspired by this dichotomy , we decompose goal-conditioned value approximation into two disentangled concerns “ where to go ” and “ what happens next ” . “ Where to go ” models how should the state be changed to get closer to the goal , while “ what I can do ” captures where the agent can move at the current state . The knowledge of “ what I can do ” is independent of “ where to go ” and thus can be re-used across new goals . We illustrate this intuition in figure 1a . As an analogy to “ what I can do ” , the red arrow denotes the succeeding state after taking an action at state s , which is goal-agnostic . Independent of action , the blue arrow represents “ where to go ” that can be viewed as a direction to the goal g from the current state s. Adhering to this intuition , we thus treat value predictions as the interactions between two vectors . Consider “ what I can do ” and “ where to go ” as two vector fields ~f ( s , a ) and ~ϕ ( s , g ) in figure 1b . It is natural to define the value predictions as the level of alignment between two vector field ~f and ~ϕ . The maximum value implicitly emerges when “ what I can do ” ( ~f ( s , a ) ) corresponding to taking an action a aligns “ where to go ” ( ~ϕ ( s , g ) ) . In fact , our intuition resembles bilinear mapping in vector analysis , that parametrizes a scalar function by two vector spaces . Our principal contribution is bi-linear value network ( BVN ) , a parameterization scheme that decomposes the goal-conditioned value function into a bilinear map between two distinct vector representations . The first component φ ( s , g ) captures the long-range topology between the current state and a distant goal , whereas the second component f ( s , a ) is akin to a local model that represents taking an immediate action at the current state . To understand how this parameterization affects learning dynamics , we evaluate the agent on goals that never appeared during training . Surprisingly , we found that the bi-linear decomposition allows the critic to generalize to unseen goals ( see section 4.3 ) , which in turn improves the overall sample complexity and policy performance . We evaluate our approach on the challenging task of dexterous manipulation with a simulated shadow hand and other object manipulation tasks . Our method , BVN , achieves state-of-the-art sample efficiency and asymptotic performance on the majority of tasks . Apart from comparing with prior works , we also produce detailed analysis on the influence of the dimensionality of the two latent spaces , and how different constructions of the binary mapping affects performance . 2 PRELIMINARIES . 2.1 MULTI-GOAL REINFORCEMENT LEARNING . We consider a special case of the Markov decision process ( MDP ) , the multi-goal reinforcement learning ( RL ) problem ( Kaelbling , 1993 ) . At the beginning of an episode , an initial state s0 and a behavior goal gb is sampled according to the distribution ρS and ρG . The behavior goal gb is fixed throughout the episode during roll-out by the policy π . For each time step t , the agent takes action at = π ( st , gb ) , receives reward rt , gb = R ( st , at , gb ) , and moves to the next state st+1 . We consider an infinite horizon MDP problem with the discount factor γ . The goal of multi-goal RL is to produce a policy π∗ that maximizes the expected return J π∗ = argmax π J ( π ) where J ( π ) = E [ ∞∑ τ=t γτ−trτ , gb |st = s , g = gb ] , ( 1 ) where the discount factor γ is set as 0.98 in common multi-goal RL benchmarks ( Plappert et al. , 2018 ) , and the reward rτ , gb is typically defined as R ( sτ , aτ , gb ) : = { 0 , goal reached −1 , otherwise ( 2 ) Note that our method , BVN , is not dependent on this reward structure . 2.2 OFF-POLICY RL WITH ACTOR-CRITIC METHODS . As we consider multi-goal continuous control tasks , we follow Andrychowicz et al . ( 2017 ) and Plappert et al . ( 2018 ) and use deterministic policy gradient algorithm ( DDPG ) ( Silver et al. , 2014 ) to learn a π∗ . DDPG is an actor-critic method . Note that a more advanced actor-critic method ( Fujimoto et al. , 2018 ) can be used , but we empirically found that DDPG is better than that in the tasks we consider . In actor-critic methods the value function , referred to as the critic , is responsible for generating the learning signal of π . The critic is trained by minimizing the loss function L ( Q ) = E ( st , at ) ∼Uniform ( Z ) ( rt , g + γQ ( st+1 , π ( st+1 , g ) , g ) −Q ( st , at , g ) ) 2 , ( 3 ) where Z denotes a replay buffer ( Mnih et al. , 2015 ) that stores multiple state transitions tuples ( st , at , rt , st+1 , gb ) . The actor π is trained to maximize the critic outputs by the gradient ∇πJ ( π ) = E [ ∇aQπ ( s , a , g ) |a=π ( s , g ) ] . ( 4 ) In multi-goal RL settings , the value function Q is often referred as universal value function approximator ( Schaul et al. , 2015 ) since Q encompasses the values of all goals in the MDP . As we are particularly interested in sample efficiency of policy learning , we highlight the importance of the critic for the sample efficiency of training the actor . A critic that provides better estimate of values for previously unseen states ( i.e. , better generalization of the critic ) will lead to faster policy improvement via optimization in Equation 4 . 3 BI-LINEAR VALUE DECOMPOSITION FOR MULTI-GOAL RL . The key insight behind the bilinear value network architecture , as illustrated in figure 2 , is that we can often decompose value estimate into the alignment between the concern of “ what I can do ” and “ where I want to go. ” When these two concerns are implicitly aligned , we consider the action to be good for reaching that goal . Whereas if these two are misaligned , the action is considered bad . These two concerns interact with each other when we try to find the solution to what I can do , but they are also disentangled because they take in different inputs . “ What I can do , ” for example , only concerns the current state of affair and the possible set of actions . When a new goal appears the agent would not have to reconsider the relationship between its current state and the available actions , therefore lowers the burden to learn . It does however need to update what it knows about getting to that goal from where it is , which is inescapable . To bring this insight into reality , we design a drop-in replacement of the standard universal value function approximator , as illustrated in figure 3 , which we refer to as the bilinear value network . For context we present the standard architecture of a universal value function approximator ( UVFA ) on the left ( see figure 3a ) . The UVFA is a single , monolithic neural network which is entangled in the way it processes the inputs . It first concatenates the state , action and the goal together into a single vector , and does not impose any inductive bias into the way these inputs interact . With bilinear value networks ( see figure 3b ) , we partition the three input vectors into two overlapping groups . In the first group that we consider to be local , the neural network ~f ( s , a ) is only concerned with the current state and the immediate actions the agent can take . This is the component that is relatively easier to learn , because it does not have to integrate over long horizons . The second group is more global . The neural network ~φ ( s , g ) captures the long-horizon relationship between the current state and the goal which can be quite far apart . To bring the information that are processed separately by these two components , we borrow the concept of bilinear map from vector analysis , and define the Q function as a dot product between the two distinct vector representations Q : Zf × Zφ 7→ R. ( 5 ) A consequence of this treatment is that the value becomes the result of an alignment between the two vector fields : Q ( s , a , g ) = ~f ( s , a ) > ~φ ( s , g ) . ( 6 ) In figure 2 , we show that passing the current state s into the global stream φ ( s , g ) is important , because to find the solution to the optimal action , the global field needs to vary depending on the current state . Figure 2a shows a goal stream that does not take into the current state – the vector field becomes a constant ∀s ∈ S , making it impossible to accommodate the complex behavior needed to succeed from the entire state space . An important issue to keep in mind is that Equation 6 is unidentifiable in the statistical sense because we can insert an arbitrary vector λ into the vector space and produce the same value estimate , as long as λ contains no zeros . In statistics , identifiability is the condition under which we can recover the true parameter of the model given infinite samples . Q ( s , a , g ) = f ( s , a ) > φ ( g ) ⇐⇒ Q ( s , a , g ) = ( λf ) > ( λ−1φ ) ∀ λ ∈ Rn . ( 7 ) What this means is that under end-to-end training , the particular semantics of the two vector fields becomes particularly irrelevant . It is the grouping of the input variables and the bilinear structure that is critical . It is also important to note that the bilinear value networks is implemented as a drop-in replacement of the universal value function approximator , as opposed to any additional algorithmic step . The training procedure we adopt in this paper follows those of deep deterministic policy gradient and hindsight experience replay , but the bilinear value networks is equally applicable to other off-policy learning algorithms under goal-conditioned scenario that uses either a value function , or a critic .

Summary: The paper proposes a modification to goal-oriented universal value functions that split the neural network into two parts. One network accepts states and actions and outputs a vector representation, while the other outputs a vector given a state and goal. Part of the contribution of the paper is the interpretation of these two vectors --- the first captures the notion of what actions can be taken at a given state, while the second represents the direction the agent should move towards to achieve the goal from its current state.  Empirical results on several simulated robotic domains demonstrate that the method outperforms other UVFA architectures, while ablations are provided to capture the most important aspects of the method.