Bi-linear Value Networks for Multi-goal Reinforcement Learning

1 INTRODUCTION ( a ) Move towards goals ( b ) Bilinear Decomposition Figure 1 : ( a ) The value towards a goal depends on both the current state s and the goal g which is far away . ( b ) we can capture this relationship by decomposing the value function into a bi-linear product between a local vector field ~f ( s , a ) and a global vector field , ~φ ( s , g ) . Gray arrows indicate the vector field ~φ ( s , g ) evaluated at all states . The black arrow indicates the best alignment with ~f ( s , a ) . Constructing agents capable of accomplishing a diverse set of goals is a long standing challenge in reinforcement learning ( Kaelbling , 1993 ) . A major issue is the high sample complexity resulting from interference caused by learning a single policy or value function for a multitude of goals . This issue can be partially mitigated by making efficient reuse of past interaction data , such as employing off-policy learning algorithms , along with hindsight experience re-labeling ( HER , see Andrychowicz et al . 2017 ) . A complementary approach improves data efficiency by encoding inductive biases in the neural network , such as the graph structure ( Li et al. , 2019 ) . Such a structure enables sharing information across goals and thereby mitigates interference and results in better generalization . In multi-goal reinforcement learning , a single , goal-conditioned universal value function ( UVFA Schaul et al . 2015 ) approximates the long-term return for a variety of goals . One possibility is to model the UVFA as a monolithic neural network , Qθ ( s , a , g ) , where state s , action a and goal g ( or the embedding of which ) are simply concatenated together as input . In such form , there is no explicit inductive bias for the network to discover shared structures between goals that could otherwise greatly improve the performance when the agent is tasked with a new goal ( Sutton et al. , 2011 ) . This ability to transfer to new goals when learning online , as the ability for the policy to perform given limited data has a significant compounding effect through the active collection of interactive data . Separating the local knowledge of the dynamics , and the global topology of the domain is a natural way to disentangle the problem . Inspired by this dichotomy , we decompose goal-conditioned value approximation into two disentangled concerns “ where to go ” and “ what happens next ” . “ Where to go ” models how should the state be changed to get closer to the goal , while “ what I can do ” captures where the agent can move at the current state . The knowledge of “ what I can do ” is independent of “ where to go ” and thus can be re-used across new goals . We illustrate this intuition in figure 1a . As an analogy to “ what I can do ” , the red arrow denotes the succeeding state after taking an action at state s , which is goal-agnostic . Independent of action , the blue arrow represents “ where to go ” that can be viewed as a direction to the goal g from the current state s. Adhering to this intuition , we thus treat value predictions as the interactions between two vectors . Consider “ what I can do ” and “ where to go ” as two vector fields ~f ( s , a ) and ~ϕ ( s , g ) in figure 1b . It is natural to define the value predictions as the level of alignment between two vector field ~f and ~ϕ . The maximum value implicitly emerges when “ what I can do ” ( ~f ( s , a ) ) corresponding to taking an action a aligns “ where to go ” ( ~ϕ ( s , g ) ) . In fact , our intuition resembles bilinear mapping in vector analysis , that parametrizes a scalar function by two vector spaces . Our principal contribution is bi-linear value network ( BVN ) , a parameterization scheme that decomposes the goal-conditioned value function into a bilinear map between two distinct vector representations . The first component φ ( s , g ) captures the long-range topology between the current state and a distant goal , whereas the second component f ( s , a ) is akin to a local model that represents taking an immediate action at the current state . To understand how this parameterization affects learning dynamics , we evaluate the agent on goals that never appeared during training . Surprisingly , we found that the bi-linear decomposition allows the critic to generalize to unseen goals ( see section 4.3 ) , which in turn improves the overall sample complexity and policy performance . We evaluate our approach on the challenging task of dexterous manipulation with a simulated shadow hand and other object manipulation tasks . Our method , BVN , achieves state-of-the-art sample efficiency and asymptotic performance on the majority of tasks . Apart from comparing with prior works , we also produce detailed analysis on the influence of the dimensionality of the two latent spaces , and how different constructions of the binary mapping affects performance . 2 PRELIMINARIES . 2.1 MULTI-GOAL REINFORCEMENT LEARNING . We consider a special case of the Markov decision process ( MDP ) , the multi-goal reinforcement learning ( RL ) problem ( Kaelbling , 1993 ) . At the beginning of an episode , an initial state s0 and a behavior goal gb is sampled according to the distribution ρS and ρG . The behavior goal gb is fixed throughout the episode during roll-out by the policy π . For each time step t , the agent takes action at = π ( st , gb ) , receives reward rt , gb = R ( st , at , gb ) , and moves to the next state st+1 . We consider an infinite horizon MDP problem with the discount factor γ . The goal of multi-goal RL is to produce a policy π∗ that maximizes the expected return J π∗ = argmax π J ( π ) where J ( π ) = E [ ∞∑ τ=t γτ−trτ , gb |st = s , g = gb ] , ( 1 ) where the discount factor γ is set as 0.98 in common multi-goal RL benchmarks ( Plappert et al. , 2018 ) , and the reward rτ , gb is typically defined as R ( sτ , aτ , gb ) : = { 0 , goal reached −1 , otherwise ( 2 ) Note that our method , BVN , is not dependent on this reward structure . 2.2 OFF-POLICY RL WITH ACTOR-CRITIC METHODS . As we consider multi-goal continuous control tasks , we follow Andrychowicz et al . ( 2017 ) and Plappert et al . ( 2018 ) and use deterministic policy gradient algorithm ( DDPG ) ( Silver et al. , 2014 ) to learn a π∗ . DDPG is an actor-critic method . Note that a more advanced actor-critic method ( Fujimoto et al. , 2018 ) can be used , but we empirically found that DDPG is better than that in the tasks we consider . In actor-critic methods the value function , referred to as the critic , is responsible for generating the learning signal of π . The critic is trained by minimizing the loss function L ( Q ) = E ( st , at ) ∼Uniform ( Z ) ( rt , g + γQ ( st+1 , π ( st+1 , g ) , g ) −Q ( st , at , g ) ) 2 , ( 3 ) where Z denotes a replay buffer ( Mnih et al. , 2015 ) that stores multiple state transitions tuples ( st , at , rt , st+1 , gb ) . The actor π is trained to maximize the critic outputs by the gradient ∇πJ ( π ) = E [ ∇aQπ ( s , a , g ) |a=π ( s , g ) ] . ( 4 ) In multi-goal RL settings , the value function Q is often referred as universal value function approximator ( Schaul et al. , 2015 ) since Q encompasses the values of all goals in the MDP . As we are particularly interested in sample efficiency of policy learning , we highlight the importance of the critic for the sample efficiency of training the actor . A critic that provides better estimate of values for previously unseen states ( i.e. , better generalization of the critic ) will lead to faster policy improvement via optimization in Equation 4 . 3 BI-LINEAR VALUE DECOMPOSITION FOR MULTI-GOAL RL . The key insight behind the bilinear value network architecture , as illustrated in figure 2 , is that we can often decompose value estimate into the alignment between the concern of “ what I can do ” and “ where I want to go. ” When these two concerns are implicitly aligned , we consider the action to be good for reaching that goal . Whereas if these two are misaligned , the action is considered bad . These two concerns interact with each other when we try to find the solution to what I can do , but they are also disentangled because they take in different inputs . “ What I can do , ” for example , only concerns the current state of affair and the possible set of actions . When a new goal appears the agent would not have to reconsider the relationship between its current state and the available actions , therefore lowers the burden to learn . It does however need to update what it knows about getting to that goal from where it is , which is inescapable . To bring this insight into reality , we design a drop-in replacement of the standard universal value function approximator , as illustrated in figure 3 , which we refer to as the bilinear value network . For context we present the standard architecture of a universal value function approximator ( UVFA ) on the left ( see figure 3a ) . The UVFA is a single , monolithic neural network which is entangled in the way it processes the inputs . It first concatenates the state , action and the goal together into a single vector , and does not impose any inductive bias into the way these inputs interact . With bilinear value networks ( see figure 3b ) , we partition the three input vectors into two overlapping groups . In the first group that we consider to be local , the neural network ~f ( s , a ) is only concerned with the current state and the immediate actions the agent can take . This is the component that is relatively easier to learn , because it does not have to integrate over long horizons . The second group is more global . The neural network ~φ ( s , g ) captures the long-horizon relationship between the current state and the goal which can be quite far apart . To bring the information that are processed separately by these two components , we borrow the concept of bilinear map from vector analysis , and define the Q function as a dot product between the two distinct vector representations Q : Zf × Zφ 7→ R. ( 5 ) A consequence of this treatment is that the value becomes the result of an alignment between the two vector fields : Q ( s , a , g ) = ~f ( s , a ) > ~φ ( s , g ) . ( 6 ) In figure 2 , we show that passing the current state s into the global stream φ ( s , g ) is important , because to find the solution to the optimal action , the global field needs to vary depending on the current state . Figure 2a shows a goal stream that does not take into the current state – the vector field becomes a constant ∀s ∈ S , making it impossible to accommodate the complex behavior needed to succeed from the entire state space . An important issue to keep in mind is that Equation 6 is unidentifiable in the statistical sense because we can insert an arbitrary vector λ into the vector space and produce the same value estimate , as long as λ contains no zeros . In statistics , identifiability is the condition under which we can recover the true parameter of the model given infinite samples . Q ( s , a , g ) = f ( s , a ) > φ ( g ) ⇐⇒ Q ( s , a , g ) = ( λf ) > ( λ−1φ ) ∀ λ ∈ Rn . ( 7 ) What this means is that under end-to-end training , the particular semantics of the two vector fields becomes particularly irrelevant . It is the grouping of the input variables and the bilinear structure that is critical . It is also important to note that the bilinear value networks is implemented as a drop-in replacement of the universal value function approximator , as opposed to any additional algorithmic step . The training procedure we adopt in this paper follows those of deep deterministic policy gradient and hindsight experience replay , but the bilinear value networks is equally applicable to other off-policy learning algorithms under goal-conditioned scenario that uses either a value function , or a critic .

Summary: This paper considers the problem of learning a goal-conditioned policy effectively. Particularly, it looks at the architecture of the value network used to learn this goal-conditioned policy if using a deterministic policy gradient algorithm. It studies whether separating the goal-conditioned value network into two components, a state-action embedding network and a state-goal embedding network, might lead to faster learning and better generalization of the policy. The hypothesis such a study tests is whether the separation can tease apart what an agent's actions in a particular state accomplish and what the agent needs to accomplish to get to a goal from a given state. This hypothesis is evaluated by conducting experiments on the Fetch and Hand domains and evaluating how quickly the goal-conditioned policy is learned and whether it generalizes to goals in parts of the state space that it was not trained on.