Year: 2021

Abstract:
Universal value functions are used to score the long-term utility of actions to achieve a goal from the current state. In contrast to prior methods that learn a monolithic function to approximate the value, we propose a bi-linear decomposed value function. The first component, akin to a global plan models how the state should be changed to reach the goal. The second component, akin to a local controller selects the optimal action to actualize the desired change in state. We simultaneously learn both the components. Such decomposition enables both the global and local components to make efficient use of interaction data and independently generalize. The consequence is superior overall generalization and performance of our system on a wide range of challenging goal conditioned tasks in comparison to the current state-of-the-art.

text: 1 INTRODUCTION
(a) Move towards goals (b) Bilinear Decomposition
Figure 1: (a) The value towards a goal depends on both the current state s and the goal g which is far away. (b) we can capture this relationship by decomposing the value function into a bi-linear product between a local vector field ~f(s, a) and a global vector field, ~φ(s, g). Gray arrows indicate the vector field ~φ(s, g) evaluated at all states. The black arrow indicates the best alignment with ~f(s, a).
Constructing agents capable of accomplishing a diverse set of goals is a long standing challenge in reinforcement learning (Kaelbling, 1993). A major issue is the high sample complexity resulting from interference caused by learning a single policy or value function for a multitude of goals. This issue can be partially mitigated by making efficient reuse of past interaction data, such as employing off-policy learning algorithms, along with hindsight experience re-labeling (HER, see Andrychowicz et al. 2017). A complementary approach improves data efficiency by encoding inductive biases in the neural network, such as the graph structure (Li et al., 2019). Such a structure enables sharing information across goals and thereby mitigates interference and results in better generalization.
In multi-goal reinforcement learning, a single, goal-conditioned universal value function (UVFA Schaul et al. 2015) approximates
the long-term return for a variety of goals. One possibility is to model the UVFA as a monolithic neural network, Qθ(s, a, g), where state s, action a and goal g (or the embedding of which) are simply concatenated together as input. In such form, there is no explicit inductive bias for the network to discover shared structures between goals that could otherwise greatly improve the performance when the agent is tasked with a new goal (Sutton et al., 2011). This ability to transfer to new goals when learning online, as the ability for the policy to perform given limited data has a significant compounding effect through the active collection of interactive data.
Separating the local knowledge of the dynamics, and the global topology of the domain is a natural way to disentangle the problem. Inspired by this dichotomy, we decompose goal-conditioned value approximation into two disentangled concerns “where to go” and “what happens next”. “Where to
go” models how should the state be changed to get closer to the goal, while “what I can do” captures where the agent can move at the current state. The knowledge of “what I can do” is independent of “where to go” and thus can be re-used across new goals. We illustrate this intuition in figure 1a. As an analogy to “what I can do”, the red arrow denotes the succeeding state after taking an action at state s, which is goal-agnostic. Independent of action, the blue arrow represents “where to go” that can be viewed as a direction to the goal g from the current state s. Adhering to this intuition, we thus treat value predictions as the interactions between two vectors. Consider “what I can do” and “where to go” as two vector fields ~f(s, a) and ~ϕ(s, g) in figure 1b. It is natural to define the value predictions as the level of alignment between two vector field ~f and ~ϕ. The maximum value implicitly emerges when “what I can do” (~f(s, a)) corresponding to taking an action a aligns “where to go” (~ϕ(s, g)). In fact, our intuition resembles bilinear mapping in vector analysis, that parametrizes a scalar function by two vector spaces.
Our principal contribution is bi-linear value network (BVN), a parameterization scheme that decomposes the goal-conditioned value function into a bilinear map between two distinct vector representations. The first component φ(s, g) captures the long-range topology between the current state and a distant goal, whereas the second component f(s, a) is akin to a local model that represents taking an immediate action at the current state. To understand how this parameterization affects learning dynamics, we evaluate the agent on goals that never appeared during training. Surprisingly, we found that the bi-linear decomposition allows the critic to generalize to unseen goals (see section 4.3), which in turn improves the overall sample complexity and policy performance. We evaluate our approach on the challenging task of dexterous manipulation with a simulated shadow hand and other object manipulation tasks. Our method, BVN, achieves state-of-the-art sample efficiency and asymptotic performance on the majority of tasks. Apart from comparing with prior works, we also produce detailed analysis on the influence of the dimensionality of the two latent spaces, and how different constructions of the binary mapping affects performance.

heading: 2 PRELIMINARIES

heading: 2.1 MULTI-GOAL REINFORCEMENT LEARNING
text: We consider a special case of the Markov decision process (MDP), the multi-goal reinforcement learning (RL) problem (Kaelbling, 1993). At the beginning of an episode, an initial state s0 and a behavior goal gb is sampled according to the distribution ρS and ρG . The behavior goal gb is fixed throughout the episode during roll-out by the policy π. For each time step t, the agent takes action at = π(st, gb), receives reward rt,gb = R(st, at, gb), and moves to the next state st+1. We consider an infinite horizon MDP problem with the discount factor γ. The goal of multi-goal RL is to produce a policy π∗ that maximizes the expected return J
π∗ = argmax π J(π) where J(π) = E [ ∞∑ τ=t γτ−trτ,gb |st = s, g = gb ] , (1)
where the discount factor γ is set as 0.98 in common multi-goal RL benchmarks (Plappert et al., 2018), and the reward rτ,gb is typically defined as
R(sτ , aτ , gb) := { 0, goal reached −1, otherwise (2)
Note that our method, BVN, is not dependent on this reward structure.

heading: 2.2 OFF-POLICY RL WITH ACTOR-CRITIC METHODS
text: As we consider multi-goal continuous control tasks, we follow Andrychowicz et al. (2017) and Plappert et al. (2018) and use deterministic policy gradient algorithm (DDPG) (Silver et al., 2014) to learn a π∗. DDPG is an actor-critic method. Note that a more advanced actor-critic method (Fujimoto et al., 2018) can be used, but we empirically found that DDPG is better than that in the tasks we consider. In actor-critic methods the value function, referred to as the critic, is responsible for generating the learning signal of π. The critic is trained by minimizing the loss function
L(Q) = E(st,at)∼Uniform(Z)(rt,g + γQ(st+1, π(st+1, g), g)−Q(st, at, g)) 2, (3)
where Z denotes a replay buffer (Mnih et al., 2015) that stores multiple state transitions tuples (st, at, rt, st+1, gb). The actor π is trained to maximize the critic outputs by the gradient
∇πJ(π) = E[∇aQπ(s, a, g)|a=π(s,g)]. (4)
In multi-goal RL settings, the value function Q is often referred as universal value function approximator (Schaul et al., 2015) since Q encompasses the values of all goals in the MDP. As we are particularly interested in sample efficiency of policy learning, we highlight the importance of the critic for the sample efficiency of training the actor. A critic that provides better estimate of values for previously unseen states (i.e., better generalization of the critic) will lead to faster policy improvement via optimization in Equation 4.

heading: 3 BI-LINEAR VALUE DECOMPOSITION FOR MULTI-GOAL RL
text: The key insight behind the bilinear value network architecture, as illustrated in figure 2, is that we can often decompose value estimate into the alignment between the concern of “what I can do” and “where I want to go.” When these two concerns are implicitly aligned, we consider the action to be good for reaching that goal. Whereas if these two are misaligned, the action is considered bad. These two concerns interact with each other when we try to find the solution to what I can do, but they are also disentangled because they take in different inputs. “What I can do,” for example, only concerns the current state of affair and the possible set of actions. When a new goal appears the agent would not have to reconsider the relationship between its current state and the available actions, therefore lowers the burden to learn. It does however need to update what it knows about getting to that goal from where it is, which is inescapable.
To bring this insight into reality, we design a drop-in replacement of the standard universal value function approximator, as illustrated in figure 3, which we refer to as the bilinear value network. For context we present the standard architecture of a universal value function approximator (UVFA) on the left (see figure 3a). The UVFA is a single, monolithic neural network which is entangled in the way it processes the inputs. It first concatenates the state, action and the goal together into a single vector, and does not impose any inductive bias into the way these inputs interact.
With bilinear value networks (see figure 3b), we partition the three input vectors into two overlapping groups. In the first group that we consider to be local, the neural network ~f(s, a) is only concerned with the current state and the immediate actions the agent can take. This is the component that is relatively
easier to learn, because it does not have to integrate over long horizons. The second group is more global. The neural network ~φ(s, g) captures the long-horizon relationship between the current state and the goal which can be quite far apart.
To bring the information that are processed separately by these two components, we borrow the concept of bilinear map from vector analysis, and define the Q function as a dot product between the two distinct vector representations
Q : Zf × Zφ 7→ R. (5)
A consequence of this treatment is that the value becomes the result of an alignment between the two vector fields:
Q(s, a, g) = ~f(s, a)>~φ(s, g). (6)
In figure 2, we show that passing the current state s into the global stream φ(s, g) is important, because to find the solution to the optimal action, the global field needs to vary depending on the current state. Figure 2a shows a goal stream that does not take into the current state – the vector field becomes a constant ∀s ∈ S, making it impossible to accommodate the complex behavior needed to succeed from the entire state space.
An important issue to keep in mind is that Equation 6 is unidentifiable in the statistical sense because we can insert an arbitrary vector λ into the vector space and produce the same value estimate, as long as λ contains no zeros. In statistics, identifiability is the condition under which we can recover the true parameter of the model given infinite samples.
Q(s, a, g) = f(s, a)>φ(g) ⇐⇒ Q(s, a, g) = (λf)>(λ−1φ) ∀ λ ∈ Rn. (7) What this means is that under end-to-end training, the particular semantics of the two vector fields becomes particularly irrelevant. It is the grouping of the input variables and the bilinear structure that is critical.
It is also important to note that the bilinear value networks is implemented as a drop-in replacement of the universal value function approximator, as opposed to any additional algorithmic step. The training procedure we adopt in this paper follows those of deep deterministic policy gradient and hindsight experience replay, but the bilinear value networks is equally applicable to other off-policy learning algorithms under goal-conditioned scenario that uses either a value function, or a critic.

heading: 4 EXPERIMENT
text: Our experiments are aimed to answer the following two questions: (1) Does our value function parametrization improve sample efficiency? (2) Does the sample efficiency gain result from better value function generalizability? Moreover, we investigated the influence of each of our design choices in section 4.4.

heading: 4.1 SETUP
text: Tasks We evaluate our method in all the multi-goal RL benchmarks in Plappert et al. (2018), Fetch robot and Shadow hand manipulation tasks, as shown in figure 4. For Fetch robot domain, the robot has to interact with the object and move the object to the target positions. In Shadow hand domain, the agent controls the joints of the hand’s fingers and manipulates the objects in hand to a target pose. For all tasks, we set discount factor γ = 0.98 based on Plappert et al. (2018). The details,including the reward functions, of these tasks can be found in the the section A.1.
Baselines and our method We compare with two baselines: UVFA (Monlithic) (Andrychowicz et al., 2017; Schaul et al., 2015) and UVFA (Bilinear) (Schaul et al., 2015). UVFA (Monlithic) and UVFA (Bilinear) are all proposed in the original UVFA paper Schaul et al. (2015). UVFA (Monlithic) is the most common multi-goal value function architecture that parametrizes the Q-function by a monolithic neural network. UVFA (Bilinear) is a bi-linear value function parameterization similar to our method. UVFA (Bilinear) decomposes the Q-function into Q(s, a, g) = f(s, a)Tϕ(g), where f and ϕ denote two neural networks respectively. In the following sections, we will refer our method to as BVN (Ours).
Implementation The baselines and our method are trained by deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016) with hindsight experience replay (HER) (Andrychowicz et al., 2017). For all the baselines, we implement the Q-function as a 3-layer neural network with 256 neurons per layer. In our method, f and ϕ are both implemented as a 3-layer neural network with 176 neurons to match the total number of parameters of the baselines’ Q-function. Following the implementation in (Andrychowicz et al., 2017), we train each agent in parallel with multiple workers (varies across tasks). For BVN (Ours) and UVFA (Bilinear), the output dimensions f , ϕ(s, g), and ϕ(g) are set as 16. The rest of hyperparameters for training are left in section A.2.
Evaluation metric We ran each experiment with 5 different random seeds and reported the mean (solid or dashed line) and 95%-confidence interval on the learning curves. The learning curves indicate the average success rate over 15 testing rollouts in a varying number of training epoch. One epoch corresponds to the number of workers times two rollouts.

heading: 4.2 SAMPLE EFFICIENCY AND ASYMPTOTIC PERFORMANCE
text: To examine if our bi-linear value function parametrization improves sample efficiency, we evaluate each method in Fetch and Shadow hand domains. For Fetch domain, figure 5 shows that our method exhibits higher sample efficiency in 4 out of 8 tasks and matches the baselines in the rest of tasks. Remarkably, our method even attains higher asymptotic performance in Bin-place. Figure 5 shows that in Shadow hand, our method substantially improves the sample efficiency in 6 out of 8 tasks and is on par with the baselines for the rest of tasks. These results suggest that our bi-linear value network consistently advances the sample efficiency. One explanation is that better Q-function’s generalizability across goals helps the agent quickly learn a useful critic that will improve the policy’s performance. As a result, the exploration will be more focused on high-return regime. The collected high-return experience in turn improves the policy.

heading: 4.3 GENERALIZATION TO UNSEEN GOALS
text: To verify our hypothesis that generalizability improves sample efficiency, we examine bi-linear value network ’s performance on unseen goals. A Q-function that generalizes well should result in a policy that obtains high rewards on unseen goals since the policy’s training signals is solely from the Qfunction in actor-critic methods. To quantify the generalizability of the Q-function we split the goal distributions at training and testing time. We consider two scenarios in the Fetch robot domain: (1) near-to-far and (2) left-to-right. In near-to-far case, we specify a threshold on the goals’ distance to the agent’s initial state, and sample goals with Euclidean distance under and above the threshold for training and testing respectively. For left-to-right scenario, we split the goal space by an axis on the table (see figure 7) into the left and right half-spaces.
The goals from the left space are sampled at training time, whereas others at test time. We present the rest of implementation details on these scenarios in section A.3. Figure 8 shows both the training and testing success rate of the policies trained by each method. It can be seen that our method outperforms the baselines in terms of testing performance, showing that our method generalizes
better to unseen goals. Moreover, it is shown that our method’s gap between training and testing success rate are smaller than that of the baselines in Push and Slide in both near-to-far and left-to-
right scenarios. Notably, figure 8 shows that UVFA (Bilinear) underperforms the UVFA (Monlithic) in most tasks, suggesting that taking states as inputs at both branches (i.e., f and ϕ) is crucial to the success of the bi-linear decomposed value function.

heading: 4.4 ABLATION STUDIES
text: We investigate the influence of each component in our method to further our understanding of BVN. First, as the output dimension of f and ϕ could affect the expressivity of our bi-linear Q-function, we are interested in what extent this dimension influences the performance. Section C shows that larger dimension helps the performance and even the 3-dimensional space shows the sample efficiency gain. Also, we investigate if dot product is a good way to combine f and ϕ by comparing various ways to combine them. Section D shows that dot-product and 2-norm perform similarly, while concating f and ϕ largely undermines the performance. Lastly, we investigate if there is other viable ways to decompose the Q-function based on grouping the inputs s, a, g.
One hypothesis is that any inputs arguments (i.e., s, a, g) grouping can enhance sample efficiency as long as the parametrization shares some variables in f and ϕ. Thus we compare with two alternative inputs grouping scheme: f(s, a)Tϕ(a, g) and f(s, a, g)Tϕ(g). Figure 9 demonstrates that both groupings cannot match the performance of our proposed f(s, a)Tϕ(s, g), suggesting that not only the decomposition is important but a carefully designed grouping scheme is also crucial and states s might be necessary to be shared in two branches..
4.5 VISUALIZATION OF THE LEARNED f(s, a) AND ϕ(s, g)
To examine whether the learned bilinear value decomposition reflects our intuition mentioned in section 1, we trained an DDPG+HER agent with our bilinear value network in a 2D maze where s ∈ R2, a ∈ R2, g ∈ R2, and the reward is the negative euclidean distance between states and goals. The dimensions of Zf and Zϕ are both 16. We collected the output vectors of f(s, a∗) and ϕ(s, g) with a grid of states ((x, y)), a fixed goal g, and a∗ = π(s, g). For visualization, we use principal component analysis (PCA) to project ϕ(s, g) to a 2-dimensional vector space, and plot them along with f(s, a∗) in figure 10(right). We plot f(s, a∗) as ϕ(s, g) with ∠(f(s, a∗), ϕ(s, g)) rotation. In figure 10(right), we found that the lengths of the vectors reflect the distances to the goal at a state, which validates our hypothesis that ϕ captures the global topology to the goal (i.e., proximity to the
goal). Moreover, we further compare the the angle ∠f,ϕ between that of the optimal action (from the policy π) ~f(s, a∗) and a sub-optimal, random action ~f(s, arand). We show that on this simple domain, the ~f(s, a∗) better aligns with ~φ(s, g), validating our intuition that the policy finds the bestaligned ~f(s, a) Note that there is an offset of 90 degrees in the relative angle between ~f and ~ϕ, that exists. This is because in this domain, the maximum value the agent can get is not 1, but zero. As a result, ~f for the optimal action will not be perfectly aligned with ~φ.

heading: 5 RELATED WORKS
text: Our work is closely related to the literature on value function approximation in the sense that both seek for a lower dimensional space to represent the value function in a more compact way. Most early works in RL approximate the value function by the linear combination of basis functions (Sutton & Barto, 2018) (e.g., polynomial basis, radial basis, fourier basis (Konidaris et al., 2011)). Instead of arbitrarily picking basis functions, proto-value function framework (Mahadevan, 2007; Mahadevan & Maggioni, 2006) compose the value function in the linear combination of task-independent basis vectors constructed by the spectral of the state transition function of the environment. The works on successor representations (Dayan, 1993; Kulkarni et al., 2016; Borsa et al., 2018; Barreto et al., 2017; 2018) seek for a value function representation that help the value function transfer to new tasks. This line of works bi-linearly compose the value function by a rewardagnostic and reward-specific parts, so the reward-agnostic part can be reused in a new task where the environment’s transition function is the same. The idea of successor representations was introduced to multi-goal RL in (Schaul et al., 2015) for decomposing the goal-conditioned value function.
Its difference to prior works on successor representations is that the reward-specific component is a function conditioned on goals. Despite the appealing performance shown in toy gridworld, the follow-up works in multi-goal RL rarely use the decomposed architecture proposed in (Schaul et al., 2015). Andrychowicz et al. (2017); Pong* et al. (2018) all parameterize the Q-function as a monolithic neural network. Section 4.2 empirically shows that the bilinear decomposition proposed in (Schaul et al., 2015) cannot extend its advantages to more complex tasks. As a result, positioned in the literature of multi-goal value function approximation, our principal contribution lies in solving the bottleneck that hinders the advantages of value function decomposition in multi-goal RL.
In addition to devising the function approximator parameterization, recent works attempted to improve RL by learning the Q-function in a hierarchical manner. Edwards et al. (2020) estimate the Q-value conditioned on the current state and the next state, and deriving the action to the optimal next state by an inverse dynamics model. Mirowski et al. (2018) trains a monolithic value network using the goals prepossessed by different networks for each environment. Nachum et al. (2018); Levy et al. (2019) learn a hierarchy of policies and multi-goal value functions, and take the highlevel policies’ outputs as the low-level policy/value function’s goals inputs. Orthogonal to this line of works, we focus on improving the value function’s parameterization, which can be a drop-in replacement in the above related works.
In the abstract sense, the spirit of our method is to model the goal-agnostic (state-action) and goalspecific (state-goal) information separately in different components. A proper decomposition is known to make function approximation easier due to modularity. Rusu et al. (2016) demonstrates the modularity in a network facilitates continual learning in new tasks, as the previously learned subnetworks can be reused in new tasks Wang et al. (2018) learns a modular policy where each network controls a joint of a robot. This modular policy is shown to be able to better transfer to new robot morphology. Galashov et al. (2019) train a specialized policy and a reusable policy restricted from the reward information for encouraging the reusability in other tasks. Different from these works, our work imposes modularity in the parametrization of value functions and focuses on improving genearlizability and sample efficiency of value functions.

heading: 6 DISCUSSION
text: Our experimental results demonstrated that our BVN significantly improves sample efficiency in multi-goal RL tasks. Also, we show that this sample efficiency gain could result from better generalizability of the value function. Beyond the aspect on the sample efficiency and generalizability in RL, an exciting future work is leveraging the decomposition to facilitate transferring across different robots’ morphology and goal spaces. One can pre-train a shared ϕ for the same state (e.g., camera) and goal space, and train f for robots with different action spaces (e.g., different manipulator for the same task). Also, one can pre-train a shared f for the same robot, and train ϕ for different tasks (i.e., goal spaces). In addition, we further discuss the additional implications of our experimental results in section E of appendix.

heading: 7 REPRODUCIBILITY STATEMENT
text: Please refer to section A.3.

heading: A APPENDIX
text: A.1 IMPLEMENTATION DETAILS OF TASKS
The implementation details of all the tasks used in section 4.2 are referred to (Plappert et al., 2018). We briefly summarize the task specifications below
• FetchReach: s ∈ R10, a ∈ R3, g ∈ R3
• FetchPush: s ∈ R25, a ∈ R3, g ∈ R3
• FetchPickAndPlace: s ∈ R25, a ∈ R4, g ∈ R3
• FetchSlide: s ∈ R25, a ∈ R3, g ∈ R3
• DrawerClose: s ∈ R25, a ∈ R3, g ∈ R3
• DrawerOpen: s ∈ R25, a ∈ R3, g ∈ R3
• ShadowHandBlockRotateZ: s ∈ R61, a ∈ R20, g ∈ R7
• ShadowHandBlockRotateParallel: s ∈ R61, a ∈ R20, g ∈ R7
• ShadowHandBlockRotateXYZ: s ∈ R61, a ∈ R20, g ∈ R7
• ShadowHandBlockRotate: s ∈ R61, a ∈ R20, g ∈ R7
• ShadowHandEggRotate: s ∈ R61, a ∈ R20, g ∈ R7
• ShadowHandEgg: s ∈ R61, , a ∈ R20, g ∈ R7
• ShadowHandPen: s ∈ R61, , a ∈ R20, g ∈ R7
• ShadowHandPenRotate: s ∈ R61, , a ∈ R20, g ∈ R7
In addition to (Plappert et al., 2018), we also use the tasks from Fetch extension1. For left/right environments used in section 4.3, the goal (x, y, z) with x < 0 are left goals, and x ≥ 0 right goals. In near/far environments, we set the distance thresholds as 0.2 for Slide, 0.2 for PickAndPlace, 0.1 for Reach, and 0.15 for Push. The goals with distance below the threshold are considered as near goals, otherwise far goals.
A.2 TRAINING HYPERPARAMETERS
Fetch We adapted the hyperparameters in (Plappert et al., 2018) for training in a single desktop. The performances of the agent trained by our hyperparameters match that in (Plappert et al., 2018).
• num workers: 2 for Reach, 8 for Push, 16 for Pick & Place, and 20 for Slide.
• Batch size: 1024
• Warm up rollouts: We collected 100 initial rollouts for prefilling the replay buffer.
• Training frequency: We train the agent per 2 environment steps. Each iteration of training uses the num workers batches for training.
The rest of hyperparameters remain the same as that used in (Plappert et al., 2018).
Shadow hand We use the hyperparameters used in (Plappert et al., 2018) and reuse the open source code2.
A.3 IMPLEMENTATION DETAILS
For the baselines, we concatenate s, a, g into a vector and feed it into the network. For our method, we concatenate (s, a) as inputs of f , and (s, g) as inputs of ϕ.
Dot product This is our proposed method. We set the outputs’ dimension of f and ϕ as 16. 1https://github.com/geyang/gym-fetch 2https://github.com/TianhongDai/hindsight-experience-replay
2-norm We parametrize Q-function as Q(s, a, g) = ‖f(s, a) − ϕ(s, g)‖2. We use set the dimension of the outputs from f and ϕ as 3. Figure 11 shows that d = 3 works the best for 2-norm.
concat We concatenate f(s, a) and ϕ(s, g), feed the concatenated vectors to a multi-layer perceptron (MLP) layer, and take the outputs of this layer as the Q-value Q(s, a, g). The dimensions of the outputs of f and ϕ are 16.
A.4 TRANSFERRING ACROSS TASKS
In multi-goal RL tasks, we observe that most of tasks are identical in terms of high-level goals, but they have different environment dynamics. For instance, the objective of ”Push” and ”Slide” are all to move the object to the destination. They only differ in the friction coefficient of the table where the agent cannot slide objects in ”Push” because of high friction. Because we decouple the state-action and goals into two different functions, f(s, a) and ϕ(s, g), one potential benefit on multi-goal RL is reusing ϕ for transferring to tasks with the same (or similar) task objectives (i.e., reward specification). Reusing ϕ(s, g) exploits the learned reward-specific state-goal relations. Then we simply need to learn a new f to recover the reward-agnostic relations in the new task. To examine this hypothesis, we select three tasks with the same state and action spaces3, and measure the performance of transferring amongst these tasks. In transferring an agent from a source task to a target task, we initialize UVFA (Monlithic) agent’s policy and Q-function by the neural networks’ weights learned in the source task and fine-tune them in the target task. Different from UVFA (Monlithic), we reinitialize f and fine-tune ϕ in UVFA (Bilinear) and BVN (Ours) when training them in the target task. Figure 12 shows the success rates of transferring from a source task to a target task. For example, ”Push to Slide” indicates transferring from ”Push” to ”Slide”. It can be seen that BVN (ours, w reset f ) attains higher success rates at convergence than the baselines and BVN (ours, w/o reset) in 4 out 6 tasks (i.e., ”Push to Slide”, ”Pick & Place to Push”, ”Push to Pick & Place”, and ”Pick & Place to Slide”), despite slight improvements.
The limited efficiency gain could result from that ϕ is dependent on states s and thus ϕ cannot be effectively reused. Even though the target task’s reward function is the same as that in the source task, the relation of states and goals can be affected by the dynamics. For instance, suppose we have a state where the object is near the goal. In ”Pick & Place”, because the friction coefficients of the table are high, the ϕ(s, g) should direct the agent to lift the object. On the other hand, in ”Slide”, the ϕ(s, g) is expected to guide the agent to push the object to slide.
3If the two tasks have different state/action spaces, we will not be able to use the weights from a task and resume training in another task.

heading: B COMPARISON WITH OTHER ALGORITHMS
text: To show that our bilinear value network achieves the state-of-the-art performance in common multigoal RL tasks (Plappert et al., 2018), we further compare our method with twin-delayed deep deterministic policy gradient (TD3) (Fujimoto et al., 2018) and soft actor critic (SAC) ((Haarnoja et al., 2018)). For TD3 specific hyperparameters, we reuse the hyperparameters reported in (Fujimoto et al., 2018) and implement TD3 in our codebase. As for SAC, we use the implementation in this codebase4. Figure 13 shows the performance of each baseline algorithm and our method (DDPG + BVN (Ours)). We show that our DDPG + BVN (Ours) outperforms the best baseline, and find that DDPG works the best among all the UVFA (Monolithic) variants. Our findings align with (Pitis et al., 2020) and the documentation in its released code5. Pitis et al. (2020) also found that DDPG works better than TD3 and SAC in multi-goal RL domains.
4https://github.com/spitis/mrl 5https://github.com/spitis/mrl

heading: C HOW DOES LATENT DIMENSION AFFECT OUR METHOD?
text: We made several variants of our method with different latent dimension of the outputs of f and ϕ and present their learning curves in figure 14. It can be seen that the performance consistently improves when we increase the latent dimension. Though the performance drops when the dimension decreases, our method with dimension 3 still outperforms the UVFA (Monlithic) baseline. These results suggest that our method benefits from increasing latent dimension but is insensitive to the choice of the dimension.
D IS COMBINING f AND ϕ BY DOT PRODUCT CRUCIAL?
We compared two variants that reduces f and ϕ by 2-norm and concatenation. In 2-norm variant, we parametrize the Q-function as Q(s, a, g) = −||f(s, a) − ϕ(s, g)||2. The negative sign is added because multi-goal RL tasks typically assume the reward is within the range [−1, 0]. For concatenation variant, the Q-function is formulated as Q(s, a, g) = MLP([f(s, a), ϕ(s, g)], 1), where we feed the concatenated f(s, a) and ϕ(s, g) to a multi-layer perceptron (MLP) network that outputs a one-dimensional value. We set the latent dimension of the 2-norm variant as the best one found in our hyperparameter search (see section A.3). The results shown in figure 15 demonstrate that Bi-linear-full (dot) significantly outperforms concat variant, and Bi-linear-full (2-norm) approximately matches the performance of Bi-linear-full (dot). This result suggests separating the f and ϕ embedding is the key to the performance gain, and concatenating them at the output diminishes the gain.

heading: E ADDITIONAL DISCUSSION
text: Why does other parameter grouping schemes fail? In section 4.4, we showed that f(s, a, g)Tϕ(g) inputs grouping fails to improve sample efficiency. One explanation to this result is that f(s, a, g) cannot prevent negative interference from new goals encountered during training since goals g couple with states-actions s, a. As a result, the merit of bi-linear value function decomposition vanishes and even causes extra burden on learning because the number of model parameters
is larger than UVFA (Monlithic). This preliminary finding shows that to benefit from decomposition, the guideline is to only expose necessary inputs to each component. Redundant inputs in either component could break the decomposition.
How is 2-norm compared with dot product? Section 4.4 demonstrated that using 2-norm to reduce f(s, a) and ϕ(s, g) into Q-value matches the performance of our BVN with even lower dimensions of Zf and Zϕ. However, 2-norm relies on an assumption that the reward function must be non-positive or non-negative since norm is defined as a non-negative value. The ‖f(s, a)−ϕ(s, g)‖ can only fit to either non-positive or non-negative Q-value by manipulating the sign of norm. On the other hand, dot product is more expressive and free from the reward function assumption.
Can our method facilitate policy transfer? Since our method decomposes the value function into a local (i.e., goal-agnostic) and a global (i.e., goal-specific) component, our BVN might facilitate policy transfer by resetting either one of components and fine-tuning another one. As multi-goal tasks considered in (Plappert et al., 2018) have the same objective in nature but different dynamics, one potential advantage of our method is to reuse ϕ(s, g) in the new task and re-train f(s, a). The results in section A.4 showed that the agent transferred by BVN converges to the the performance of training from scratch in the target task slightly faster than the baselines. One reason for the limited policy transferring performance is that ϕ(s, g) conditions on states s. Consequently, environment dynamics shifts could affect the representations of ϕ. We, nevertheless, showed that states variables are crucial in ϕ. One exciting avenue is to disentangle the representations for keeping the rich expressivity as well as desired modularity for transferring across tasks.

Review summaries:
The paper proposes a modification to goal-oriented universal value functions that split the neural network into two parts. One network accepts states and actions and outputs a vector representation, while the other outputs a vector given a state and goal. Part of the contribution of the paper is the interpretation of these two vectors --- the first captures the notion of what actions can be taken at a given state, while the second represents the direction the agent should move towards to achieve the goal from its current state.  Empirical results on several simulated robotic domains demonstrate that the method outperforms other UVFA architectures, while ablations are provided to capture the most important aspects of the method.

This paper proposes a new decomposition for the universal value function that disentangles local and global components. The global component depends on the goal and state and tells the agent “where to go” and the local component depends on the state and action and tells the agent “what to do”. The authors conduct a set of experiments on a number of environments to demonstrate that this decomposition results in more data efficiency and helps to generalise to the new goals.

This paper considers the problem of learning a goal-conditioned policy effectively. Particularly, it looks at the architecture of the value network used to learn this goal-conditioned policy if using a deterministic policy gradient algorithm. It studies whether separating the goal-conditioned value network into two components, a state-action embedding network and a state-goal embedding network, might lead to faster learning and better generalization of the policy. The hypothesis such a study tests is whether the separation can tease apart what an agent's actions in a particular state accomplish and what the agent needs to accomplish to get to a goal from a given state. This hypothesis is evaluated by conducting experiments on the Fetch and Hand domains and evaluating how quickly the goal-conditioned policy is learned and whether it generalizes to goals in parts of the state space that it was not trained on.