Bi-linear Value Networks for Multi-goal Reinforcement Learning
Authors: Zhang-Wei Hong
Ge Yang
Pulkit Agrawal
Venue: ICLR 2022 Poster (2022)
Keywords: Multi-goal reinforcement learning
universal value function approximator

Abstract: Universal value functions are a core component of off-policy multi-goal reinforcement learning. 
The de-facto paradigm is to approximate Q(s, a, g) using monolithic neural networks which lack inductive biases to produce complex interactions between the state s and the goal g. In this work, we propose a bilinear decomposition that represents the Q-value via a low-rank approximation in the form of a dot product between two vector fields. The first vector field, f(s, a), captures the environment's local dynamics at the state s; whereas the second component, ϕ(s, g), captures the global relationship between the current state and the goal.
We show that our bilinear decomposition scheme improves sample efficiency over the original monolithic value approximators, and transfer better to unseen goals. We demonstrate significant learning speed-up over a variety of tasks on a simulated robot arm, and the challenging task of dexterous manipulation with a Shadow hand.

Published as a conference paper at ICLR 2022
BILINEAR VALUE NETWORKS
Zhang-Wer Wei Hong; Ge Yang" & Pulkit Agrawal 8 Improbable AT Lab NSF AI Institute for Al and Fundamental Interactions (IAIFI)", MIT-IBM Watson Al Lab² Massachusetts Institute Technology
ABSTRACT
The dominant framework for off-policy multi-goal reinforcement learning in- volves estimating goal conditioned Q-value function. When learning to achieve multiple goals, data efficiency is intimately connected with generalization of the Q-function to new goals. The de-facto paradigm is to approximate Q(s, a, us- ing monolithic neural networks. To improve generalization of the Q-function, we propose a bilinear decomposition that represents the Q-value via low-rank ap- proximation in the form of a dot product between two vector fields. The first vector field, [(s,, a), captures the environment's local dynamics the state s; whereas the second component, (888, g). captures the global relationship between the cur- rent state and the goal. We show that our bilinear decomposition scheme sub- stantially improves data efficiency, and has superior transfer to out-of-distribution goals compared to prior methods. Empirical evidence is provided on the simulated Fetch robot muskes and dexterous manipulation with Shadow hand.
INTRODUCTION
Learning Q-value function (Watkins & Dayan, 1992) is among the core problems in reinforcement learning (RL) (Sutton & Barto, 2018). It is often desired to learn a goal-conditioned Q-value func- tion (Kaelbling, 1993) that estimates rewards for multiple goals. In multi- goal RL, the Q-function takes the form of Q(s, a.g) which indicates the utility of taking action at state for approaching the goal g. enable the agent to work across wide range of goals, it is usual to train Q(s, a, g) for large set of goals. Such training requires substantially more data than learning a Q-value function for a single goal (Schaul et al., 2015).
The data efficiency of learning Q(s, a,g) has a close relationship with the generalization of the Q-function to new goals. To understand why, consider the training process where goals are sam- pled from pre-defined training distribution. The target of Q-lernning is to reduce the error on the Q-function for all goals in the training set. At an intermediate point in the training process, the agent would have encountered a subset of training goals. Suppose these goals the error in ap- proximating the Q-function low. If the learned Q-function generalizes to new goals sampled in subsequent training, less data would be required correct the small errors in Q-function for these goals. Conversely, poor generalization would necessitate collecting large amount of data.
Currently popular methods (Schaul et al., 2015; Andrychowicz al., 2017) represent the Q(s, a, .g) with a monolithic neural network, shown in Figure la, whose ability to generalize to new goals solely derived from the neural network architecture. Because such models jointly process the tuple (s, a, g). the Q-function fails to generalize on new combinations of previously seen individual components. F.g, given training tuples, (SA, ga) and (*,, aB; 9B), the monolithic network might fail generalize to new data (s a, gB) despite that SA. a.A and gø have been seen at least once during training.
We proposed bilinear value networks (BVN) to improve such generalization by introducing an inductive bias in the neural network architecture that disentangles the local dynamics from the ques- tion of how to reach the goal. Concretely, propose to represent the goal-conditioned Q-value
Equal contribution. order determined randomly. Authors are also with Computer Science and Artificial I Laboratory (CSAIL.), and affiliated with the Laboratory for Information and Decision Systems (LIDS). Corre- spondence (zwhong, geyang) esail .mi edu

---

Published as a conference paper at ICLR 2022
F(s.0) a)
f(s, a)
Q(s, a. g)
Q(s, a g)
((s.9)
spss g)
(a) A Monolithic value network
(b) Bilinear value network
(c) Bilinear decomposition
Figure I: Illustration of why Bilinear Value Networks (BVNs) improve generalization to new goals. (a) The traditional monolithic goal-conditioeed value function jointly models the inputs (s, a, g). (b) Bilinear Value Networks (BVN) introduce an inductive bias in the Q-function computation by disentangling the effect of actions at the current state from how to reach the goal. This is realized by representing Q(8, a, 9) Φ(8, 9)2
"h)"""""" [(s a),, where the vector fields represent "whal happens (ss, a) and "where to go next" ((8.9) g). (c) A 2D navigation toy example, where state & and goal g are both (z, y) positions. The task of this toy example is reach the goal position g. Gray arrows indicate the vector field o(s, g) evaluated at all states. The black arrow indicates the best alignment with (s)
function as interaction between two concerns, ((,,9) ("where to go") and f(ss, a) ("whal happens next"), where each concern can be represented as a neural network. The architecture of BVN il- lustrated in Figure lb. ((s,g) g) models how to change the state get closer to the goal, while f a) captures where the agent can move from the current state. The Q-function is computed as the align- ment between these concerns: 6(s, g) f(s, a). As the ø-network only requires state-goal (s, g) and f f-network simply needs state-action (s,a), the learned Q-value function can generalize to the new data (s A: a A: 9R) since both As aA) and (s A: 9B) are the seen data for and ø, respectively. Such decomposition is possible because the knowledge of local dynamics (/) is independent of where to go (Ø) and thus f(s a) be re-used across goals.
Another intuition about the proposed BVN decomposition is: BVN would predict a high Q-value when the vector representation of the desired future state, o(s, g). aligns with the vector representa- tion of the future state, f(s, a), after executing action a. We illustrate this intuition with help of a toy example shown in Figure Ic. Here the agent's state and the goal g are 2D positions, and the task is to get close to g shown as a pink cross. As an analogy to "what happens next", the black arrow originating at state s (shown as the blue circle) denotes the succeeding state after taking an action at state 8, which is goal-agnostic. Independent of action, the grey arrows can be viewed as the direction to the goal from the current state 8 (i.e., o²(s, g) or "where to go"). While this illustra- tion was in 2D, we find that the proposed bilinear value decomposition substantially improves data efficiency (Section 5. and generalization to new goals (Section 5.2) in complex and standard simu- lated benchmark tasks: (i) dexterous manipulation with twenty efgrees-of-freedonn (DoF) shadow hand and (ii) object pick-plan place tasks using the Fetch manipulator.
2 PRELIMINARIES
We consider the multi-goal reinforcement learning problem (Kaelbling, 1993; Plappert et al., 201 8), formulated as a goal-conditioned Markov Decision Process (MDP). We assume infinite horizon with the discount factor 7, and use S, AA, G for the state, action, and goal spaces. At the beginning of each episode, an initial state 8o and a goal is sampled according to the distribution goal g is fixed throughout the episode. At every time step the agent takes an action according to a, T($,,)) g), and receives the reward It.. R(s,, g) while transitioning to the next state $141- The objective of multi-goal RL produce policy x that maximizes the expected return J
and Pg. The
π²" = arg max (T) where (TT) = r rrg
19:
(1)
7
T=t
2

---

Published as a conference paper at ICLR 2022
where the reward Fr is often structured as the follows:
0, goal reached otherwise.
R(87, 17, Or:
9)
VT EN IN
(2)
Note that our proposed parameterization scheme, BVN, does not rely on this reward structure in general. All of the baselines use discount factor y = 0.98.
2.1 OFF-POLICY RL WITH ACTOR-CRITIC METHODS
It is common use actor-critic algorithms such as deep deterministic policy gradient (DDPG) (Sil- ver et al., 2014), twin-delayed DDPG (TD3) (Fujimoto et al., 2018) and soft actor-critic (SAC) (Haarnoja et al., 2018), to solve multi-goal continuous state-action space tasks considered this paper. In DDPG, the critic Q is learned by minimizing the temporal-differenc (TD) loss:
( = ((tr..))>~uullom()((() + * (81+111 g). 9) -
ar, (),,))
(3)
where D is the the replay buffer (Mnih et al., 2015) containing state transition tuples (8t, de, rt, St+1, g). The actor is trained to maximize the score from the critic, using the deter- ministic policy gradient:
-=J(=)= (( (e.
In multi-goal RL settings, we refer the goal-conditioned critic as the universal value function approximator (UVFA) (Schaul et al., 2015) because ((s, a. g) approximates the expected utility for all goals E g. Because the actor relies on the critic for supervision, the ability for the critic to quickly learn and adapt to unseen goals is a key contributing factor to the overall sample efficiency of the learning procedure.
critic that provides better value estimates using less sample and opti- mization gradient steps, will lead to faster policy improvement via the gradient in Equation 4. For data efficiency it is common to use off- pollcy actor-critic methods with hindsight experience replay (HER) (Andrychowicz et al., 2017). It is known that TD3 and SAC outperform DDPG on single- task continuous control problems. However, the multi-goal setting when used along with HER, we found DDPG to outperform both TD3 and SAC on the benchmark tasks considered (see Sec- tion B). We therefore use DDPG for majority of our experiments and report SAC/TD3 results in the appendix.
BILINEAR VALUE NETWORKS
Hs
Bilinear value network (BVN) factorizes the value function approximator into bi- linear form from two vector representa- tions into a scalar:
x(g)
Ō(5,9)
a(s.a,g0
((s,g). g).
(5)
where f: x A Rd and @ S + Rd. The first component f(s, a) is lo- cal field that only concerns the immediate action the current state The second component is more global in the sense that o(s,g) captures morghorizon relationship between the current state and the goal.
(a) [(s ((s,a)Tá() [(())
(b) f(s a) (().))
Figure 2: Comparison between two value decomposi- tion schemes. The task, states, and goals are the same Figure 1c. The black square indicates obstacles that the agent cannot pass through. Gray arrows indicate the vector field @ evaluated a particular goal g, for all states E S. (a) As in prior work Schaul et al. (2015), the vector field ((g) is constant and does not depend on s. (b) BVN parameterizes ó as function of both and g making more expressive. but still maintaining the benefit of disentanglement from as)
Closely related to our work is the low- rank bilinear decomposition, Q(s, a.
f(s, a) Φ(g), proposed in Schaul et al. (2015). Though Schaul et al. (2015) showed this decomposition attains bet- ter data efficiency in simple environments (c.g., gridworlds), this decomposition is
not expressive enough model value functions in complex environments, which will show
3

---

Published as a conference paper at ICLR 2022
Maze
Latent Space
100
20
-3
40
((1,g)
As,a")
As, Amaa
(8)
(b)
(C)
Figure 3: (a) The U-maze environment where the states and goals 9 are (ær y) on the plane. The task is to reach the goal position (red circle). (b) The learned Q-value predictions Q(8, a"; .².9) with the best action a" states in the 2-dimensional state space for goal g indicated by the red circle. Overall, values decrease as the agent gets further and further away from the goal. (c) The latent vectors produced by the agent, projected to 2D vector space. There is an overall trend for to decrease in magnitude one gets closer to the goal. (c Inset) Comparing the alignment of vector representation of random (suboptimal) action produced by the component, f(s, Grand) (pink) against the vector corresponding to the optimal action from the policy f(s, a*) (blue) from the phi vector (black). The optimal alignment corresponds to an angle of 90 because the maximum Q-value in this environment is 0 and not 1. We find that the optimal action consistently produces better alignment in comparison to sub-optimal action.
in Section 5.2. Consequently, the majority of multi-goal RL algorithms monolithic network shown in Figure la to represent the value function.
To motivate the design of factorization, we present vector analysis perspective previous value decomposition Schaul et al. (2015) and our method in Figure 2, where each point on the 2D plane state In Figure 2a, we illustrate the vector field of low-rank bilinear UVFA (Schaul et al., 2015), where f :S x A + Rd and Φ G + IRd, It can be seen that ((g) is a constant across states since ((g) does not depend the s in Figure 2a. This makes learning more challenging due to restricted expressivity. The expressivity of low-rank bilinear UVFA is restricted since the such a decomposition can only model linear relationships between states and goals. Consequently, the burden of modeling different Q-values for different states falls completely on J. In contrast, in Figure 2b, the field 0bx,, 8, 9) produced by BVN is also a function of the state 8. This makes ş BVN strictly more expressive than low-rank bilinear UVFA that do not depend on
Bilinear value networks are a drop-in replacement for the monolithic neural networks for approxi- mating universal value function approximator. Our implementation is not specific to method and be used to learn value function using a variety of methods such as DDPG, SAC, TD3 or even methods that we do not make use of temporal differenceing.
3.1 A Toy EXAMPLE: VISUALIZING f(s, a) AND o(s,g)
To examine whether the learned bilinear value decomposition reflect intuition mentioned in Section 1, we trained a deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016) with hindsight experience replay (HER) (Andrychowicz et al., 2017) agent using BVN in 2D maze shown in Figure (a), where E R², E R E IR2 and the reward is the negative euclidean distance between states and goals. The output dimensions of f and are both 16.
We computed the vectors 2 = f(s, ) and 2ạ = .(s,g) densely on grid of states (x, with respect to the fixed goal g, where a' = [Ts g) is the optimal action predicted by the learned policy. For visualization, we used principal component analysis (PCA) to project 26 to a 2-dimensional vector and is plotted as black arrows in Figure 3(c). We cannot use PCA to plot 2 on the 2D plane plot along with Zphi because f(s,a") and d(s.g) .g) live in two different vector spaces. Instead, f(s, a²) as a rotation from ΣΦ denoted as [(f(s, a²) (,), ((s g)). Specifically, we first plot the 2D projection of 2phi 0(s, g) 2D vector and then plot 27 f(s, a²) by rotating the plotted 2s by angle of (Js(s, a²), (){,
4

---

Published as a conference paper at ICLR 2022
Further, Figure 3(c) shows that the length of 2phs correlates to the distance between the state (s) and the goal (g), providing further evidence in support of our hypothesis that o(s, g) captures proximity to the goal. If [s, a) indeed captures the dynamics, then 27 (i.c., vector representation of future state after executing the optimal action a²) should be closely aligned to Sphi to produce the maximal Q- value. To test if this is the case, we also plot in pink 27 denoting the vector representation of future state after executing a random action arand We find that in comparison to [(s, a'and), f(s, a") better aligns with o(s, g) validating our hypothesis. Note that the optimal alignment corresponds to an angle 90º because the maximum Q-value in this environment and not 1. As a result, f for the optimal action will be orthogonal to ø.
EXPERIMENTAL SETUP
We evaluate our method in all the multi-goal RL benchmarks in Plappert et al. (2018), Fetch robot and Shadow hand manipulation tasks, shown in Figure 10. For Fetch robot domain, the robot required to move the object to a target position. In Shadow hand domain, the agent controls the joints of the hand's fingers and manipulates the objects in hand to target pose. For all tasks, we set discount factor 7 = 0.98 based on Plappert et al. (2018). The details, including the reward functions, of these tasks can be found in the Section A.2.
Tasks
Baselines We compare with two baselines, UVFA (Monlithic) (Andrychowicz et al., 2017: Schaul et al., 2015) and UVFA (Bilinear) proposed in Schaul et al. (2015). UVFA (Monlithic) is the most common multi-gon gooii value function architecture that parametrizes the Q-function by monolithic neural network. UVFA (Bilinear) is a bi-linear value function that decomposes the Q-function as Q(s, a. f(s, a)" ő(g), where and é denote two neural networks respectively. Our method uses a different bilinear decomposition and we refer to it as BVN (Ours).
Implementation The baselines and our method are trained by deep deterministic policy gradi- ent (DDPG) (Lillicrap al., 2016) with hindsight experience replay (HER) (Andrychowicz al., 2017). We use DDPG with HER as the base algorithm for our method and the baselines because outperforms other recent Q-learning algorithms (Haarnoja et al., 2018; Fujimoto al., 2018) on the task of learning a condiiondt Q-function. Comparison of DDPG against SAC (Haarnoja et al., 2018) and TD3 (Fujimoto et al., 2018) is provided in Section B. For all basclines, the Q-function is implemented as a three layer neural network with 256 neurons per layer. In our method, f and ø are both implemented as 3-layer neural network with 176 neurons to match the total number of pa- rameters of the baselines' Q-function. For BVN (Ours) and UVFA (Bilinear), the output dimensions f, ((s,))) g), and d(g) are set as 16. The rest of hyperparameters for training are left in Section A.3.
Evaluation metric We ran each experiment with 5 different random seeds and report the mean (solid or dashed line) and 95%e-confidenc interval (shaded region) using bootstrapping method (Di- Ciccio & Efron, 1996). Each training plot reports the average success rate over testing rollouts as function of training epochs, where the goal is sampled in the beginning of each testing rollout. Each epoch consists of fixed amount of data (see Section A.3 for details).
5 RESULTS
Our experiments answer whether BVN improves sample efficiency and generalizes better to unseen goals. In Section 5.3, we provide ablation studies probing the importance of several design decisions.
5.1 SAMPLE EFFICIENCY AND ASYMPTOTIC PERFORMANCE
Figure compares the sample efficiency of our method, BVN (Ours), against baselines on the Fetch tasks. The results show that our method has higher data efficiency in four out of eight tasks and matches the baselines in other tasks. Our method also attains higher asymptotic performance in Bin-place. Figure 5 shows that on the Shadow hand task suite, BVN improves the sample efficiency in six out of the eight tasks and is on-par with the baselines for the other two tasks. These results suggest that BVN consistently improves sample efficiency of learning coal-conditioned policies.
5

---

Published as a conference paper at ICLR 2022
Reach
Push
Slide
Pick & Place
rate
75%
50%
0%
20
30
30
100
125
30
100
200
30
100
150
200
Drawer-Oper Open
Drawer-Clon
Bin- Ain-place
Bin-pick
1000%
75%
Success
25%
086
25
75
100
75
100
25
50
100
25
100
Epuch
Epoch
Epoch
Epoch
UVFA (Monolithi))
UVFA (Bilinear)
BVN (Ours)
Figure 4: Learning curves on the Fetch gym environments, and the Fetch-extension (see Section A.4) task suit. BVN attained higher success rates using less data than the baselines in out 8 domains. Also, BVN achieved higher asymptotic performance in Bin-place.
Block Rotate Z
Block Parallel
Block Rotate XYZ
Block
100%
75%
50%
096
20
30
30
75
25
30
100
25
100
Egg Rotate
Egg
Pen Rotate
Pen
1000%
Success
30%
25%
20
75
100
10
25
100
Epach
Epoch
Epoch
ppoch
UVFA (Monolithic)
UVFA (Bilincar)
BVN (Ours)
Figure 5: Learning curves on the Shadow hand dexterous manipulation suite. BVN learns faster
than the baselines six out of eight domains.
5.2 GENERALIZATION TO UNSEEN GOALS
Our central hypothesis is that BVN increases data efficiency by generalizing better to new goals. To test this, split the training and evaluation goals in ways in the Fetch robot domain: (1) left-to-right and (2) near-to-far. In near-to-far scenario, all goals within a pre-specifed euclidean distance are used for training (near) and goals sampled outside the threshold distance are used for testing (i.c., far). The train-test split of goals is illustrated in Figure 6b by colors green and pink respectively. For left-to-right scenario, we split the goal space into the left and right half-spaces (see Figure 6a). During training, we disable HER to prevent the Q-function from receiving goals in testing split due to relabeling. We train the baselines and our method until they achieve similar success rates (see Section A.3 for details). For testing, we finetune the learned Q-function and the policy using the weights from the best model over 3 random seeds from training time. We finetune the Q-function and the policy using HER.
The fine-tuning performance of the baselines and BVN is reported Figure 6c. We find that BVN (shown in blue) quickly adapts to new goals and outperforms the monolithic UVFA (shown in grey). This result suggests that BVN generalizes better than UVEA (Monlithic) to new goals.
Recall that in the BVN formulation, f(s, a) does not encode any goal specific information. We therefore hypothesized that only finetuning Ø(s, g) should be sufficient for generalizing to new goals. We call this version of finetuning BVN (freeze f). Results in Figure 6c support hypothesis

---

Published as a conference paper at ICLR 2022
(a) Left & Right
(c) Learning curves of fine-ture tuning in goal distributions
Push
Slide
Pick & Place
Sucoess rate
(left-to-1) (ott-to-righ))
Test
50%
25%
0%
Train
100
200
200
(h) Near
100%
Success miee
(near-to-fi (oo-fr))
75%
50%
25%
Test
0%
0
25
50
75
100
125
150
50
100
150
200
50
100
150
200
Epoch
Epoch
Epoch
Train
UVFA (Monolithic)
BVN (finetunc f) BVN (freeze f)
Figure 6: (a & b) Examples showing (a) how we split the goal distribution into left and right and (b) according to the radius to the center (near versus far). We pretrain on the green regions, then finetune on the red regions. The state distribution remain identical the vanilla Fetch environment. (c) Fine- tuning curves on unseen goals. BVN achieves better performance than the UVFA (Monlithic) on all tasks in both left-to-right and near-to-far adaptation scenarios. Freezing / causes BVN performance to plateau, indicating that both and are needed for approximating the multi-goal value function.
BVN (freeze f) (orange) indeed outperforms UVTA (Monlithic). However, there significant performance gap between BVN (finetume f) and BVN (freeze f). This result indicates that while BVN provides an inductive bias disentangle the local dynamics. f(s, a). from the goal-specific component, o(s, g), such disentanglement is not fully learned. This could be due to several reasons such learning on limited data because the disentangelment between f and Φ is under-specified upto a scalar factor (i.e., Xf and 1 yield the same result for scalar A), etc. Nonetheless, the results convincingly demonstrate that the proposed bilinear decomposition increased data efficiency and improves generalization. The topic of achieving perfect disentanglement is outside the scope of current work and a worthwhile direction for future research.
5.3 ABLATION STUDIES
How does latent dimension affect our method? Though the dimensions of the latent space (i.e., output dimension of and Φ) are set as 16 across all the experiments above, we investigate the ef- fect of the dimensionality of the latent space the agent performance. Figure 7 shows the learning curve for BVN with 3, 4, 8 and 16 dimensional latent spaces. The performance consistently im- proves as latent dimensions increase, and even BVN with just a 3-dimensional latent space still outperforms the monolithic UVFA baseline. Hence bilinear value decomposition produces consis- tent gain with respect the monolithic UVFA baseline regardless of the latent dimensions, and benefits performance-wis wise from a larger latent space.
As it is unclear if the dot- dot-product is the best choice model the interaction between two vector fields f and o, we compare two alternative ways to decompose the value function using the same variable grouping the BVN. In the first variant,
Dot product ". e Metric I1
replace the dot-product with an e metric || l12. and parameterize the Q function via Q(s, a, g) = -I]
ø(s, 22 The negative sign is added because multi-goal RL tasks typically assume the reward is within the range [-1, 0). We set the latent dimension to be three as it is the best one found through hyperparameter search (see Section A.4). In the second variant we linearly combine the two latent vectors: (s,, a, g) = w f.(s, a) (x g), where x' y; implics a sum over the indices implicitly. This is the Einstein notation, or "einsum". The results shown in Figure 8 shows that both the bilinear

---

Published as a conference paper at ICLR 2022
Push
Pick Place
Push
Place
100%
75%
75%
Succce
50%
50%
25%
0%
0
25
50
75
100
125
150
30
100
150
200
100
150
200
Epech
Epoch
Epoch
Epoch
MLP(s,
BVN(d=8)
BVN(d=3)
wftti'g
16N(d=16) BVN(d-4)
Figure 7: Increasing the latent dimension of f and & improves our method's performance. De- spite that, the result shows that low dimensional latent space (d 3) is sufficient to enable our method to outperform the baseline UVFA (Mon- lithic).
Figure 8: Replacing the dot- dot-product with an l norm does not affect the final performance or the sample complexity. The initial performance is af- fected marginally. Whereas the linear combina- tion is too restrictive.
and the metric decomposition attain good performance. The e² metric decomposition marginally slower during the initial learning phase. The linear combination works poorly, likely due to the restricted expressivity.
Alternate Bi-Linear Decomposition Lastly, we investigate if there other viable ways to de- compose the Q-function based on grouping the inputs a. An alternative hypothesis is that any grouping of the inputs arguments (i.c., s, a, could improve the sample efficiency. To test this is the case, we compare our bilinear value decomposition scheme against two alternatives, f a) p(a. g) and (s, a. g) "(g), as well as the scheme f(s, a) T "((g) shown in Section 5.1. The results in Figure 9 show that neither matches the performance of the bilinear value network, which suggests correct grouping does play a crucial role. is important to process s and together and duplicate the state input on both branches.
Push
Slide
Pick & Place
100%
75%
Sucoess
50%
25%
0%
0
25
30
75
100
125
150
100
200
300
400
500
50
100
130
200
Epoch
Epoch
Epoch
fss a) ((g) - - ((g)
Ms, a) p(a.g)
- a) ((s.g)
Figure 9: Comparison between alternative factorizations show that our bilinear value decomposition scheme, fs. a) ((s, g), outperforms these alternatives. This suggests that this particular input grouping is crucial.
RELATED WORKS
A large number of prior works have explored ways to decompose value function to improve sample efficiency of value function approximation. Classic works (Sutton & Barto, 2018; Littman et al., 2001; Singh et al., 2003) approximate the value function in set of linear basis. Successor fea- tures (Dayan, 1993; Kulkarni et al., 2016; Borsa et al., 2019) factorize the value function as the dot product between expectation of state-action feature occupancy and vector of reward coefficients to the features.r Laplacian reinforcement learning (Mahadevan & Maggioni, 2006; 2007) represents the value function using set of Fourier basis. The above previous works investigate value function decomposition in single task learning. Instead, bilinear value networks focused on decomposing a multi-goal value function by disentangling the representations for goals and actions. Also, bilinear
8

---

Published as a conference paper at ICLR 2022
value networks end-to-end training to learn the decomposition instead of separating the basis learning and the value approximation. To summarize prior works and their choice parameteriza- tion, we compile key publications in Table I, ordered by the year of publication, in the appendix.
CONCLUSION AND DISCUSSION
Our empirical evaluation shows significant sample efficiency gain when bilinear value networks is used with multi-goal RL tasks. Moreover, bilinear value networks does not introduce extra hyper- paramters for training, except for the output dimensions of and Φ. Our ablations demonstrate that the benefits of BVN result from faster adaptation of the value function to new goals. An exciting direction for future research to facilitate transfer across different robots' morphology and goal spaces. One could keep the state and goal spaces fixed and train single ((s.g) g) that is shared be- tween the different morphologies. The difference between morphologies could be accommodated by different (e.g., different manipulators for the same task). Also, can pre-train shared f for the same robot and train ø for different tasks (i.c., goal spaces). In addition, we further discuss the additional implications of our experimental results in Section C of appendix.
ACKNOWLEDGMENTS
We thank members of Improbable AI Lab for the helpful discussion and feedback. We are grate- ful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC re- sources. The research this paper was supported by the MIT-IBM Watson Al Lab, part by the Army Research Office and was accomplished under Grant Number W91INF-21-1-033 and by the National Science Foundation AI Institute for Artificial Intelligence and Fundamental Interac- under Cooperative Agreement PHY-2019786. The views and conclusions
tions htttp:://aiif..rr contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REPRODUCIBILITY STATEMENT
We provide detailed instructions for reproducing the results this paper in the Appendix. Please refer Section Section A.4.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems, 2017.
1, 4, 5,
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. In Proceedings of the International Conference Learning Representations, 2019. 8, 11
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Comput., 5(4):613-624 1993. ISSN 0899-7667. 8
Thomas DiCiccio and Bradley Efron. Bootstrap confidence intervals. Staristical science, 11(3):
189-228, 1996. 5
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Proceedings of the International Conference Machine Learning, 2018.
3, 5, 14
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference Machine Learning, 2018. 3, 5, 14

---

Published as a conference paper at ICLR 2022
Leslie Pack Kaelbling. Learning to achieve goals. In Proceedings of the International Joint Confer- ence on Artificial Intelligence, 1993. 1,2
Tejas D. Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J. Gershman. Deep successor reinforcement learning. ArXiv, (as//1666023966 2016. 8, 11
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Pro- ceedings of the International Conference on Learning Representations, 2016.
M Littman, R Sutton, and Satinder Singh. Predictive representations of state. 2001. 8, 11
Sridhar Mahadevan and Mauro Maggioni. Value function approximation with diffusion wavelets and laplacian eigenfunctions. Advances in Neural Information Processing Systemx, 2006. 8,
Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A laplacian framework for learn- ing representation and control in markov decision processes. Journal of Machine Learning Re- search, 2007. 8, 11
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, al. Learning navigate in complex environments. Proceedings of the International Conference on Representation Learning, 2017.
11
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain ex- ploration for long horizon multi-goal reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2020. 14
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics enviromments and request for research. arXiv preprint arXiv:1802.09460 09402. 2018. 2, 11, 12, 14, 15
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In Proceedings of the International Conference Machine Learning, 2015. 1, 3, 4, 5, 11,
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, 2014. 3
Satinder P Singh, Michael L Littman, Nicholas K Jong, David Pardoe, and Peter Stone. Learning predictive state representations. Proceedings the International Conference on Machine Learn- ing, 2003. 8, 11
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, Oc- tober 2018. ISBN 9780262352703. URL hattps://play. http::///laaa..ooglee. gwwgle.oooooooooooooooooooo.. htt.g/l..ccmm/eeeoeeeeeeeeeeeeee store//tr////oocccocoooooooceeeo haaan//ttoe/bbooko /bokks/ 8, 11
details? 1tw.11/s...duW////ARAA...
Richard S Sutton, Joseph Modayil, Michael Delp. Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, 2011. 1. 11
Christopher JCH Watkins and Peter Dayan. Qelerning. Machine learning, 8(3):279-292 1992.
10

---

Published as a conference paper at ICLR 2022
Table Comprehensive list of linear value decomposition schemes from the literature.
Reference
Parameterization
Linear basis (Sutton & Barto; Littman al.; Singh et al.)
Q(s.a) = [ (s
Proto-Value Function (Mahadevan & Maggioni, 2006; 2007)
V(s) E (s)
General Value Approximator (GVF/Horde, Sutton et al. 2011)
(s, 0) o(s,
Successor Representation (SR, Kulkarni et al. 2016)
Q(s.a) = Ms, s')('s
Monolithic MLP UVFA (Schaul et al.: Andrychowicz et al.)
(s.((s,a.)
Recurrent UVFA (Mirowski et al., 2017)
RNN(2..,, 2x. 2g are embeddings
Low-rank Bilinear UVFA (Schaul et al., 2015)
Q(s.a.g) [(s w(g)
USFA (Borsa et al., 2019)
Q(s,a,2) = f( v
Bilinear Value Network (ours)
Q(s,a.g) (8, (s,
A APPENDIX
A.1 DETAILS ON PREVIOUS LINEAR DECOMPOSITION SCHEMES
We present the comparison of parameterizzations in Table 1.
A.2 IMPLEMENTATION DETAILS OF TASKS
All experiments in Section 5.1 happen the standard object and dexterous manipulation tasks from the gym robotics suite (Plappert al., 2018).
We briefly summarize the specifications of the tasks in the list below and illustrate these tasks in Figure 10:
Table 2: Environment Specifications.
Fetch Robot
Shadow Hand
Task
S
g
Task
A
g
Reach
R10
R3
R3
BlockRotate7
R61
R20
R7
Push
R25
R3
Ra
BlockRotateParallel
RG1
R2D
PickAndPlace
R25
R4
Ra
BlockRotateXYZ
R61
R
R7
Slide
R46
Ra
Ra
BlockRotate
RG1
R20
DrawerOpen
22
R3
R3
Egg
R61
R20
R7
DrawerClose
R25
R3
R3
EggRotate
R61
R20
R7
Pen
R
R20
RT
PenRotate
R61
R20
R7
In addition to (Plappert al., 2018), we also use the tasks from Fetch extension For left/right environments used Section 5.2, the goal (x. y, 2) with x are left goals, and x 2 right goals. In near/far environments, we set the distance thresholds as 0.2 for Slide, 0.2 for Pick AndPlac,, 0.1 for Reach, and 0.15 for Push. The goals with distance below the threshold are considered as near goals, otherwise far goals.
A.3 TRAINING HYPERPARAMETERS
Fetch We adapted the hyperparameters in (Plappert et al., 2018) for training in single desktop. In Plappert et al. (2018), the agent is trained by multiple workers collecting data in parallel, where each worker collects rollouts using the same policy with different random seeds. The performances of the agent trained by our hyperparameters match that in (Plappert et al., 2018).
num_workers: 2 for Reach, for Push, 16 for Pick & Place, and 20 for Slide.
Batch size: 1024
11

---

Published as a conference paper at ICLR 2022
Figure 10: Renderings of the fetch robot and shadow hand domains considered this work. Left- to-right: (top) Fetch push, pick-and-place, slide, bin-pick, box-open; (bottom) Shadow hand block, egg, pen. For Fetch, the goal to move the gray object the dot by controlling the arm. For Shadow hand, the objective is to reorient the object (e.g.. block, egg, and pen) to the desired poses rendered by the transparent object.
Table 3: The success rates of each method in the pretrained tasks used in Section 5.2.
Left
Near
Push
Pick & Place
Slide
Push
Pick & Place
Slide
UVFA (Monolithic)
96%
50% 59%
81% 48%
93%
91% 96%
96%
BVN
96%
95%
90%
Warm up rollouts:
We collected 100 initial rollouts for prefilling the replay buffer. Training frequency: We train the agent per environment steps. Each iteration of training uses the num_workers batches for training.
The of hyperparameters remain the same that used in (Plappert al., 2018).
Shadow hand We the hyperparameters used in (Plappert al., 2018) and reuse the open source ?ode?.
Adaptation experiments (Section 5.2) We present the success rates after pretraining in Table 3. Note that in the pretraining stage of each method and task, trained the agents with different random seeds and took the best-performing models as the initialization at the fine-tuning time.
A.4 IMPLEMENTATION DETAILS
Q-function architecture The monolithic network baselines in this paper represents the Q-function multi-layer perceptron (MLP) network that takes in a, and g concatenated together single vector. The bilinear UVFA (Schaul et al., 2015) uses an MLP f that takes the concatenated and a as inputs, and an MLP é using g as inputs. Our biliniear value network on the other hand, concatenate [s, a] together as the input to f, and [s, g] as the inputs of ø.
Policy architecture
We use an MLP as the policy network for all the methods.
For method and its variants presented in Section 5.3
the following nomenclature
denote them:
1. Dot product This our proposed method. We set the outputs' dimension of f and Φ as 16.
12

---

Published as a conference paper at ICLR 2022
2. 2-norm We parametrize Q-function Q(s, [[((s ((s.g) We set the dimen- sion of the outputs from f and & as 3. Figure 11 shows that works the best for 2-norm.
3. concat We concatenate f(s, a) and o(s, g). feed the concatenated vectors to an MLP layer, and take the outputs this layer as the Q-value Q(s,, a, g). The dimensions of the outputs of f and are 16.
Push
Pick Place
100%
Success rate
UVFA
75%
d=16
50%
d=8
25%
d=4
0%
d=3
50
75
100
125
150
200
Epoch
Epoch
Figure 11: 3-dimensional latent space works the best for 2-norm variant (see Section A.4).
A.5 TRANSFERRING ACROSS TASKS
In multi-goal RL tasks, observe that most of tasks identical in terms of high-level goals, but they have different environment dynamics. For instance, the objective of "Push" and "Slide" are all to move the object to the destination. They only differ in the friction coefficient of the table where the agent cannot slide objects in "Push" because of high friction. Because we decouple the state-action and goals into two different functions, [(s, a) and ((s,g), one potential benefit on multi-gon RL is reusing & for transferring to tasks with the same (or similar) task objectives (i.e., reward specification). Reusing d(s. g) exploits the learned reward-specific state-goal relations. Then we simply need to learn new to recover the reward-agnostic relations in the new task. To examine this hypothesis, we select three tasks with the same state and action spaces", and measure the performance of transferring amongst these tasks. In transferring an agent from a source task to a target task, we initialize UVFA (Monlithic) and BVN (no reset) agent's 's policy and Q-function by the neural networks' weights learned in the source task and fine-tune them in the target task. For BVN (reset f), we reinitialize f and fine-tune ø when training them in the target task. Figure 12 shows the success rates of transferring from source task to target task. For example, "Push to Slide" indicates transferring from "Push" to "Slide". It can be seen that BVN (resel l') attains higher success rates convergence than the baselines and BVN (no reset) in 4 out 6 tasks (i.e., "Push to Slide". "Pick & Place to Push", "Push to Pick & Place", and "Pick & Place to Slide").
The limited efficiency gain could result from that Φ is dependent on states and thus & cannot be effectively reused in new tasks. Even though the target task's 's reward function is the same as that in the source task, the relation of states and goals can be affected by the dynamics. For instance, suppose we have state where the object is near the goal. In "Pick & Place", because the friction coefficients the table are high, the Φ(s, (s g) should direct the agent lift the object. On the other hand, in "Slide", the ((s g) is expected to guide the agent to push the object to slide.
IIf the two tasks have different state/action spaces, will not be able to use the weights from task and resume training another task.
13

---

Published as a conference paper at ICLR 2022
Pick&Place => Push
Push => Plck&Place
Pick&Place => Slide
100%
100%
Success Rate
Shecess Rate
Success Rate
75%
75%
75%
50%
50%
50%
25%
25%
25%
0%
0%
0%
100
125
150
50
100
150
200
100
150
200
Epoch
Epoch
Epoch
Push => Slide
Slide => Push
Slide Pick&Place
100%
100%
Success Rate
Success Rate
Success Rate
75%
75%
75%
50%
25%
25%
25%
096
0%
0%
50
100
150
200
50
75 Epoch
100
125
150
200
Epoch
Epoch
MLP (no reaet)
BVN reset)
BVN (reaet f
Figure 12: The two-stream architecture of the BVN allows us to improve transfer by resetting f. Re- setting f and fine-tuning ø achieve higher success rate than the baselines when transferring between push and pick & place, and to slide.
Reach
Push
Slide
Pick & Place
100%
75%
50%
106
216
306:
406
506:
100k
2006
300k
400k
500k
500%
1.0M Stops
1.5M
2.00
500k
1.b
1.5M
Steps
Stops
Steps
DDPG + BVN
DDPG
MLP
TD3 MLP
MLP SAC
Figure 13: We show that our BVN improves the sample efficiency over the best baseline, DDPG + MLP, in commonly used environments in multi-goal RL.
B COMPARISON AGAINST SOTA ALGORITHMS
Bilinear value networks achieves State-Of-Ther Art (SOTA) performance in common multi-goal re- inforcement learning tasks (Plappert et al., 2018). In figure 13, compare bilinear value networks against two strong baselines: soft actor critic (SAC, see Haarnoja et al. 2018) and twin-delayed deep deterministic policy gradient (TD3, Fujimoto al. 2018). We reuse the hyperparameters reported Fujimoto et al. 2018) and implement TD3 in our codebase. As for SAC, we use the implementation in this codebase Figure 13 shows the performance of each baseline algorithm and our method (DDPG + BVN), We show that our DDPG + BVN outperforms the best baseline, and find that DDPG works the best among all the MLP variants. Our findings align with (Pitis al., 2020) and the documentation in its released code'. Pitis et al. (2020) also found that DDPG works better than TD3 and SAC in multi-goal RL domains.
C ADDITIONAL DISCUSSION
Why does other parameter grouping schemes fail? In Section 5.3, showed that f(s, a. g)" ((g) inputs grouping fails to improve sample efficiency. One explanation to this result that f(s a. cannot prevent negative interference from new goals encountered during training
hhaps://githhhbccm///isttttt
14

---

Published as a conference paper at ICLR 2022
since goals g couple with states-actions As result, the merit of bi-linear value function decom- position vanishes and even causes extra burden on learning because the number model parameters larger than UVFA (Monlithic). This preliminary finding shows that to benefit from decomposition, the guideline is to only expose necessary inputs to each component. Redundant inputs either component could break the decomposition.
How is 2-norm compared with dot product? Section 5.3 demonstrated that using 2-norm to reduce f(s, a) and c(s.g) into Q-value matches the performance of our BVN with even lower dimensions of 7 and Z However, 2-norm relies assumption that the reward function must be non-positive or non-negative since norm defined as non-negative value. The [I a) áss, g)ll can only fit to either non non-positive non-negative Q-value by manipulating the sign of norm. On the other hand, dot product is more expressive and free from the reward function assumption.
Can method facilitate policy transfer? Since method decomposes the value function into a local (i.e., goal-agnostic) and a global (i.e., goal-specific) component, our BVN might facilitate policy transfer by resetting either one of components and fine-tuning another one. As multi-goal tasks considered in (Plappert et al., 2018) have the same objective in nature but different dynamics, one potential advantage our method to reuse ((s, g) in the new task and re-train f(s a). The results in Section A.5 showed that the agent transferred by BVN converges to the the performance of training from scratch the target task slightly faster than the baselines. One reason for limited policy transferring performance is that Φ(8, g) conditions on states 8. Consequently, environment dynamics shifts could affect the representations of ø. We, nevertheless, showed that states variables crucial in d. One exciting avenue is disentangle the representations for keeping the rich expressivity as well as desired modularity for transferring across tasks.
15


Reviews:
[{"review_id": "b1oQIwG2DkF", "reviewer": "Reviewer_QczY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Main Review\nThis paper studies the structure of the function approximators used to learn goal-conditioned policies.\n\n### Strengths\n* The paper presents a valid hypothesis that a structure that separates the task-specific information from the action-specific effects should lead to better learning. Their rationalization for this hypothesis is also clear.\n* The particular approach this paper takes to effect such a separation is fairly straightforward.\n* The difference compared to the bi-linear approach suggested in [1] is also clear.\n* The experiments validate one part of the hypothesis by showcasing faster learning with the separated networks on a few of the domains tested on, as well as better transfer to goals in parts of the state space that the agent has not trained on.\n\n### Weaknesses\n* The paper presents a clear hypothesis for how they expect the separate networks to learn different aspects of the problem, but then do not construct or find a simpler domain where this hypothesis can be evaluated. Specifically, seeing an experiments that learns a vector field like Fig 2 (b) in actuality would have strengthened the case for the paper quite a bit.\n* The idea of separating the learning into what can be accomplished by the agent in a given state and what needs to be done for a given goal could be generalized and analyzed more carefully and deeply than has been done in this paper. As it stands, this paper shows some interesting empirical results with a new network architecture and presents a credible hypothesis for why this architecture might be helping learning, but does not dig deep enough to validate the hypothesis. It also does not attempt to generalize and extract insights beyond what are applicable to the few domains that it evaluates on.\n* The paper mentions that the value network is decomposed into the bilinear networks the paper studies. But how are the policy networks handled? Are they keeping their monolithic structure? If so, why? These questions are not answered in the paper effectively.\n\n### Minor issues\n* The problem setup seems imprecise. A reader not familiar with reinforcement learning might not be able to parse this section. Specifically the structure of the reward function is not clarified, not to mention the range of discounting is unclear.\n* The writing of the paper is a bit imprecise in other places as well. In the experimental section, it refers to [2] as UVFA and [1] as a bilinear network, while the title of paper [1] is \"Universal Value Function Approximators\". It does so in other parts of the paper as well, where the implication becomes that a UVFA has to be a monolithic network.\n* The discussion section seems like an extension of the experimental section. There is a variety of possible generalizations and implications of this work that should be discussed which are ignored in the discussion section for going over a few more experimental details.\n* The attribution of [2] should be for the NeurIPS version, rather than the ArXiv one.\n\n### Other points\n* The idea of separating what can be done in a state and what needs to be done for a task bears some resemblance to the Q(s, s') learning in [3]. Perhaps it might be useful for discussion.\n\n## References\n[1] Universal value function approximators, Schaul et al., ICML 2015\n[2] Hindsight Experience Replay, Andrychowicz et al., NeurIPS 2017.\n[3] Estimating Q (s, s') with Deep Deterministic Dynamics Gradients, Edwards et al., ICML 2020.", "summary_of_the_paper": "This paper considers the problem of learning a goal-conditioned policy effectively. Particularly, it looks at the architecture of the value network used to learn this goal-conditioned policy if using a deterministic policy gradient algorithm.\nIt studies whether separating the goal-conditioned value network into two components, a state-action embedding network and a state-goal embedding network, might lead to faster learning and better generalization of the policy. The hypothesis such a study tests is whether the separation can tease apart what an agent's actions in a particular state accomplish and what the agent needs to accomplish to get to a goal from a given state. This hypothesis is evaluated by conducting experiments on the Fetch and Hand domains and evaluating how quickly the goal-conditioned policy is learned and whether it generalizes to goals in parts of the state space that it was not trained on.", "summary_of_the_review": "The paper presents a valid hypothesis: that separating what the effect of an action will be in a given state and what the agent needs to do to solve a task should be separated while learning in order to aid the speed of learning as well as generalization. To do so the paper presents the separation of the value network into two networks whose vector predictions are combined to predict the value.\nThe downstream effects of this new architecture are validated experimentally fairly well. However, whether the new architecture actually learns the hypothesized separation is not evaluated. Additionally, the writing can be tightened up a bit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T5_aUkc4jcR", "reviewer": "Reviewer_vt3X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper proposes a modification to UVFAs that split them into two networks, each responsible for learning a particular \"concept\". On the one hand, the approach is a slight modification to the kind of architecture proposed in Schaul et al (2015), which the paper acknowledges. Having said that, the interpretation of the architecture is certainly interesting and, to the best of my knowledge, novel. Furthermore, in this instance, I feel like the simplicity and small change required is actually a strength of the paper, since the approach will be easy to integrate into existing codebases, which will help with adoption.  \n\nOne particular thought I had when reading the paper that continues to nag at me is the relative contribution of the two vectors. Intuitively, it feels like $\\phi$ is much more important than $f$. For example, in Figure 2, $f$ directs the agent passed the obstacle. While it makes sense in general not to collide with an obstacle, why should that be a preferred direction. The goal could be to the left of the agent instead, which would mean it should actually be moving away from the obstacle, and not around it. This would obviously require $\\phi$ to do all the \"heavy lifting\" but I can't help wondering when $\\phi$ isn't the vector that is most important. As a simple example, we could imagine an unbounded 2D plane with some goal. Then the direction proposed by $\\phi$ is the only thing that actually matters and $f$ can be completely ignored. I may be missing something, but I feel (at least in the navigation case which is easier to reason about) that this would generally be the case.  It would be great to have some kind of experiment (even in a toy domain) that speaks to this. I'm imagining something where an agent is trained and then the average magnitude (or something like that) of the vectors is recorded to try measure the influence of each component.\n\nAlong similar lines, it would also be interesting to see a setup (again, this could be a synthetic environment) where $\\phi$ directs the agent towards the goal, but that's actually the wrong thing to do (because there is a dead-end or pit of lava or something to that effect). I would be curious to see the behaviour elicited in this case.\n\nWhile the experiments were generally well-executed, I feel like they could have benefited from at least one more baseline. While the comparison to UVFAs is natural, it would have been nice to see how it performed against more recent approaches like SAC (Levy et al, 2019) or HIRO (Nachum et al, 2019)\n\n\nThough it is not directly related, the approach of dividing into the two concepts of \"where to go\" and \"what to do\" put me in mind of a number of different goal-oriented approaches and it may be worth including a discussion of some in the related work.  For example, the separation of concerns is reminiscent of Edwards et al (2020) who learn a state-next state value function and then subsequently learn how to act to reach that. A slightly different take is to have a value function propose a subgoal region to visit, and then a controller that attempts to achieve that (Levy et al, 2019; Nachum et al, 2018; Hejna et al, 2020; Christen et al, 2021)\n\nMinor comments:\n\n  1. It is very difficult to see the vector field in Figures 1 and 2 because the arrows are a very light grey\n  2. Figure 3 appears before Figure 2\n  3. \\citep is used in the first sentence of Section 2.1 instead of \\citet\n  4. Last sentence on page 5: \"Euclidean\"\n\nReferences\n\n1. Levy, Andrew, et al. \"Learning Multi-Level Hierarchies with Hindsight.\" International Conference on Learning Representations. 2019.\n2. Nachum, Ofir, et al. \"Near-Optimal Representation Learning for Hierarchical Reinforcement Learning.\" International Conference on Learning Representations. 2019\n3. Edwards, Ashley, et al. \"Estimating q (s, s’) with deep deterministic dynamics gradients.\" International Conference on Machine Learning. PMLR, 2020.\n4. Hejna, Donald, Lerrel Pinto, and Pieter Abbeel. \"Hierarchically decoupled imitation for morphological transfer.\" International Conference on Machine Learning. PMLR, 2020.\n5. Christen, Sammy, et al. \"Learning functionally decomposed hierarchies for continuous control tasks with path planning.\" IEEE Robotics and Automation Letters 6.2 (2021): 3623-3630.\n", "summary_of_the_paper": "The paper proposes a modification to goal-oriented universal value functions that split the neural network into two parts. One network accepts states and actions and outputs a vector representation, while the other outputs a vector given a state and goal. Part of the contribution of the paper is the interpretation of these two vectors --- the first captures the notion of what actions can be taken at a given state, while the second represents the direction the agent should move towards to achieve the goal from its current state.  Empirical results on several simulated robotic domains demonstrate that the method outperforms other UVFA architectures, while ablations are provided to capture the most important aspects of the method.", "summary_of_the_review": "Though the actual method proposed is fairly incremental, the interpretation of it is novel and its simplicity means it could likely be incorporated into the goal-oriented learning toolkit quite easily. The experiments were all well executed and showed improved empirical performance, and ablation studies are provided to help understand exactly which design decisions are responsible for improved performance.\n\n***********************\nPOST REBUTTAL\n***********************\n\nThe newly-added experiment in 4.5 strengthens the paper in my view, since it highlights the particularities of the method. I think the approach is an interesting one and could be valuable to the existing body of work on goal-oriented learning. It is also conceptually simple enough that it could easily be widely adopted and used as baselines or built upon going forward", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IUgH8NmUp16", "reviewer": "Reviewer_AnrU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPRO\n\n* well written and easy to follow\n* simple method with good motivation\n* good ablation studies and discussion for the different aspects of the proposed method\n\nCONS\n* it's not discussed why the method performs worse or on par in some cases\n* only tests the method with DDPG and in two domains. So it's unclear how general the proposal is\n\nThere are a number of open questions\n\nI would have hoped to see a more general evaluation though as essentially this seems to be a very general method that could plugin to value function approaches more generally but also into different actor-critic methods. Why did you choose to focus only on two domains and DDPG? Does the approach generalize to other domains and other actor-critic MGRL methods?\n\nIs it important that this is bilinear? \n\nWhat are the NN dimensions of the value networks? Does a change here have any effect? \n\nIs adding more structure like an MLP really fair? Clearly you add a lot of structure through the bi-linear constraint. So just making it more complex and adding an MLP doesn't really address the question of whether the larger model is giving the performance gain", "summary_of_the_paper": "The paper proposes a new approach to Universal Value Functions in Reinforcement Learning by decomposing monolithic universal value functions into two components and exploring different ways of combining the results of the decomposition. The paper shows that this leads to better sample efficiency and has desirable properties for generalization.", "summary_of_the_review": "The paper is well written and does tackle an interesting and important part of Reinforcement Learning.\nThe idea presented is quite simple but does show promising results in the experiments.\n\nAuthors do a good job of ablating and discussing different choices of the systems\n\nSome information seems missing around generality of the method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E8Hy9GWs4ZM", "reviewer": "Reviewer_ZNRs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is proposed modification of the universal value function approximation implementation by making the goal-dependent component also depend on the state so that now it is f(s,a)*\\phi(s,g) instead of f(s,a)*\\phi(g). The improvement is simple and well motivated. However, the method is not explained very well. After the idea is presented in section 3, nothing is said about how the method is implemented (how the functions are parameterized and trained?). Are f and \\phi the networks? What kind of networks? How states, actions and goals are processed in them? As a consequence, it is also not clear to me how the comparison to the baselines is done. For example, is the number of training parameters the same? Besides, it would be informative to include the information about the state, action and goal representations and their dimensions as it would help to understand the experiments like the ones in Figure 9.\n\nMy other concern is the novelty of the proposed method. It seems to me that the decompositions into a global component that depends on goal and state and local that looks at state and action has been already used in the literature (maybe without stating it as the main contribution). For example, see such decomposition is use for transfer between navigation tasks as in the network structure of “Learning to Navigate in Cities Without a Map” by Mirowski et al. 2019 (figure 2b). Could the authors comment on the similarities and differences between these methods? How would the proposed method perform compared to such a baseline?\n\nThe main experiments clearly demonstrate the usefulness of the method for the purpose of data-efficiency and transfer to unseen goals. The analysis of the method is useful, but does not provide sufficient light into what kind of decompositions are learnt. Does the training always result in similar decomposition? How can it be illustrated that the decomposition has similar meaning as said in the motivations (figure 2)? One of the ablation studies that is supposed to answer the question “Does the performance gain result from a larger model?” does not answer that question in my opinion. That experiment shows that the particular architecture modification -- concatenation of the outputs of f and \\phi followed by an MLP -- does not further improve the performance. To answer the original question the authors should: 1) provide information about the number of training parameters in the baseline methods and the proposed method, 2) try to increase the number of parameters without architecture modifications (for example, by increasing the number of hidden units in the existing layers). The ablations with the decomposition f(s,a,g)*\\phi(g) seems to me not very informative because such a baseline is quite logically can’t improve the performance as it contains the full UVFA f(s,a,g) which is then multiplied by the value depending on the goal which does not have any particular motivation. Also, maybe other decompositions could be considered, such as f(s,a)*\\phi(a,g)?\n\nPros:\n\n- The proposed idea is simple and elegant.\n\n- The environments and the design of the experiments chosen for a study seem to be interesting and challenging.\n\n- The benefits of the method in terms of data efficiency and transfer to new goals are clear.\n\nCons:\n\n- Details about the implementation of the method and comparison to the baselines are missing.\n\n- Novelty of the method might be limited given that some prior work already looked at decomposing learning of the local component (depending on state and action) and global component (depending on state and goal).\n\n- The analysis of the method is not convincing. Moreover, one of the ablation studies seems to me to be incorrectly designed and another one not very informative.\n\nOther comments:\n\n- Some phrases are unclear or not well written, for example “transfer to new goals type of transfer to new goals”, “generalizability improves sample efficiency”.\n\n- The references of the paper need formatting.\n", "summary_of_the_paper": "This paper proposes a new decomposition for the universal value function that disentangles local and global components. The global component depends on the goal and state and tells the agent “where to go” and the local component depends on the state and action and tells the agent “what to do”. The authors conduct a set of experiments on a number of environments to demonstrate that this decomposition results in more data efficiency and helps to generalise to the new goals.", "summary_of_the_review": "The proposed method is simple and elegant, but it is unclear to me how it relates to some of the previous literature that proposed similar decomposition through the network structure. Some details of the method implementation and experiments are missing. The main experiments are convincing and well designed, but the analysis is not sufficient and does not provide enough insight into the behaviour of the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new bilinear decomposition for universal value functions.  The bilinear network has one component dependent on state and goal and another component that depends on state and action.  The experiments with the DDPG algorithm in robot simulations show that the proposed architecture improves performance data efficiency and task transfer over several baseline algorithms, including improvements on earlier bilinear decompositions.

The reviews noted several aspects of the paper could be improved, and the author response addressed several of these concerns. Multiple reviewers appreciated the insights from the experiment added in section 4.5 on a simple grid environment, which enabled a direct interpretation of the vector fields used in the method.  Several aspects of the presentation were clarified based on the reviewers comments. Additional details were also provided on the problem specification and the solution methods. During the discussion, the reviewers agreed that the revised paper presented a useful addition to the literature.

Four knowledgeable reviewers indicate to accept the paper for its contribution of an effective network architecture for a goal-conditioned universal value function approximator.  The paper is therefore accepted.
Decision: Accept (Poster)