Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new approach to Universal Value Functions in Reinforcement Learning by decomposing monolithic universal value functions into two components and exploring different ways of combining the results of the decomposition. The paper shows that this leads to better sample efficiency and has desirable properties for generalization.
main_review: 
PRO

* well written and easy to follow
* simple method with good motivation
* good ablation studies and discussion for the different aspects of the proposed method

CONS
* it's not discussed why the method performs worse or on par in some cases
* only tests the method with DDPG and in two domains. So it's unclear how general the proposal is

There are a number of open questions

I would have hoped to see a more general evaluation though as essentially this seems to be a very general method that could plugin to value function approaches more generally but also into different actor-critic methods. Why did you choose to focus only on two domains and DDPG? Does the approach generalize to other domains and other actor-critic MGRL methods?

Is it important that this is bilinear? 

What are the NN dimensions of the value networks? Does a change here have any effect? 

Is adding more structure like an MLP really fair? Clearly you add a lot of structure through the bi-linear constraint. So just making it more complex and adding an MLP doesn't really address the question of whether the larger model is giving the performance gain
summary_of_the_review: The paper is well written and does tackle an interesting and important part of Reinforcement Learning.
The idea presented is quite simple but does show promising results in the experiments.

Authors do a good job of ablating and discussing different choices of the systems

Some information seems missing around generality of the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new decomposition for the universal value function that disentangles local and global components. The global component depends on the goal and state and tells the agent “where to go” and the local component depends on the state and action and tells the agent “what to do”. The authors conduct a set of experiments on a number of environments to demonstrate that this decomposition results in more data efficiency and helps to generalise to the new goals.
main_review: The main contribution of the paper is proposed modification of the universal value function approximation implementation by making the goal-dependent component also depend on the state so that now it is f(s,a)*\phi(s,g) instead of f(s,a)*\phi(g). The improvement is simple and well motivated. However, the method is not explained very well. After the idea is presented in section 3, nothing is said about how the method is implemented (how the functions are parameterized and trained?). Are f and \phi the networks? What kind of networks? How states, actions and goals are processed in them? As a consequence, it is also not clear to me how the comparison to the baselines is done. For example, is the number of training parameters the same? Besides, it would be informative to include the information about the state, action and goal representations and their dimensions as it would help to understand the experiments like the ones in Figure 9.

My other concern is the novelty of the proposed method. It seems to me that the decompositions into a global component that depends on goal and state and local that looks at state and action has been already used in the literature (maybe without stating it as the main contribution). For example, see such decomposition is use for transfer between navigation tasks as in the network structure of “Learning to Navigate in Cities Without a Map” by Mirowski et al. 2019 (figure 2b). Could the authors comment on the similarities and differences between these methods? How would the proposed method perform compared to such a baseline?

The main experiments clearly demonstrate the usefulness of the method for the purpose of data-efficiency and transfer to unseen goals. The analysis of the method is useful, but does not provide sufficient light into what kind of decompositions are learnt. Does the training always result in similar decomposition? How can it be illustrated that the decomposition has similar meaning as said in the motivations (figure 2)? One of the ablation studies that is supposed to answer the question “Does the performance gain result from a larger model?” does not answer that question in my opinion. That experiment shows that the particular architecture modification -- concatenation of the outputs of f and \phi followed by an MLP -- does not further improve the performance. To answer the original question the authors should: 1) provide information about the number of training parameters in the baseline methods and the proposed method, 2) try to increase the number of parameters without architecture modifications (for example, by increasing the number of hidden units in the existing layers). The ablations with the decomposition f(s,a,g)*\phi(g) seems to me not very informative because such a baseline is quite logically can’t improve the performance as it contains the full UVFA f(s,a,g) which is then multiplied by the value depending on the goal which does not have any particular motivation. Also, maybe other decompositions could be considered, such as f(s,a)*\phi(a,g)?

Pros:

- The proposed idea is simple and elegant.

- The environments and the design of the experiments chosen for a study seem to be interesting and challenging.

- The benefits of the method in terms of data efficiency and transfer to new goals are clear.

Cons:

- Details about the implementation of the method and comparison to the baselines are missing.

- Novelty of the method might be limited given that some prior work already looked at decomposing learning of the local component (depending on state and action) and global component (depending on state and goal).

- The analysis of the method is not convincing. Moreover, one of the ablation studies seems to me to be incorrectly designed and another one not very informative.

Other comments:

- Some phrases are unclear or not well written, for example “transfer to new goals type of transfer to new goals”, “generalizability improves sample efficiency”.

- The references of the paper need formatting.

summary_of_the_review: The proposed method is simple and elegant, but it is unclear to me how it relates to some of the previous literature that proposed similar decomposition through the network structure. Some details of the method implementation and experiments are missing. The main experiments are convincing and well designed, but the analysis is not sufficient and does not provide enough insight into the behaviour of the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a modification to goal-oriented universal value functions that split the neural network into two parts. One network accepts states and actions and outputs a vector representation, while the other outputs a vector given a state and goal. Part of the contribution of the paper is the interpretation of these two vectors --- the first captures the notion of what actions can be taken at a given state, while the second represents the direction the agent should move towards to achieve the goal from its current state.  Empirical results on several simulated robotic domains demonstrate that the method outperforms other UVFA architectures, while ablations are provided to capture the most important aspects of the method.
main_review: 
The paper proposes a modification to UVFAs that split them into two networks, each responsible for learning a particular "concept". On the one hand, the approach is a slight modification to the kind of architecture proposed in Schaul et al (2015), which the paper acknowledges. Having said that, the interpretation of the architecture is certainly interesting and, to the best of my knowledge, novel. Furthermore, in this instance, I feel like the simplicity and small change required is actually a strength of the paper, since the approach will be easy to integrate into existing codebases, which will help with adoption.  

One particular thought I had when reading the paper that continues to nag at me is the relative contribution of the two vectors. Intuitively, it feels like $\phi$ is much more important than $f$. For example, in Figure 2, $f$ directs the agent passed the obstacle. While it makes sense in general not to collide with an obstacle, why should that be a preferred direction. The goal could be to the left of the agent instead, which would mean it should actually be moving away from the obstacle, and not around it. This would obviously require $\phi$ to do all the "heavy lifting" but I can't help wondering when $\phi$ isn't the vector that is most important. As a simple example, we could imagine an unbounded 2D plane with some goal. Then the direction proposed by $\phi$ is the only thing that actually matters and $f$ can be completely ignored. I may be missing something, but I feel (at least in the navigation case which is easier to reason about) that this would generally be the case.  It would be great to have some kind of experiment (even in a toy domain) that speaks to this. I'm imagining something where an agent is trained and then the average magnitude (or something like that) of the vectors is recorded to try measure the influence of each component.

Along similar lines, it would also be interesting to see a setup (again, this could be a synthetic environment) where $\phi$ directs the agent towards the goal, but that's actually the wrong thing to do (because there is a dead-end or pit of lava or something to that effect). I would be curious to see the behaviour elicited in this case.

While the experiments were generally well-executed, I feel like they could have benefited from at least one more baseline. While the comparison to UVFAs is natural, it would have been nice to see how it performed against more recent approaches like SAC (Levy et al, 2019) or HIRO (Nachum et al, 2019)


Though it is not directly related, the approach of dividing into the two concepts of "where to go" and "what to do" put me in mind of a number of different goal-oriented approaches and it may be worth including a discussion of some in the related work.  For example, the separation of concerns is reminiscent of Edwards et al (2020) who learn a state-next state value function and then subsequently learn how to act to reach that. A slightly different take is to have a value function propose a subgoal region to visit, and then a controller that attempts to achieve that (Levy et al, 2019; Nachum et al, 2018; Hejna et al, 2020; Christen et al, 2021)

Minor comments:

  1. It is very difficult to see the vector field in Figures 1 and 2 because the arrows are a very light grey
  2. Figure 3 appears before Figure 2
  3. \citep is used in the first sentence of Section 2.1 instead of \citet
  4. Last sentence on page 5: "Euclidean"

References

1. Levy, Andrew, et al. "Learning Multi-Level Hierarchies with Hindsight." International Conference on Learning Representations. 2019.
2. Nachum, Ofir, et al. "Near-Optimal Representation Learning for Hierarchical Reinforcement Learning." International Conference on Learning Representations. 2019
3. Edwards, Ashley, et al. "Estimating q (s, s’) with deep deterministic dynamics gradients." International Conference on Machine Learning. PMLR, 2020.
4. Hejna, Donald, Lerrel Pinto, and Pieter Abbeel. "Hierarchically decoupled imitation for morphological transfer." International Conference on Machine Learning. PMLR, 2020.
5. Christen, Sammy, et al. "Learning functionally decomposed hierarchies for continuous control tasks with path planning." IEEE Robotics and Automation Letters 6.2 (2021): 3623-3630.

summary_of_the_review: Though the actual method proposed is fairly incremental, the interpretation of it is novel and its simplicity means it could likely be incorporated into the goal-oriented learning toolkit quite easily. The experiments were all well executed and showed improved empirical performance, and ablation studies are provided to help understand exactly which design decisions are responsible for improved performance.

***********************
POST REBUTTAL
***********************

The newly-added experiment in 4.5 strengthens the paper in my view, since it highlights the particularities of the method. I think the approach is an interesting one and could be valuable to the existing body of work on goal-oriented learning. It is also conceptually simple enough that it could easily be widely adopted and used as baselines or built upon going forward
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the problem of learning a goal-conditioned policy effectively. Particularly, it looks at the architecture of the value network used to learn this goal-conditioned policy if using a deterministic policy gradient algorithm.
It studies whether separating the goal-conditioned value network into two components, a state-action embedding network and a state-goal embedding network, might lead to faster learning and better generalization of the policy. The hypothesis such a study tests is whether the separation can tease apart what an agent's actions in a particular state accomplish and what the agent needs to accomplish to get to a goal from a given state. This hypothesis is evaluated by conducting experiments on the Fetch and Hand domains and evaluating how quickly the goal-conditioned policy is learned and whether it generalizes to goals in parts of the state space that it was not trained on.
main_review: ## Main Review
This paper studies the structure of the function approximators used to learn goal-conditioned policies.

### Strengths
* The paper presents a valid hypothesis that a structure that separates the task-specific information from the action-specific effects should lead to better learning. Their rationalization for this hypothesis is also clear.
* The particular approach this paper takes to effect such a separation is fairly straightforward.
* The difference compared to the bi-linear approach suggested in [1] is also clear.
* The experiments validate one part of the hypothesis by showcasing faster learning with the separated networks on a few of the domains tested on, as well as better transfer to goals in parts of the state space that the agent has not trained on.

### Weaknesses
* The paper presents a clear hypothesis for how they expect the separate networks to learn different aspects of the problem, but then do not construct or find a simpler domain where this hypothesis can be evaluated. Specifically, seeing an experiments that learns a vector field like Fig 2 (b) in actuality would have strengthened the case for the paper quite a bit.
* The idea of separating the learning into what can be accomplished by the agent in a given state and what needs to be done for a given goal could be generalized and analyzed more carefully and deeply than has been done in this paper. As it stands, this paper shows some interesting empirical results with a new network architecture and presents a credible hypothesis for why this architecture might be helping learning, but does not dig deep enough to validate the hypothesis. It also does not attempt to generalize and extract insights beyond what are applicable to the few domains that it evaluates on.
* The paper mentions that the value network is decomposed into the bilinear networks the paper studies. But how are the policy networks handled? Are they keeping their monolithic structure? If so, why? These questions are not answered in the paper effectively.

### Minor issues
* The problem setup seems imprecise. A reader not familiar with reinforcement learning might not be able to parse this section. Specifically the structure of the reward function is not clarified, not to mention the range of discounting is unclear.
* The writing of the paper is a bit imprecise in other places as well. In the experimental section, it refers to [2] as UVFA and [1] as a bilinear network, while the title of paper [1] is "Universal Value Function Approximators". It does so in other parts of the paper as well, where the implication becomes that a UVFA has to be a monolithic network.
* The discussion section seems like an extension of the experimental section. There is a variety of possible generalizations and implications of this work that should be discussed which are ignored in the discussion section for going over a few more experimental details.
* The attribution of [2] should be for the NeurIPS version, rather than the ArXiv one.

### Other points
* The idea of separating what can be done in a state and what needs to be done for a task bears some resemblance to the Q(s, s') learning in [3]. Perhaps it might be useful for discussion.

## References
[1] Universal value function approximators, Schaul et al., ICML 2015
[2] Hindsight Experience Replay, Andrychowicz et al., NeurIPS 2017.
[3] Estimating Q (s, s') with Deep Deterministic Dynamics Gradients, Edwards et al., ICML 2020.
summary_of_the_review: The paper presents a valid hypothesis: that separating what the effect of an action will be in a given state and what the agent needs to do to solve a task should be separated while learning in order to aid the speed of learning as well as generalization. To do so the paper presents the separation of the value network into two networks whose vector predictions are combined to predict the value.
The downstream effects of this new architecture are validated experimentally fairly well. However, whether the new architecture actually learns the hypothesized separation is not evaluated. Additionally, the writing can be tightened up a bit.
review_title: null

Meta-review: This paper proposes a new bilinear decomposition for universal value functions.  The bilinear network has one component dependent on state and goal and another component that depends on state and action.  The experiments with the DDPG algorithm in robot simulations show that the proposed architecture improves performance data efficiency and task transfer over several baseline algorithms, including improvements on earlier bilinear decompositions.

The reviews noted several aspects of the paper could be improved, and the author response addressed several of these concerns. Multiple reviewers appreciated the insights from the experiment added in section 4.5 on a simple grid environment, which enabled a direct interpretation of the vector fields used in the method.  Several aspects of the presentation were clarified based on the reviewers comments. Additional details were also provided on the problem specification and the solution methods. During the discussion, the reviewers agreed that the revised paper presented a useful addition to the literature.

Four knowledgeable reviewers indicate to accept the paper for its contribution of an effective network architecture for a goal-conditioned universal value function approximator.  The paper is therefore accepted.
Decision: Accept (Poster)