Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes (i) a decomposition for successor features $\phi(s,a,s)^Tw(g)$ which allows the Q-value function to adapt to new goals easily and (ii) then incorporate the learned successor feature representation into an actor critic algorithm (SAC) for goal conditioned tasks (or for solving multiple tasks). The proposed method is tested on continuous control tasks.
main_review: **Strengths**: The authors proposed a principled method for incorporating successor feature into an actor-critic algorithm and the decomposition of successor features as $\phi(s,a,s)^Tw(g)$ allows it to adapt to new goals as shown through reacher envs.

**Concerns**: 
(i) The proposed decomposition $\phi(s,a,s)^Tw(g)$ feels very similar to that used in (Ma et. al, 2020). Moreover, (Ma et. al, 2020) also incorporates the learned successor features into an actor critic framework and uses it for continuous control tasks. Hence, I am not sure what novelty the proposed work is bringing.

(ii) The environments used in the experiments are limited (only reacher and door close tasks are considered). Moreover, the performance of the proposed method is very similar to the goal conditioned SAC baseline on meta world tasks (Figure 7)

(iii) At an intuitive level, if $\phi(s,a,s')$ is capturing the dynamics and $w(g)$ is capturing the reward information, shouldn't $w$ be also conditioned on $s$ as well (i.e. the final decomposition being $\phi(s,a,s)^Tw(s,g)$). This is because $g$ alone doesn't give reward information. Both $s$ and $g$ are needed for reward information.

**References**
1. Universal Successor Features for Transfer Reinforcement Learning. Ma et. al, 2020. https://arxiv.org/pdf/2001.04025.pdf
summary_of_the_review: Weighing the strengths and concerns as listed above, I am currently recommending the paper for a rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work looks at the use of successor features for solving simple continuous control tasks (in particular, reaching to different locations and door closing). The two contributions they enumerate are a ``practical implementation of SF framework for continuous state and action domains'' and jointly learning $\phi$ and $w$ (that is the successor features and the task weights $w$). They show their approach outperforms a goal-conditioned SAC baseline on these control tasks including generalizing to target locations that were not used during training.

main_review: There are a number of concerns about this work.

1. The paper states that it introduces an actor-critic approach to SF in continuous action spaces. The paper outlines the approach to learning the action-value (and successor features which can be used to compute the action-value function) [the critic], but the only details on how the actor (policy) is represented, trained or generalizes is provided implicitly in the appendix. In particular, a key challenge in continuous action spaces is that it is, in general, non-trivial to find the optimal policy even given the action-value function. How to benefit from the SF representation given this challenge is non-obvious (since SF approach provides an action-value function for a new task, but unlike in small discrete action spaces, this does not automatically allow you to infer a good policy). The challenges of doing this in continuous action spaces is outlined in e.g.\ Hunt et al. (2018).

The approach used here appears to be (from the appendix) that a goal condition policy is learned. Therefore, after training successor features are not used (since the policy does not directly use them), and the test performance is simply based on goal conditioned policies. This seems like a significant limitation of this approach, since it means that the structure of successor feature representation is only able to be used during training. It also seems a bit misleading to suggest in the introduction some sort of solution to successor features is provided for continuous action spaces, when it is more just ignored and SFs are not used after training (and therefore the claims of more structured generalization don't apply). Given this limitation some discussion about why SAC does not generalize as well to unseen tasks would also be helpful since it is also learning a goal conditioned policy and its not obvious why it should be inferior.

2. I have some concerns about the novelty of this work. Hunt et al. (2018) applied successor features to the max-ent RL framework. Kulkarni et al. (2016) learned jointly $\phi$ and $w$. Goal conditioned policies are an old idea. I think combining existing ideas can be of interest, if for example, it demonstrates compelling empirical results. However, the empirical results here are not particularly challenging tasks, and there is no comparison with other goal-condition policy learning approaches (such as Hindsight Experience Replay).

It would be helpful if the authors consider releasing code for reproducing the experiments in this paper. Particularly, as there are a number of implementation details that are unclear.

This work does appropriately cite and explain prior work in this topic.

Minor:

Inconsistent vector notation ($\mathbf{w}$ is bolded, by e.g. $\mu = (x, y)$ is not.
summary_of_the_review: This paper puts together an approach to apply SFs to some robotic control tasks. However, the approaches are not particularly novel, the empirical results are not compelling and key details regarding training of the actor are not included.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method to incorporate Successor Features (SFs)  in domains with continuous state and action spaces. It proposes an actor-critic architecture (a variation of the Soft Actor-Critic method) that learns disentangled representations for the environment dynamics and the tasks. The network architecture guarantees such disentanglement with two independent modules: one for the representation of the dynamics $\phi$ (fed by the current state, action, and next state) and one for the task representation $\boldsymbol{w}$ (fed by task-specific information, such as the goal). These modules are learned jointly in a single-stage training procedure, contrasting prior work [1]. The main contribution of this model is the enablement of the SFs for continuous domains without relying on the costly inference mechanism from the classic SFs framework implementation while enabling generalization among similar tasks.
main_review: Pros:
- The use of SFs is well motivated in the multi-task learning scenario for continuous domains since they can promote skill reuse and composability. More than that, the proposed method also allows transfer learning among tasks through the task-specific module.

- The experimental evaluation in Figure 6 shows a clear improvement from the proposed method (ACSF) compared to the goal-conditioned SAC policy, which suggests that ACSF is learning meaningful representations that enable generalization across tasks.

- It is interesting the way that ACSF modifies the SAC framework to enable successor features. It requires some detailed design choices to consider double Q-functions and target networks and keep their stability effect.

- The introduced training method (single-stage procedure) is also valuable due to the simplification from the prior method [1].

Major Concerns:

There are two general points of criticism that should be addressed during the rebuttal period. First, the paper proposes a method that is quite similar to prior literature (e.g.,  Ma et al. [2]), which is not contrasted or discussed in the paper. Secondly, the experimental section does not communicate the intentions well: given the motivations for this method, the experiments miss the critical gains of it compared to prior methods. Furthermore, it lacks a solid analysis of the results and limitations. In more details:

- The contribution of this paper is fairly incremental, and the novelty is questionable. Indeed, Ma et al. [2] proposed “Universal Successor Features”  to capture the underlying dynamics of the environment while allowing generalization to unseen goals. It also extends to the continuous scenario, which seems to refute the claim that this paper “is the first to show the applicability of SFs for continuous control tasks”, as mentioned in the Introduction. Furthermore, the paper does not analyze [2] in the Related Work section, despite the similarity. Could the authors discuss this paper from the perspective of the proposed method during the rebuttal session, please?

- Although I think interesting the level of details to enable the SAC framework for successor features, these minor decisions are quite straightforward (e.g., applying average to reduce dimensionality, and replicating the Q-function inference for the double-Q scenario). The inference scheme and training procedure are also inherited from the SAC framework. I understand that they will work simpler and better due to other components of the proposed method, but they do not present novel ideas themselves.

- The evaluation methodology described in Figure 5 trains the compared methods in different MDPs: the SAC Policy using sparse rewards and ACSF using dense rewards. In my perspective, the sparsity should also harm SAC generalization, and, therefore, this comparison seems unfair. I would like to ask the authors if they think that this is an issue and why.

- As described in the Experiments section, the proposed method relies on expert data to work. This is a critical variable to consider (especially in terms of the method’s practicality). In this way, I would say that the use of expert data should be ablated in the proposed method (so that we can evaluate how dependent the method is on this expert data). Another alternative is to also use expert data in the baseline methods (so that we can evaluate how much other methods also take advantage of this expert data).

- The results from Figure 6 seem promising, but they are not analyzed/discussed in the paper. Indeed, the paper does not cite Figure 6 in the text in any circumstance.

- There is no direct comparison between the proposed network architecture and that one from the prior method (Barreto et. al [1]). I understand that the prior method does not extend to the continuous case, but this ablation regarding the network in the proposed SAC framework will help understand how the proposed architecture improves transferability/generalization. 

- In section 3.2, the paper states that a central question for its scope is “Do SFs introduce implicit composability at the policy level?”, but I could not find any analysis/evaluation for this. The proposed experiments do not evaluate such property, and we could not conclude whether it emerges from the representation or not.

- The paper lacks some details in the experimental setup (how to collect this expert data, number of random seeds, the dimensionality of  $\psi$ for each task, number of network layers, how the goal is perturbed in the Reacher task, what is exactly the input for the task component in each scenario), making it difficult to reproduce the results. This information should be added to the paper (perhaps through the appendix). I also would like to invite the authors to release the source code for the ACSF implementation and for the experiments if this is possible (not required), which will address all the issues related to reproducibility.

- Figure 7 shows that the proposed method performs on par with a goal-conditioned policy, as described in the text. However, it is not clear what is the intention behind this experiment: if they perform equivalently, what makes the use of successor features a good idea? It would be simpler to use the goal-conditioned baseline if there is no advantage.

Minor Concerns:
- The paper introduces some symbols in the text without any definition (for example, $\phi$, $\psi$, $\boldsymbol{w}$). It makes the text harder to follow in the first read.
- In section 3.2.2, “We forumlate” -> “We formulate”
- In section 7, Fig 7c is cited twice, but the first time should be Fig 7b (door close task)
- In Table 1, the number of layers seems very high (1024, 64, 1024). Perhaps it refers to the dimension of these layers?

Additional Suggestions:
There are additional suggestions. They are not necessary for acceptance, but in my perspective, they could improve the evaluation of the proposed method.

- I recommend introducing other Representation Learning baselines as comparison methods. There are other works with different representation mechanisms [3,4,5] and such comparison will help validate the use of SFs for representation in the continuous multi-task setting.
- Although is implicit that the proposed method improves the inference problem from the prior method [1], a proper evaluation (mathematical or experimental) would help understand it objectively  (i.e., “how much” it improves). The paper claims that this is the main contribution of the method, but this is not highlighted during the evaluation.
- An interesting way to evaluate the final representation is to plot them in a lower-dimensional space. I would suggest plotting these representations (dynamics and task-specific) to evaluate interpretability and disentanglement visually.

References:

[1] André Barreto, Shaobo Hou, Diana Borsa, David Silver, Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 2020.

[2] Chen Ma, Dylan R. Ashley, Junfeng Wen, Yoshua Bengio. Universal Successor Features for Transfer Reinforcement Learning. CoRR, abs/2001.04025, 2020. URL http://arxiv.org/abs/2001.04025

[3] Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, Martin Riedmiller. Learning an Embedding Space for Transferable Robot Skills. International Conference on Learning Representations, 2018.

[4] Amy Zhang, Harsh Satija, Joelle Pineau. Decoupling Dynamics and Reward for Transfer Learning. CoRR, abs/1804.10689, 2018. URL http://arxiv.org/abs/1804.10689.

[5] Dibya Ghosh, Abhishek Gupta, Sergey Levine. Learning Actionable Representations with Goal-Conditioned Policies. CoRR, abs/1811.07819, 2018. URL http://arxiv.org/abs/1811.07819.




summary_of_the_review: Although the paper is well motivated and presents interesting implementation details, the proposed method misses the discussion of concrete similar work. This related work makes the proposed method seem fairly incremental and only marginally novel. Furthermore, the evaluation setup presents some flaws that make it hard to validate the method and its limitations. Therefore, my perspective is that this paper does not meet the requirements for acceptance.
review_title: null

Meta-review: The paper presents an actor critic type of method consisting of two types of features -- dynamics and tasks, in the multi-task continuous control setting. While the topic of the research is interesting and relevant to ICLR, the reviewers have concerns with the novelty and technical significance of the work. Specifically, the proposed method is very similar to several other works leading to an incremental novelty. In addition, the method is evaluated only on simple environments. The concerns remain after the discussion period. 

In the next version of the manuscript, the authors are encouraged to pursue more difficult settings and modify the method to work on those problems. That would make the paper stronger, and lead to a more novel method evaluated on harder problems.
Decision: Reject