Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a novel state representation technique that is based on Value Functions (VFs). The core idea of the paper is to use VFs to construct a high-level space representation in an hierarchical reinforcement learning (RL) scenario. The contributions of the paper are:

- Value Function Spaces (VFS): learned state representation
- Practical algorithms with VFS both for model-free and model-based RL settings
- Evaluation of VFS in 2 scenarios (MiniGrid and manipulation task)
main_review: Strengths
-------------

- Well written paper: clear, to the point and nice to read
- VFS is a very interesting, simple and elegant state representation method
- VFS appears to work very well in practice (if the low-level skills are selected/trained by hand). I particularly liked the manipulation example: this is a challenging task and VFS seem to be quite effective

Weaknesses
------------------

- In the HRL setting, the most challenging task is discovering and identifying the low-level skills automatically. This is not touched at all by the paper. Rather the authors pick well-selected and fully trained low-level policies. I would have liked to see one of the following:
    - A setting where the low-level skills are not "perfect". A few examples consist: skills that overlap, skills that are not well-trained, skills that achieve the desired effect but in a sub-optimal way (e.g. make a big curve to reach a point), etc..
    - Experiments on how to integrate VFS inside a full HRL pipeline that automatically discovers low-level skills and uses them for planning with VFS representation; this is would actually be the use case of VFS (since it is assuming a rather prior-free setting).

Without one of the two it is actually difficult to assess if VFS is actually useful.
- "*in Figure 1b, states with varying object or arm positions (i-iii), different background textures (iv), and distractor objects (v) are functionally equivalent for planning, and map to the same embedding in our representation*" -> How is this true? Isn't $Z_t = [V_{o1}(s_t), ..., V_{ok}(s_t)]$? Passing the raw image state inside the VF learned by an RL algorithm will definitely affect the outcome and thus the high-level states will not necessarily be close to each-other. I'd like to see more discussion on this as this is quite crucial.
- Similarly, "*This representation captures positional information about the contents of the scene, preconditions for interactions, and the effects of executing a feasible skill, making it suitable for high-level planning*" -> Although I understand what the authors want to convey, this sentence is overselling what VFS do. Or at least there is no real validation in the paper suggesting that it does all of those things.
- By the way, especially in the MiniGrid world (that has a field of view of the agent as the raw observation) many (different) states will actually be encoded very similarly using VFS (see the Go-Explore paper {1} for a big discussion on this). How is the algorithm addressing this? What is the intuition behind the success in the MiniGrid world?
- The success rate metric is nice but not the only interesting metric. E.g. in the manipulation task, the algorithm might be able to position the objects but make unnecessarily big curves. At least a cumulative reward metric should also be defined (even if this is not given to the algorithms for learning).
- There is no mention of the neural network architectures for the VFs.

References
----------------

{1}: Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K.O. and Clune, J., 2021. First return, then explore. Nature, 590(7847), pp.580-586.
summary_of_the_review: The main idea of the paper is interesting and as far as I can tell novel. The results indicate that it is working well. However, the most difficult part in an hierarchical RL setting is actually discovering the low-level skills and how to identify them automatically. The paper is not touching this at all and assumes access to pre-defined (well-selected) low-level skills.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes that given a set of skills or options, the vector of the value functions for the individual skills becomes the abstract state representation in the approach proposed by this paper. The authors then go on to develop a model-free RL algorithm akin to DQN and a model-based planning algorithm for planning in the state-space defined by the value functions of the component skills, and the action-space being the set of executable options. 

The authors demonstrate that the value function space serves as a better representation than some baseline algorithms for representation learning for RL in a maze solving and robotic manipulation task.

The key assumptions made are as follows:
 - The options are pre-defined and each option has an associated value function
 - The new task is solvable with the defined set of options.
main_review: **Major comments**

*Source of options set:* The authors considered the options as pre-defined or learned using an options learning algorithm. However they assume that such a technique to construct the options also provides an associated value function for the options. However there are multiple options learning approaches in the prior research that do not provide an explicit value function for each of those options. Rather, the options are discovered through segmentation. The authors should provide a broader survey of options learning algorithms compatible with their approach and the popular approaches that are not compatible in the interest of better scholarship.

*Positioning with respect to prior work in options-based learning and planning:* The authors idea of using value functions of each of the pre-trained options in the state abstraction is not entirely novel. I can think of at least two works that have proposals along similar lines if not exactly the same. Please refer to the options keyboard by Baretto et al. [1] and AOSM by Rosen et al. [2]. I would like the authors to address their work in context of these works.

*Relationship with transfer learning and option reuse:* Works that attempt transfer learning through the reuse of options, are also themed around constructing better state features or abstractions tied to action availability and feasibility. The authors do not explicitly address this connection, and I would like to see them do that.

I quite like the idea of action abstractions, or action oriented constructions of state abstractions, especially for robotic domains, and this paper does propose a simple but potentially effective idea for achieving it. My primary concern centers around the fact that the presentation in the paper leave a lot to be desired in terms of positioning the paper with prior work, and tying it in with option discovery work that it relies upon. 



[1] - Barreto, André, et al. "The option keyboard: Combining skills in reinforcement learning." NeurIPS(2019).

[2] - Rosen, Eric, et al. "Building Plannable Representations with Mixed Reality." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
summary_of_the_review: I find value function spaces to be a simple but potentially effective idea. The comparisons have largely centered around representation learning frameworks applied to deep-RL, but not around reinforcement learning techniques aimed at option reuse and option discovery. While the approach proposed is different, these works deserve to be discussed in context. As such, I urge the authors to address these concerns qualitatively or empirically.

Post Response:

I am quite happy with the updated version of the paper, and willing to upgrade my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents Value Function Spaces (VFS), an abstract "skill-centric" representation for reinforcement learning and long-horizon planning. The core idea of VFSs is to leverage the learned value functions that are often trained alongside skills during skill learning to abstract knowledge about the world. Each element of this low-dimensional representation corresponds to the value estimated from one of the learned value function, so that the representation (by construction) tends to ignore distractor information unnecessary for accomplishing the skills. Using this representation on a set of pre-trained skills, the authors demonstrate improved performance on long-horizon planning tasks in grid-based maze and "locked door" environments, showing improved success probability on a number of such tasks compared to competitive baselines. They also show the ability of their abstraction to facilitate model-based goal-directed planning by learning a state-transition model (in VFS-space) that allows one to predict the outcome of executing a skill.
main_review: This paper presents an interesting approach by which one can get more out of trained skills; their primary contribution (the novel VFS representation) is easy to construct for skill-driven agents and could therefore serve as a relatively general representation to allow for performance improvement across a range of applications in which skills are already being used to make progress. Overall, though the idea is somewhat simple, it seems to me to be novel and has wide-ranging applications, including a number of applications demonstrated here.

My biggest concern is the lack of performance metrics beyond simply "success rate". Their omission is somewhat surprising and the authors should either include these additional performance metrics or discuss why they are unnecessary. The zero-shot generalization results are indeed promising, but it is hard to really evaluate how well the VFS agent performed compared to the HRL-Target agent (which is trained directly on this problem).
"Success-weighted path length" (SPL), a metric used by the Vision-and-Language community, would be somewhat appropriate, though presenting the "average performance/cost of successful trials" would perhaps be more useful. While I am not sure that this addition is *essential* for publication, these results are rather important for understanding the value of the approach and so the authors should make an effort to include them. The authors should either add this metric or adequately justify its omission.

Relatedly, it would be helpful to understand the nature of the failure cases: What does a failure look like? Some understanding of when and why the system fails is essential to understanding where future work is needed. For instance, in Sec 5.2, the agent fails in 32% of the more challenging environments: when it fails, is this a limitation of the low-level skills, of the ability of the agent to correctly predict the transition model, or both? Even if the authors do not readily have the answers to these questions, a few sentences commenting on the nature of experiments that would be necessary to answer them would be a welcome addition to the paper and help to improve the understanding of both readers and those that would build upon this approach. Similarly, what are the limitations of this approach? When would we expect that it would not be effective or fail? The Discussion section touches upon some of the potential directions for future improvement, but additional comments on the limitations of this version would help understanding. Additional experiments are not necessary here (nor are specific examples); a high-level discussion of these questions will suffice.

Questions and smaller suggestions:
- [Sec 6.1] Where does the "goal latent sate" come from? My intuition is that there would a non-latent goal state provided to the planner, which would be subsequently "encoded" into its VFS form. Is that correct? Either way, this should be clarified in the paper (along with, perhaps, a more comprehensive overview of the planning procedure).
- [Figure 1(b)] Though it is mostly clear from context, using an "equal" sign is not quite correct here, since the states themselves are in fact different. It is certainly possible to understand what the authors are going for, but changing the figure to reflect that applying the VFS encoding function makes all these states equal to one another would be a more appropriate way of communicating this knowledge, especially important since this figure is supposed to outline the core of the authors approach.
- [Figure 6] What are the skills available to the agent in this environment? Adding a few examples would help the reader understand the clusters better, though the idea that "the pose of objects is distractor information" is already clear from the figure.
summary_of_the_review: This paper presents an interesting approach by which one can get more out of trained skills; their primary contribution (the novel VFS representation) is easy to construct for skill-driven agents and could therefore serve as a relatively general representation to allow for performance improvement across a range of applications in which skills are already being used to make progress. One of my primary concerns is the lack of inclusion of cost-based performance metrics for their experiments, which I feel are somewhat important for understanding the performance of the approach. Even without them, I think this work represents a (mostly) well-executed, interesting idea, and so I am generally in favor of publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a state abstraction in hierarchical reinforcement learning (HRL), called Value Function Spaces (VFS), that is constructed from the value functions of task-conditioned lower-level policies (or skills). The idea is that the value functions capture affordances of the lower-level skills while ignoring task-irrelevant information.
The lower-level policies and the corresponding value functions are given a prior, and the state is constructed by concatenating the value estimate of each lower-level policy.
The state abstraction is evaluated in a model-free Q-learning and model-based MPC type scenario.
main_review: # Strengths:

(1) The idea of using the value estimates of different pre-existing policies as state representation is, to my best knowledge, novel.

(2) Since the value estimates of pre-trained policies are used directly as states of the higher-level policy, the method can make use of the large set of pre-existing policies available in the wild. 

(3) The paper is generally well written, and the narrative is easy to follow.

# Weaknesses: 

(1) I have the feeling that the comparison with the baselines is not entirely fair. The baselines have to learn a good state embedding and a good higher-level policy simultaneously, while the proposed method only needs to learn the higher-level policy. However, this completely ignores the fact that the proposed method had access to a much larger corpus of transition since it uses the pre-trained value functions. In contrast, the baselines do not have access to the value functions of the skills. 

(2) I would like to see the method evaluated in much more environments. From the existing experiments, it is hard to judge how the method performs in different settings.

# Needs clarification and general comments

(1) How does this method deal with different scalings of the value estimates? Especially in the dense reward case, value estimates for different skills can differ significantly, i.e. because of reward shaping.

(2) Paragraphs 2 and 3 in section 4 repeat more or less what is written in paragraph 4 of the introduction.

(3) To me, it does not feel right to artificially restrict the embedding spaces of the baselines to the maximum size of the VFS embedding space. 
What is the justification for that? What are the best-case results for the baselines without that restriction?

(4) In the case of the offline baselines: How many transitions were seen during training of the embedding space compared to the number of transitions used for learning the different skills/policies and their respective value functions?
Similarly, in the case of the online baselines: How many transitions were used in total (pre-training plus training of the higher-level policy) to train the proposed method compared to train the baselines (which do not have access to the value function estimates of the skills)?

# Final decision due to rebuttal:

The authors addressed many of my concerns and provided new experimental results that led to interesting new insights. Hence, I am happy to raise my score from 5 to 6.
summary_of_the_review: The idea of using the value estimates of pre-trained policies is interesting and the empirical results look promising. However, as I pointed out in the weaknesses section, I do not think that the comparison with the baselines is entirely fair. Additionally, I would like to see the method evaluated in more environments. Because of that, I tend towards a weak reject, but I am willing to change my score if the authors can address these issues.
review_title: null

Meta-review: This paper is good but at a borderline. One reviewer increased the score during the discussions. However, no reviewer was in strong favor. So that this paper is still a borderline one, and it is up to the SAC to decide.
Decision: Accept (Poster)