Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
Authors: Dhruv Shah
Peng Xu
Yao Lu
Ted Xiao
Alexander T Toshev
Sergey Levine
brian ichter
Venue: ICLR 2022 Poster (2022)
Keywords: hierarchical reinforcement learning
planning
representation learning
robotics

Abstract: Reinforcement learning can train policies that effectively perform complex tasks. However for long-horizon tasks, the performance of these methods degrades with horizon, often necessitating reasoning over and chaining lower-level skills. Hierarchical reinforcement learning aims to enable this by providing a bank of low-level skills as action abstractions. Hierarchies can further improve on this by abstracting the space states as well. We posit that a suitable state abstraction should depend on the capabilities of the available lower-level policies. We propose Value Function Spaces: a simple approach that produces such a representation by using the value functions corresponding to each lower-level skill. These value functions capture the affordances of the scene, thus forming a  representation that compactly abstracts task relevant information and robustly ignores distractors. Empirical evaluations for maze-solving and robotic manipulation tasks demonstrate that our approach improves long-horizon performance and enables better zero-shot generalization than alternative model-free and model-based methods.

Published as a conference paper at ICLR 2022
VALUE FUNCTION SPACES: SKILL-CENTRIC STATE ABSTRACTIONS FOR
REASONING
Dhruv Shah??? Peng Xu², Yao Lu", Ted Xiao? Alexander Toshev?, Sergey Levine 29, Brian Ichter?
"Google Research, Robotics @ Google
Berkeley AI Research, UC Berkeley
ABSTRACT
Reinforcement learning can train policies that effectively perform complex tasks. However for long-horizon tasks, the performance of these methods degrades with horizon, often necessitating reasoning over and chaining lower-level skills. Hier- archical reinforcement learning aims enable this by providing a bank of low- level skills as action abstractions. Hierarchies can further improve on this by ab- stracting the space states as well. We posit that a suitable state abstraction should depend on the capabilities of the available lower-level policies. We propose Value Function Spaces: a simple approach that produces such a representation by using the value functions corresponding to each lower-level skill. These value functions capture the affordances of the scene, thus forming a representation that compactly abstracts task relevant information and robustly ignores distractors. Empirical evaluations for maze-solving and robotic manipulation tasks demonstrate that our approach improves long-hon horizon performance and enables better zero-shot gener- alization than alternative model-free and model based methods.
INTRODUCTION
For an agent to perform complex tasks in realistic environments, it must be able effectively reason over long horizons, and parse high- seth-imeensonal observations to infer the contents of a scene and its affordances. It can do so by constructing compact representation that is robust to distractors and suitable for planning and control. Consider, for instance, robot rearranging objects on a desk. To succcessfully solve the task, the robot must learn to sequence a series of simple skills, such as picking and placing objects and opening drawers, and interpret its observations to determine which skills are most appropriate. This requires the ability to understand the capabilities of these simpler skills, as well as the ability to plan to execute them in the correct order.
Hierarchical reinforcement learning (HRL) aims to enable this by leveraging abstraction, which simplifies the higher-level control planning problem. Typically, this is taken to mean abstraction of actions in the form of primitive skills (c.g., options (Sutton et al., 1999)). However, significantly simplifying the problem for the higher level requires abstraction of both states and actions. This is particularly important with rich sensory observations, where standard options frameworks provide a greatly abstracted state space, but do not simplify the perception estimation problem. The nature of the ideal state abstraction in HRL is closely tied to the action abstraction, as the most suitable ab- straction of state should depend on the kinds of decisions that the higher-level policy needs make, which in turn depends on the actions (skills) available to it. This presents challenge in designing HRL methods, because it is difficult to devise a state abstraction that is both highly abstracted (and therefore removes many distractors) and still sufficient to make decisions for long-horizon tasks. This challenge differs markedly from representation learning problems in other domains, like com- puter vision and unsupervised learning (Schroff et al., 2015; den Oord et al., 2018; Chen et al., 2020), since it is intimately tied to the capabilities exposed to the agent via its skills.
We therefore posit that suitable representation for higher-level policy in HRL should depend on the capabilities of the skills available to it. If this representation is sufficient to determine the abilities and outcomes of these skills, then the high-level policy can select the skill most likely to perform the desired task. This concept is closely tied to the notion of affordances, which has long been action-centric representation of state (Gibson,
studied in cognitive science and psychology

---

Published as a conference paper at ICLR 2022
Pol
race due prouee da unser
Pase Radl Orosen
Class
Open Drawer
Blue
Place
Close
(a) A trajectory through the skill value function space for the task "Place blue block in drawer". VFS, visualized lop of corresponding scene, captures positional information about the contents of the scene, preconditions for interactions, the effects of executing a feasible skill, making suitable high-level planning.
the
Moone an Ouarte
Plot an Dover Duen Doamct
(b) VFS ignores task-irrelevant factors like arm pose, color of the desk, or distractors, All configurations shown above are functionally equivalent and hence, map the same VFS representation.
Figure 1: Visualizing VFS embeddings an example desk rearrangement task. VFS can capture the affor-
dances of the low-level skills while ignoring exogenous distractors.
1977), and has inspired techniques in robotics and RL (Zech et al., 2017; Xu et al., 2021; Mandikal & Grauman, 2021). Given a set of skills that span the possible interactions an environment, we provide representation that
propose that the value functions corresponding to these policies suitably captures the capabilities of the skills in the current state and thus can be used to form a compact embedding space for high-level planning. We call this the Value Function Space (VFS). Note that we intend to use these skills as high-level actions without modifications the important problems of skill discovery and test-time adaptation are beyond the scope of this paper.
Figure la illustrates the state abstraction constructed by VFS for the desk rearrangement example discussed above: VFS captures the affordances of the skills and represents the state of the envi- ronment, along with preconditions for the low-level skills, forming a functional representation to plan over. Since VFS constructs skill-centric representation of states using the value functions of the low-level skills, it captures functional equivalence of states in terms of their affordances: in Figure 1b, states with varying object arm positions, background textures, and distractor objects are functionally equivalent for planning, and map to the same VFS embedding. This simplifies the high-level planning problem, allowing the higher level policy to generalize to novel environments.
Statement of Contributions. The primary contribution of this work is VFS, a novel state representa- tion derived from the value functions of low-level skills available to the agent. We show that VFS leverages the properties of value functions, notably their ability model possible skills and com- pleted skills, to form an effective representation for high-level planning, and compatible with both model-free and model-based high-level policies. Empirical evaluations in maze-solving and robotic manipulation demonstrate that the skill-centric representation constructed by VFS outperforms rep- resentations learned using contrastive and cettmaation-thhorette objectives in long-horizer tasks. We also show that VFS can generalize to novel environments in a zero-shot manner.
RELATED WORK
Hierarchical RL has been studied extensively in the literature, commonly interpreted as a temporal abstraction of the original MDP. Early works have interpreted the hierarchy introduced in this setting as an abstraction of state and action spaces (Sutton et al., 1999; Dietterich, 2000; Thomas Barto, 2012). The popular options framework Sutton et al. (1999); Precup (2000) provides natural way of incorporating temporally extended actions into RL systems. An agent that possesses the transition model and reward model for such SMDP (known planning in discrete (Kocsis & Szepesvári, 2006) and continuous state spaces (Konidaris et al., 2014; Gopalan et al., 2017). However, doing in environments with digh-dimensional observations (such as images) is challenging. In this work, we explore the efficacy of learned skills operating on high- dimensional observations for long-horizon control in realistic environments.
option model) is capable of sample-based
2

---

Published as a conference paper at ICLR 2022
To improve the quality of lower-level policies, recent work in HRL has studied various facets of the problem, including discovery of skills (Konidaris & Barto, 2009; Zhang et al., 2021b; Florensa et al., 2017; Warde-Farley et al., 2018), end-to-end training of both levels (Kulkarni et al., 2016; Tessler et al., 2017) and integrating goal-conditioned behaviors (Ghosh et al., 2019; Nachum et al,, 2019). In this work, assume that the low-level skills are given, and do not focus on discovering them. Instead, we focus on how skills can simplify the higher-level control problem by providing a representation that is suitable for either model-based or model-free control.
The problem of learning meaningful long-horizon behavior by combining set of lower-level skills, or options, has been studied extensively in prior work. Popular approaches to composing skills involve combining them in the space of value functions (Todorov, 2009; Hunt et al., 2019; da Silva et al., 2009; Ziebart, 2010; Haarnoja et al., 2018), and more recently, using pseudo-rewards (Barreto et al,, 2019). While these methods have been very effective at composing low-level skills, they do not address the question of learning a meaningful state abstraction using these skills. We take inspiration from these works and use the value function space to construct skill-centrr state abstraction, along with a high-level policy that composes these skills for temporally extended tasks.
Our method can hence be interpreted as representation learning approach. Representation learning techniques have been employed extensively in model-free RL by augmenting auxiliary tasks based on reconstruction losses (Lange et al., 2012; Higgins et al., 2017: Yarats et al., 2019) and predicting the future conditioned on past observations (Schmidhuber, 1990; Jaderberg et al., 2017; van den Oord et al., 2018; Shelhamer et al., 2016). Contrastive learning has also been used in recent works to discover a meaningful latent space and extract reward signals for RL (Sermanet et al., 2017; Warde-Farley et al., 2018; Dwibedi al., 2018; Laskin al., 2020). Another popular approach to learning useful representation is to optimize for effective transfer of the lower-level skills and the learned state abstraction (Gupta et al., 2017: Konidaris & Barto: Goyal et al., 2019). Unlike these prior works, our aim is specifically to learn representation that is grounded in the capabilities of the low-level skills, which gives us skill-centric abstraction of high-dimensional observations. While we do not explicitly optimize for transfer like Gupta et al. (2017), our experiments show that VFS can generalize to novel environments while being robust to functionalll-irreelvan distractors.
Alongside developments in model-free RL, prior work has also sought to learn predictive models of the environment for sampling and planning. This has been demonstrated by learning dynamics using future predictions (Watter et al., 2015; Oh et al., 2017: Ebert et al., 2017; Banijamali et al., 2018; Ha& & Schmidhuber; Hafner et al., 2018; Ichter & Pavone, 2019; Hafner et al., 2020; Zhang et al., 2019), learning belief representations (Gregor et al., 2019; Lee et al., 2019) and representing state similarity using the bisimulation metric (Castro, 2020; Zhang et al., 2021a; Agarwal et al., 2021). The combination of learned model-free policies with structures like graphs (Savinov al., 2018; Eysenbach et al., 2019) and trees (Ichter et al., 2021) to plan over extended horizons has also been demonstrated to improve generalization and exploration. While our primary objective is not to develop better model- based RL algorithms, we show that our proposed state abstraction can be utilized by model-based controllers to plan over temporally extended skills.
3 PRELIMINARIES
We assume that an agent has access to a finite set of temporally extended options O, or skills, which it can sequence to solve long-horizon tasks, These skills can be trained for a wide range of tasks by using manual reward specification (Huber & Grupen, 1998; Stulp & Schaal, 2011), using relabeling techniques (Andrychowicz et al., 2017). or via unsupervised skill discovery (Daniel et al., 2016; Fox et al., 2017: Warde-Farley et al., 2018; Sharma et al., 2020). Since many prior works have focused on skill discovery, we do not explicitly address how these skills are produced.
We assume that each skill has a maximum rollout length of T time steps, after which is terminated. We also assume that each skill o; E O is accompanied by critic, or value function, V., denoting the expected cumulative skill reward executing skill o, from current state st. This is generally available for policies trained with RL (Huber & Grupen, 1998; Andrychowicz et al., 2017; Kalashnikov et al., 1)2 or learned via options discovery (Bacon et al., 2017; Klissarov et al., 2017; Tiwari & Thomas, 2019: Riemer et al., 2018b). While this may not be true for skills discovered by other methods, such as segmentation, graph partioning or bottleneck discovery (McGovern & Barto. 2001; Menache et al., 2002; Krishnamurthy et al., 2016; Şimşek & Barto, 2004; Şimşek et al., 2005; Riemer al.,
3

---

Published as a conference paper at ICLR 2022
2018a), we use policy evaluation to automatically obtain value function this can be done via regression to empirical returns or temporal difference learning (Dann et al., 2014).
Our setting is closely related to the options framework of Sutton et al. (1999). Options are skills that consist of three components: a policy x : & A - [o, 1]. a termination condition 3 S + [0, 1], and initiation set I C S, where S, A are the low-level state and action spaces in fully observable decision process. An option (I, म. (3) is available at state &, and only if Se E I. We do not assume we have an initiation set, and show that the value functions provide this information. We assume that the policies come with a termination condition, or alternatively, have fixed horizon т. Next, we describe framework to formulate a decision problem that uses these options for planning.
We assume that the low-level observation and action space of the agent can be described as fully observable semi-Markov decision process (SMDP) M described by tuple OS, O, R. P. T, 7). where S S I is the n-dimensional continuous state space; O is finite set of temporally extended skills, or options, with a maximum horizon of time steps; R(s' |s, 0;) the task reward received after executing the skill o E O at state E S; P(s'|s 8, 04) is PDF describing the probability of arriving in state 8' E S after executing skill o E at state 5 E S: 7 E (0, 1] is the discount factor. Given a finite set of options, our objective is to obtain high-level decision-policy that selects among them.
Note that generally, many image-based problems are partially observed so that the entire history of observation-action pairs may be required to describe the state. Explicitly addressing partial observ- ability is outside the scope of our work, so we follow the convention in prior work by assuming full observability and refer images as states (Lillicrap et al., 2016; Nair et al., 2018).
4 SKILL VALUE FUNCTIONS AS STATE SPACES
The notion of value in RL is closely related to affordances, in that the value function predicts the pabilities of the skill being learned. The supervised learning problem of affordance prediction (Ugur al., 2009) can in fact be cast as special case of value prediction (Sutton & Barto, 2018; Graves et al., 2020). In this section, we construct a skill-centric representation of state derived from skill value functions that captures the affordances of the low-level skills, and empirically show that this representation is effective for high-level planning.
Given an SMDP M(S. R. P. come rewards and their corresponding value functions V.., we construct an embedding space Z by stacking these skill value functions. This gives us an abstract representation that maps a state 8 to k-dimennional representation (sx)) == [V,, (sz), Voz (8t),.... .... Vo, (s+)). which we call the Value Function Space, or VFS for short. This representation captures functional information about the exhaustive set of interactions that the agent can have with the environment by means of executing the skills, and is thus a suitable state abstraction for downstream tasks.
P. 7. with a finite set of skills o, O trained with sparse out-
Figure 1(a) illustrates the proposed state abstraction for a conceptual desk rearrangement task with eight low-level skills. A raw observation, such as an image of the robotic arm and desk, is abstracted by a 8-dimensional tuple of skill value functions. This representation captures positional information about the scene (e.g., both blocks are on the counter and drawer is closed since the corresponding values are fo). preconditions for interactions (e.g., both blocks can be lifted since the "Pick" values are high at to), and the effects of executing a feasible skill (c.g., the value corresponding to the "Open Drawer" skill increases on executing at fj), making it suitable for high-level planning.
We hypothesize that since VFS learns skill-centric representation of the scene, it is robust to exogenous factors of variation, such as background distractors and appearance of inreleate components of the scene. This also enables VFS to generalize to novel environments with the same set of low-level skill, which we demonstrate empirically. Figure 1(b) illustrates this for the desk rearrangement task. All configurations shown are functionally equivalent and can be interacted with identically. Intuitively, VFS would map these configurations to the same abstract state by ignoring factors like arm pose, color of the desk, or additional objects, which do not affect the low-level skills.
5 MODEL-FREE RL WITH VALUE FUNCTION SPACES
In this section, we instantiate hierarchical model-free RL algorithm that uses VFS as the state abstraction and the skills as the low-level actions. We compare the long-horizon performance of VFS to alternate representations for HRL trained with constrastive and information theoretic objectives
4

---

Published as a conference paper at ICLR 2022
for the task of maze-solving and find that VFS outperforms the next best baseline by up to 54% on the most challenging tasks. Lastly, we compare the zero-shot generalization performance of these representations and empirically demonstrate that the skill-centric representation constructed by our method can successfully generalize to novel environments with the same set of low-level skills.
5.1 AN ALGORITHM FOR HIERARCHICAL RL
We instantiate hierarchical RL algorithm that learns Q-function Q o) at the high-level using VFS as the "state" Z and the skills o, E O as the temporally extended actions. Given such Q- function, a greedy high-level policy can be obtained by xQ = ( arg max =O Q(Z, We use DQN (Mnih et al., 2015), which uses mini-batches sampled from replay buffer of transition tuples (z, Tt: 7(+1) to train the learned Q-function to satisfy the Bellman equation. This done using gradient descent on the loss L = E (Q(Z,, 0z) - y²) -y+)=
where rt. + 7 maxoy ( Or).
In order to make the optimization problem more stable, the targets yt are computed using separate target network which is update at a slower pace than the main network. Q-learning also suffers from an overestimation bias, due to the maximization step above, and harms learning. Hasselt et al. (2016) address this overestimation by decoupling the selection of action from its evaluation. We use this variant, called DDQN, for subsequent evaluations in this work. Note that we are using the skills as given, without updating or finetuning them.
While there have been several improvements DDQN (Hessel et al., 2018) and alternative algo- rithms for model-free RL in discrete action spaces (Schaul et al., 2015; Bellemare et al., 2017; Christodoulou, 2019), and these improvements will certainly improve the overall performance of our algorithm, our goal is to evaluate the efficacy of VFS as state representation and we study it in the context of a simple DDQN pipeline.
5.2 EVALUATING LONG-HORIZON PERFORMANCE IN MAZE-SOLVING
To evaluate the long-horizon performance of VFS against commonly used representation learning methods, we use the versatile MiniGrid environment (Chevalier-Boisse observable setting, where the agent receives a top-down view of the environment. We consider two tasks: (i) MultiRoom, where the agent is tasked with reaching the goal by solving a variable- sized maze spanning up 10 rooms. The agent must cross each room and open the door to ac- cess the following rooms; (ii) KeyCorridor, where the agent is tasked with reaching a goal up to 7 rooms away and may face locked doors. The agent must find the key corresponding to a color- coded door and open it to access subsequent rooms. Both these tasks have sparse reward that only provided for successfully reaching the goal. This presents challenging domain for long- horizon sequential reasoning, where tasks may require over 200 time steps to succeed, making great testbed for evaluating the ability of the state abstractions to capture relevant information for sequencing multiple skills. The agents have access to the following temporally extended skills- GoToOb ject, Pickupob ject, DropOb ject and UnlockDoor where the first three skills text-conditioned and Ob ject may refer to door, key, box circle of any color. Since Min- iGrid is easily reconfigurable, we also generate a set of holdout MultiRoom mazes with different grid layouts to evaluate the zero-shot generalization performance of the policies trained with these representations. Note that the grid layouts, as well as object positions, for these tasks are randomly generated for every experiment and are not static. Example grid layouts are shown in Figure 2. We provide further implementation details in Appendix A.I.
et al., 2018) in a fully
Baselines:
We compare VFS extensively against a variety of competitive baselines for representa- tion learning in RL using contrastive and informaaion-thhoretiic objectives. We consider representa- tions learned both offline and online (in loop with RI.); note that VFS constructed entirely from the values of the available skills and is not learned. Further, all baselines have access to these skills.
1. Raw Observations: We train a high-level policy operating on raw input observations.
2. Autoencoder (AE): We use an autoencoder to extract a compact latent space using a recon- struction loss on an offline dataset of trajectories, similar to Lange et al. (2012).
3. Contrastive Predicting Coding (CPC): We learn a representation by optimizing the In- foNCE loss over offline dataset of trajectories (van den Oord et al., 2018).
5

---

Published as a conference paper at ICLR 2022
Representation
MultiRoom
Key Corridor
2
6
10
3
7
Raw Observations
0.64
0,46
0.42
0.29
0.47
0.32
AE (Lange et al., 2012)
0.70
0.64
0.51
0.34
0.59
0.33
CPC (van den Oord et al., 2018)
0.77
0.69
0.55
0.37
0.63
0.35
VAE (Yarats et al., 2019)
0.79
0.74
0.58
0,49
0.79
0,50
CURL (Laskin et al., 2020)
0.82
0.76
0.63
0.43
0.83
0.54
VFS (Ours)
0.98
0.92
0.83
0.77
0.82
0.68
Table Success rates different representations for model-free RI. across varying levels difficulty and time horizons (second row denotes complexity in terms of number of rooms). VFS explicitly captures the capabilities of the low-level skills, and outperforms all baselines. Online methods (denoted by ') learned jointly with the RL objective outperform their offline counterparts (AE and CPC), but their performance degrades for longer-horizon tasks. Refer to Appendix B additional performance metrics and ablations.
GoToGreenDoor
Go DooeeDooo
GoToYellowDoor
Doreeeen
GoToRedDoor
GoTeGreenBox
GoToRedDoor
GoToYellowee
PickUpYellawKey
GoToYelloWDoor
DropYellowKey
GoToYellowCircle
Figure 2: Successful rollouts of HRL using VFS to solve long-horizon tasks in MultiRoom-6 (top) and KeyCorridor-7 (bottom) by sequentially executing multiple low-level skills (labeled under the arrows).
4. Online Variational Autoencoder (VAE): We learn a VAE representation jointly (online) with the high-level policy operating this representation (Yarats et al., 2019).
5. Contrastive Unsupervised Representations for RL (CURL): We learn a representation by optimizing contrastive loss jointly (online) with the RL objective (Laskin et al., 2020).
Evaluation: We run experiments for the tasks with varying number of rooms, to study the performance of the algorithms with increasing time horizon, and report the success rates in Table 1. HRL from raw observations demonstrates a success rate 64% in the two-room environment, which can be attributed to the powerful set of skills available to the high-level policy, but this quickly drops 29% in the largest environment. The offline baselines (AE and CPC) construct a compact state abstraction, which makes the learning problem easier for DDQN, and show significant improve- ments smaller environments (MultiRoom-22, and KeyCorridor33), However, they are unable to improve the performance in larger environments. We hypothesize that this is due to the inability of the representations to capture information necessary for the high-level policy, since they are learned independent of the controller. The performance of representations learned online (VAE and CURL), which are implemented analogous to their offline counterparts (AE and CPC, respectively) support this hypothesis and improves the performance with the controller, and scoring up to 54% in the most challenging environment. We hypothesize that the limited performance of these methods is due the lack direct influence of the downstream task on the representation. Despite being constructed offline, VFS explicitly captures the capabilities of the low-level skills and provides an action-centric representation for the high-level policy, greatly simplifying the control problem. This is reflected in the performance of VFS across all tasks outperforms baselines, scoring 98% on the simplest task and up to 68% on the most challenging task, beating the next best method by over 25%. Figure 2 shows sample rollouts of the high-level policy using VFS as the state representation in the two tasks discussed. It is important to note that while VFS outperforms the baselines in these tasks, method is largely orthogonal to these repre- sentation learning algorithms and can be combined constructively to improve task performance show this, along with further ablations in Appendix B.
all tasks by learning representations jointly

---

Published as a conference paper at ICLR 2022
Representation
MR4
MRI0
Agent
Raw
0.15
0.03
AE
0.41
0.23
Unlocked Door
CPC
0.47
0.20
VAET
0.25
0.03
Locked Door
CURL
0.27
0.07
VFS (Ours)
0.87
0.67
Key
HRL-Target 0.92
0.77
(b)
Table 2: Success rates for zero-shot generalization. VFS can generalize zero-shot to novel environments.
We study the zero-shot generalization by using policies trained in KeyCorrider (a) and deploying in MultiRoom (b).
Figure 3:
5.3 ZERO-SHOP GENERALIZATION
We evaluate the generalization abilities of these representations by training them for the KeyCorridor task (as above) and evaluating on MultiRoom. Since the skills available in MultiRoom are subset of those in KeyCorridor, we ignore any invalid actions executed by the agent. Note that the high-level agent has not been trained in MultiRoom and the policies are not updated in the target environments.
Table shows the success rates rooms (MRIO). Unsurprisingly, HRL with observations fails to generalize, because the maze layout differs significantly (c.g. see Figure 3). AE and CPC learn a compact representation from high-dimensional observations and allow the high-level policy to generalize to simpler tasks like MR4 and achieve up to 47% success rate. Interestingly, their online counterparts (VAE and CURL) also fail generalize and perform poorly, likely because representations learned jointly with the RL policy can overfit to the source environment. VFS learns a skill-centric representation that can generalize zero-shot tasks that the same skills, and thus outperforms the next best baseline by up to 180%. closely matching the performance of an HRL policy trained from scratch in the target environment with online interaction SHRL-Targer
representations in the MultiRoom tasks with (MR4) and 10
MODEL-BASED PLANNING WITH VALUE FUNCTION SPACES
In this section, we introduce simple model-based RL algorithm that uses VFS the "state" for planning, which we term VFS-MB. We study the performance of VFS in the context of robotic manipulation task, and compare to alternate representations for model-based RL. We find that the performance of VFS as an abstract representation for raw image observations outperforms all base- lines and closely matches that of pipeline with access oracular state of the simulator, showing the efficacy of our method in long-horizon planning.
6.1 A SIMPLE ALGORITHM FOR MODEL-BASED RL
We use a simple model-based planner that learns one-step predictive model, using VFS as the "state." Specifically, this model learns the transition dynamics 7-1 f(Z,, ox) Vo: E O via supervised learning using a dataset of prior interactions the environment. Note that the predictive model (and the subscript of Z) is over high-level policy steps, which may be up to 7 steps of the low- level skills. This dataset can be collected simply by executing the available skills the environment for steps, where is the maximum horizon of the SMDP M, or until termination.
In order to use the learned model f(z,, ot), goal latent state Zy. and scoring function (e.g. mean squared error) for the high-level task, we need to solve the following optimization problem for the optimal sequence of skills (Ot,...,014 014H to reach the goal:
(Ot...., ......,0+H-). arg min c(i+H, Zg) 2, = Ze, Z041 f(2z, (Zzz.rr) or)
(1)
O H-1
We use sumpling-based method to find solutions to the above equation. We use random shooting (Rao, 2009) to randomly generate K candidate option sequences, predict the corresponding se- quences using the learned model f. compute the rewards for all sequences, and pick the candidate action sequence leading to latent state closest the goal, according to Equation 1. We execute the

---

Published as a conference paper at ICLR 2022
policy using model-prediciiee control: the policy executes only the first skill or. receives the updated Z4+1 from the environment, and recalculates the optimal action sequence iteratively, Figure 4 shows an overview of the algorithm. We provide further implementation details in Appendix A.2
Note that our goal is to study the behavior of VFS as a state representation in existing RL pipelines, rather than developing a better model-based algorithm. Using sophisticated methods that adjust the sampling distribution, as in the cross-entropy method (Finn & Levine, 2017: Hafner et al., 2018; Chua al., 2018) or path integral optimal control (Williams et al., 2015; Lowrey et al., 2019), as well as more sophisticated models and planning or control methods would likely improve overall performance further, but is outside the scope of this work.
No x -7
2
"llaee both in drawer'
2,00)- [V]()))
argmin x 112 2,10
Figure 4: Overview of model-based planning with VFS. We learn a one-step predictive model for the embed- ding Ze and use random shooting with scoring function find the best action the encoded goal Zg.
6.2 APPLICATION: ROBOTIC MANIPULATION
We evaluate the performance of VFS-MB in a complex image-based task using a simulated ma- nipulation environment with an 8-DoF robotic arm, similar the setup used by Jang et al. (2021). The robot only has access to high-dimensional egocentric visual observations. We consider the task of semantic rearrangement, which requires rearranging objects into semantically meaningful positions-e.. grouped into (multiple) clusters, or moved to corner (Figure 5). Efficiently solv- ing this task requires the robot to plan over series of manipulation maneuvers, interacting with up to 10 distinct objects. This requires long-horizon planning, and presents major challenge for RL methods. Each of the methods we compare have access to a library of MoveANearB skills, trained multi-task policy with MT-Opt (Kalashnikov et al., 2021). There are 10 objects, and 9 possible destinations near which they can be moved, resulting in a total of 90 skills, which then comprise the action space for planning. The skills control the robot at frequency of 0.5 Hz. taking an average of 14 time steps to execute with success rate of 94%. We provide further details about these skills in Appendix A.2.
We consider two versions of the task, which arrange either 5 or 10 objects to semantic positions. The 05 environment uses a random subset of the 10 objects in every experiment; lesser objects allow a smaller planning horizon, making planning problem simpler than in the 010 environment, which has all 10 objects. We randomize the object positions on the table and command the algorithms to reach the same goal with a planning horizon H 7 steps for the smaller environment and H = 15 steps for the larger environment, reporting the task success rate averaged over 20 experimental runs.
Baselines: To evaluate the efficacy of the proposed representation for high-level model-based planning, we compare it against four alternative representations used in conjunction with the algo- rithm described in Section 6. 1. All methods have access the skills and use them as the low-level action space. Raw Image learns a policy on raw visual observations from the robot's 's onboard cam- era. VAE uses a variational autoencoder to project the observations to a -inmemnsiion learned embedding space, similar to Corneil et al. (2018). We train this VAE offline from trajectories col- lected by rolling out the models. CPC uses contrastive predictive coding (van den Oord et al., 2018) to learn similar representation from offline trajectories. We also compare against an Oracle State baseline that has access to privileged information the simulator state and learns the model on this. This gives an upper bound for the performance of the baselines.
Evaluation: Table 3 shows the success rates on the two versions of the task for each prior method. The model trained using image observations fares poorly and fails at all but the simplest starting configurations. It succeeds in 2/20 experiments in 010, largely due poor model predictions. The VAE and CPC representations, which learn a more compact embedding space to plan over, succeed in up to 65% of the tasks in the simpler 05 environment. However, their performance falls sharply in
8

---

Published as a conference paper at ICLR 2022
Representation
05
010
Raw Image
0.25
0.1
VAE
0.6
0.3
CPC
0.65
0.4
Move Bowl Near Sponge
Move Bottle Near Bow!
Move Grees Near Sporge
Move Apple Near Bow!
VFS (Ours)
0.8
Oracle State
0.85
Figure 5: Example rollout of model-based RL with VFS as the state representation for robotic manipulation. The robot plans over multiple low-level skills to achieve the semantic goal "move all to top-right cor- ner". Red arrows specify next skill planned by the model, overlaid for visualization purposes only.
Table 3: Success rates for robotic manipulation. VFS outperforms baselines and closely matches orac- ular performance.
the 010 environment, where the longer horizons read to greater error accumulation. VFS constructs an effective representation that captures the state of the scene well as the affordances of the skills, and succeeds in all tasks in the 05 environment, matching the oracle performance. It also succeeds 80% of the tasks in 010, closely matching the oracle's performance of 85%.
To understand the factors captured by VFS, we sample encoded observations from large number of independent trajectories and visualize their t-SNE embeddings (van der Maaten & Hinton, 2008). Figure 6 shows that VFS can successfully capture information about objects in the scene and affordances (c.g. which object in the robot's arm and can be manipulated), while ignoring distractors like the poses of the objects the table and the arm.
Figure 6: A t-SNE diagram of observations encoded by VFS, showing functionally equivalent observations mapped the same representation. VFS discovers clusters with (top) the arm grasping the bowl with apple and chocolate the table, (right) bottle in with glass and chocolate on tahle; (left) sponge in with chocolate on the table. Note that these observations occur across independent trajectories and are unlabeled.
DISCUSSION
We proposed Value Function Spaces state abstractions: a novel skill-centric representation that captures the affordances of the low-level skills. States are encoded into representations that are invariant exogenous factors that do not affect values of these skills. We show that this rep- resentation is compatible with both model-free and model-based policies for hierarchical control, and demonstrate significantly improved performance both terms of successfully performing long- horizon tasks and in terms of zero-shot generalization to novel environments, which leverages the invariances that are baked into our representation.
The focus of our work is entirely on utilizing a pre-specified set of skills, and we do not address how such skills are learned. Improving the low-level skills jointly with the high-level policy could lead to even better performance particularly complex tasks, and would be an exciting direction for future work. More broadly, since our method connects skills directly to state representations, could be used to turn unsupervised skill discovery methods directly into unsupervised representation learning methods, which could be an exciting path toward more general approaches that retain the invariances and generalization benefits of method.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This research was partially supported by ARL DCIST CRA W911NF-I7-2-0181. The authors would like thank Matthew Benice for help with setting up and debugging the robotic manipulation en- vironment, and Ryan Julian for training the skills used in our experiments. The authors would also like thank Sumeet Singh, Katie Kang and Kyle Hsu for feedback on an earlier draft of this paper.
REPRODUCIBILITY STATEMENT
The primary contribution our work, VFS, skill-centric representation designed to work with existing model-base and model-frce pipelines. The implementation simply involves concatenating the value functions corresponding to the available skills into an embedding vector that can be used for HRL planning. We provide all necessary information for setting up VFS to work with reader's s RL algorithm of choice in Sections 5.1, 1, 6.1 and Appendix A. Our model-free experiments open-source gym environment and provide configuration details for setting up our quantitative experiments in Appendix I. We hope that this encourages the community to utilize and build upon the ideas presented in the paper. We plan release more information about the proprietary environments and tasks used in a public release' of this article at a later date.
conducted
REFERENCES
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference Learning Representations, 2021. URL https: http///pppeeeeeeeeeeeeeeeee www.neet/ooo. forum?
id=gda7-sVg84.
Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Sys- tems, 2017. URL
neuriis.....ip..... cecslps.cc/paae cc//paper/2017/Elle Paper. pdf.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI' 17, 2017.
Ershad Banijamali, Rui Shu, Mohammad Ghavamzadeh, Hung Hai Bui, and Ali Ghodsi. Robust locally-linear controllable embedding. In AISTATS, 2018
Andre Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Jonathan hunt, Shibl Mourad, David Silver, and Doina Precup. The option keyboard: Combining skills in reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buce E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL https: httpps:/prroceediinnt http///eeooeee.eccoeeoo neurips. ca/ httppp//w....l///2lecf6622222222
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. Proceedings of the 34th International Conference on Machine Learning. 2017. URL
https: fettss:/proceediine
pttpp/picceednnnnnneee www.mmrrrrere.... press, 7700///lllmaaae 7a hhml.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In AAAI, 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, 2020. URL ptttttss//axxxi.oo/ org/ 0te5/2002.05703
Maxime Chevalier- Chevalier-Boisvertt, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym, 2018. URL https: thtbb
Boisvert,
thub. com/ com/max max mwccm/mmmmmminnnnmmmm
cs. www.bbrkkeeeeyy.. heeeeeelley.ebb/saahh//rsss
10

---

Published as a conference paper at ICLR 2022
Petros Christodoulou. Soft actor-critic for discrete action settings. CoRR, 2019. URL http: //arxiv. wttpp//pgpg...o./o////o /2700
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.
Dane Corneil, Wulfram Gerstner, and Johanni Brea. Efficient model-based deep reinforcement learning with variational state tabulation. In Proceedings of the 35th International Confer- ence on Machine Learning, 2018. URL https: corneill html.
mlr /tt.e//gggg.......//SSS.
Özgür Şimşek and Andrew G. Barto. Using relative novelty to identify useful temporal abstrac- tions in reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning. 2004. URL http:: http:::/ddo...rg.... 10..:.1111/111111 111.1..11111111101111111 10153333.
Özgür Şimşek, Alicia P. Wolfe, and Andrew Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, 2005. ISBN 1595931805. URL https:: /ttts:://oi.... org/10.
1145/1102351 1.055...10244 1102454.
Marco da Silva, Frédo Durand, and Jovan Popović. Linear bellman combination for control of character animation. ACM Trans. Graph., 2009. URL httpps://do http:::///cii.or////// 1145/ 1.531326, 153115831,8
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative tropy policy search. Journal of Machine Learning Research, 2016. hRL /imrm.lrroo org/
papers/v17/15 188 .tmml.
Christoph Dann, Gerhard Neumann, and Jan Peters. Policy evaluation with temporal differences: A survey and comparison. Journal of Machine Learning Research, 15(24)8809-883 883, 2014. URL http: jmlr r .wall..orrg//oe waargg/ppprrsse /v1 5115///aamm/ 5/dann1 4a. html.
Thomas G. Dietterich. Hierarchical reinforcement learning with the maxq value funetion decompo- sition. J. Artif. Int. Res., 2000.
Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning actionable rep- resentations from visual observations. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Annual Conference on Robot Learning, Proceedings of Machine Learning Research, 2017.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Process- ing Systems, 2019. URL https:/pproceeding sewerriiiinnnn.niis..e.......s Sc48ff18e0at wtwa://ffg.eeeeeeeeee22.. Paaple-aappppppppppee
neurips....... http/appe//2119/////
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In ICRA, 2017.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical re- inforcement learning. International Conference on Learning Representations, 2017. URL
httpss:/<penreevier net.iiewwweeeeeeeeeeeeeeeeeeeeee htt.://oorrmmmddd110lfeee.e
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. CoRR, 2017. URL
Dibya Ghosh, Abhishck Gupta, and Sergey Levine. Learnnng actionable representations with goal conditioned policies. In ICLR, 2019. URL Hye91nCet7.
James J. Gibson. The theory of affordances. In Perceiving, acting, and knowing: toward an ecologi- cal psychology. 1977. URL https: : /hal.
fr/ha 1t/hal-00660 006920033.

---

Published as a conference paper at ICLR 2022
Nakul Gopalan, Marie desJardins, Michael L. Littman, James MacGlashan, S. Squire, Stefanie Tellex, John Winder, and Lawson L. S. Wong. Planning with abstract markov decision processes. In ICAPS, 2017.
Anirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Sergey Levine, and Yoshua Bengio. Transfer and exploration via the information bottleneck. In International Conference on Learning Representations, 2019. URL https: /pwpereeeeeeeeeeeeeee/ee/ee
net/ net/forum?
Forum? Jg
Daniel Graves, Johannes Günther, and Jun Luo. Affordance as general value function: A computa- tional model. CoRR, 2020. URL https:://///// /awxivaivvv////////////aa////ai/ org/ abs 22000 14289.
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aäron van den Oord. Shaping Belief States with Generative Environment Models for RL. 2019.
Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. In 5th International Conference Learning Representations, ICLR 2017, 2017. URL https: http:///pppreeee...eoooo ner//orum?? id-Hyg4yhile.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems. URL solicte--oolyyycoootteee
nips. cctccc//ppaa ec/paper/
7512 recurrent- modddddoolll---- facilitate f
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6244-6251, 2018. doi: 10. 1091/
8460756.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.00 1.0455,, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
URL https: http//peeevveeeeeeeeeeeeeeeeeeee net forum?
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. Proceedings of the AAAI Conference on Artificial In- telligence, 32(1), 2018. URL https:// www.aa................ee view/11796.
Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher P. Burgess, Alexander Pritzel, Matthew M. Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. ArXiv, abs/1 707. 707.08475, 2017.
Manfred Huber and Roderic Grupen. Learning robot control using control policies as abstract actions. In In N9PS Workshop: Abstraction and Hierarchy in Reinforcement Learning, 1998.
Jonathan J Hunt, Andre Barreto, Timothy P Lillicrap, and Nicolas Heess. Composing entropic policies using divergence correction, 2019. URL. httpp::://ppeeeeeeeee/ ew. net. forum? ? dd= SJ4272Rctm.
Brian Ichter and Marco Pavone. Robot motion planning in learned latent spaces. IEEE Robotics and Automation Letters, 2019.
Brian Ichter, Pierre Sermanet, and Corey Lynch. Broadly-exploring, local-policy trees for long- horizon task planning. In Annual Conference on Robot Learning, 2021. URL https: /poenreeviww net/forum? (eoormmmmddyhhyy5--rrr
12

---

Published as a conference paper at ICLR 2022
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czamecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations. OpenReview.net, 2017. URL https:
/ //openreviee net./ forum? i d=SJJ
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference on Robot Learning, 2021. URL https: oopereviiee ew. net. forum? dd-9kbb22tssGV.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. CoRR, 2021. URL https: /arxiv.. org/ abs///a///a 2204. 08212.
Martin Klissarov, Pierre-Luc Bacon, Jean Harb, and Doina Precup. Learnings options end-to-end for continuous action tasks. CoRR, abs/1712.00004, 2017. URL
abs/
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Machine Learning: ECML 2006, 2006.
George Konidaris and Andrew Barto. Building portable options: Skill transfer in reinforcement learning. Proceedings of the 20th International Joint Conference on Artificial Intelligence.
George Konidaris and Andrew Barto. Skill discovery continuous reinforcement learning domains using skill chaining. In Y. Bengio, D. Schuurmans, Lafferty, C. Williams, and A. Culotta (eds.), Advances in Neural Information Processing Systems, 2009.
George Konidaris, Leslie Kaelbling, and Tomas Lozano-Perez. Constructing symbolic represen- tations for high-level planning. Proceedings of the AAAI Conference on Artificial Intelligence, 2014. URL https: /ojs. aaai org index.idxx.... phpp Atw.//tttt.aa/////aaaaaaaaaaaaaa
9004.
Ramnandan Krishnamurthy, Aravind S. Lakshminarayanan, Peeyush Kumar, and Balaraman Ravin- dran. Hicrarchical reinforcement learning using spatio-temporal abstractions and deep neural networks. 2016. URL pest:://azaee org/ eabs/1600 05359.
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Joshua Tenenbaum. Hierar- chical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
S. Lange, Martin A. Riedmiller, and Arne Voigtländer. Autonomous reinforcement learning on raw visual input data in a real world application. The 2012 International Joint Conference on Neural Networks (IJNNN,, 2012.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, 2020. URL
https: http:///oooeddddeeeec ngs mlr w.r.ppress...... / /v11 9 lask i n20 html.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforeement learning with a latent variable model. In Advances in Neural Information Processing Systems, 2019. URL http:://aette wtt.
abss///////// / 007.0095553.
Timothy P. Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. CORR, abs/1 509.00971,,000 2016.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In International Conference on Learning Representations, 2019. URL https: : opw.revieeeeeee ew net. forum? i.d=Byeyn055QQ.
Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances, 2021.
13

---

Published as a conference paper at ICLR 2022
Amy McGovemn and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density, In Proceedings of the Eighteenth International Conference on Machine Learning. ICML '01, 2001.
Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut dynamic discovery of sub-goals in rein- forcement learning. In Machine Learning: ECML 2002, 13th European Conference on Machine Learning, volume 2430 of LectureNotes in Computer Science, pp. 295-306, 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015. URL http: / doi. org/100 1038/ naturel4236.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016. URL https:: stttps:/prroceedingg mlr prwss/v48 //miiaail...... html.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning?, 2019.
Ashvin Nair, Vitchyr Pong. Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2017. URL http:ttts//prccceiiiigggo/// neurips. co/ co/paper
pdf.
Doina Precup. Temporal abstraction in reinforcement learning, 2000.
Anil V. Rao. A survey of numerical methods for optimal control. 2009.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. In Advances in Neu- Information Processing Systems, 2018a. URL https: / //procee///////e/e//// ngs neurips citps..ce ca/ p
Matthew
Riemer, Miao Liu, and Gerald Tesauro. In Advances in Neural Information Processing Systems, volume 31, http://sccccpppp///////... pdf.
Learning abstract op-
tions.
2018b.
URL
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi Semi-paramctric topological memory for navigation. In International Conference on Learning Representations, 2018. URL https: / openrevvie.. net./ forum? ddmmm//isssdyddddbb
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In Proceedings of the 32nd International Conference Machine Learning, 2015. URL
.ag.nngsmmm................ wtw.:ng....msssss........m.sss.. vv7 / 15tmm.l....
Jürgen Schmidhuber. Making the world differentiable: On using self self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in stationary environments. Technical report, 1990.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. URL http: www. http://xx..ooo////..//// 11099/101111/111/11111 1w111/19R.....// 2015. 788.
14

---

Published as a conference paper at ICLR 2022
Pierre Sermanet, Corey Lynch, Jasmine Hsu, and Sergey Levine. Time- networks: Self- supervised learning from multi-view observation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics Dynamics-aware aware unsupervised discovery of skills. In International Conference Learning Representations, 2020.
URL https: //oppeeeee ew net wwttteeroorrneeee forum? d=#Jgg.224Kv4
Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its reward: Self-supervision for reinforcement learning. CoRR, 2016. URL http: httt::://axxvvoo. orgpvv..orraaaoee abs/ 1612. 22122.7707.
Freek Stulp and Stefan Schaal. Hierarchical reinforcement learning with movement primitives. In 2011 11th IEEE-RAS International Conference Humanoid Robots, 2011.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. 2018. URL html.
http:/ / ncompl etei deas net/bbook/the-boook-
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelli- gence, 1999. URL wttps///ww..c.eeceiireetttt com/ wtt peccieceearticle//ee $0C0437029900052..
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, and Shie Mannor. A deep hi- erarchical approach lifelong learning in minecraft. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Philip Thomas and Andrew G. Barto. Motor primitive discovery. 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), 2012.
Saket Tiwari and Philip S. Thomas. Natural option critic. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5175-5182, Jul. 2019. URL https: aaa.. org/indez
Emanuel Todorov. Compositionality of optimal control laws. In Advances in Neural Information
Processing Systems, 2009. URL
nwwrips..ccceeeo / Paperr.p.... pap.r.pdf..
Emre Ugur, Erol Şahin, and Ethan Oztop. Predicting future object states using learned affordances. 2009. URL https: //htl handle htt.annleeeeet11111/535335
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- 03748.
tive coding, CoRR, 2018. URL http:
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008. URL http://www. jmlr. org.org///ooo pttps/ppprrs/////////
vardermaatenD3 html.
David Warde-Farley, Tom Van de Wiele, Tejas D. Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-paraeetric discriminative rewards. CoRR, 2018. URL http:// http///xxvvvv.o/ee/ee aass///b///// 1811.11352
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems, 2015.
Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral control using covariance variable importance sampling. CoRR, 2015. URL http:// /axxvv. org/abs 150..... 01149.
Danfei Xu, Ajay Mandlekar, Roberto Martín-Martin, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Deep affordance foresight: Planning through what can be done in the future, 2021.
15

---

Published as a conference paper at ICLR 2022
Denis Yarats, Amy Zhang, llya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv- ing sample efficiency in model-free free reinforcement learning from images. CoRR, abs/1910.01741, 2019. URL http: http::///axxvviieee ptggabs///91// c1741.
Philipp Zech, Simon Haller, Safoura Rezapour Lakani, Barry Ridge, Emre Ugur, and Justus Pi- ater. Computational models of affordance in robotics: taxonomy and systematic classification. Adaptive Behavior, 2017. URL https: //ooi.//oii//// otg/ioorgg1.. 11777709111112337773333
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Con- ference on Learning Representations, 2021a. URL https::/
net forum?
Ld=-2FCWDKRREU.
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations, 2021b. URL https:// openreview net forrmoid=r-- jpmw.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. SOLAR: Deep structured representations for model-based reinforcement learning. In ICML, 2019. URL https: pttppe://rooeeeegggs mmr wwl.ppress/vp zhangl 9m. html.
Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, 2010.
16


Reviews:
[{"review_id": "kBVcGm0AEXY", "reviewer": "Reviewer_WuH2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Major comments**\n\n*Source of options set:* The authors considered the options as pre-defined or learned using an options learning algorithm. However they assume that such a technique to construct the options also provides an associated value function for the options. However there are multiple options learning approaches in the prior research that do not provide an explicit value function for each of those options. Rather, the options are discovered through segmentation. The authors should provide a broader survey of options learning algorithms compatible with their approach and the popular approaches that are not compatible in the interest of better scholarship.\n\n*Positioning with respect to prior work in options-based learning and planning:* The authors idea of using value functions of each of the pre-trained options in the state abstraction is not entirely novel. I can think of at least two works that have proposals along similar lines if not exactly the same. Please refer to the options keyboard by Baretto et al. [1] and AOSM by Rosen et al. [2]. I would like the authors to address their work in context of these works.\n\n*Relationship with transfer learning and option reuse:* Works that attempt transfer learning through the reuse of options, are also themed around constructing better state features or abstractions tied to action availability and feasibility. The authors do not explicitly address this connection, and I would like to see them do that.\n\nI quite like the idea of action abstractions, or action oriented constructions of state abstractions, especially for robotic domains, and this paper does propose a simple but potentially effective idea for achieving it. My primary concern centers around the fact that the presentation in the paper leave a lot to be desired in terms of positioning the paper with prior work, and tying it in with option discovery work that it relies upon. \n\n\n\n[1] - Barreto, André, et al. \"The option keyboard: Combining skills in reinforcement learning.\" NeurIPS(2019).\n\n[2] - Rosen, Eric, et al. \"Building Plannable Representations with Mixed Reality.\" 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.", "summary_of_the_paper": "This paper proposes that given a set of skills or options, the vector of the value functions for the individual skills becomes the abstract state representation in the approach proposed by this paper. The authors then go on to develop a model-free RL algorithm akin to DQN and a model-based planning algorithm for planning in the state-space defined by the value functions of the component skills, and the action-space being the set of executable options. \n\nThe authors demonstrate that the value function space serves as a better representation than some baseline algorithms for representation learning for RL in a maze solving and robotic manipulation task.\n\nThe key assumptions made are as follows:\n - The options are pre-defined and each option has an associated value function\n - The new task is solvable with the defined set of options.", "summary_of_the_review": "I find value function spaces to be a simple but potentially effective idea. The comparisons have largely centered around representation learning frameworks applied to deep-RL, but not around reinforcement learning techniques aimed at option reuse and option discovery. While the approach proposed is different, these works deserve to be discussed in context. As such, I urge the authors to address these concerns qualitatively or empirically.\n\nPost Response:\n\nI am quite happy with the updated version of the paper, and willing to upgrade my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QQUJ03efzgN", "reviewer": "Reviewer_8FMV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths:\n\n(1) The idea of using the value estimates of different pre-existing policies as state representation is, to my best knowledge, novel.\n\n(2) Since the value estimates of pre-trained policies are used directly as states of the higher-level policy, the method can make use of the large set of pre-existing policies available in the wild. \n\n(3) The paper is generally well written, and the narrative is easy to follow.\n\n# Weaknesses: \n\n(1) I have the feeling that the comparison with the baselines is not entirely fair. The baselines have to learn a good state embedding and a good higher-level policy simultaneously, while the proposed method only needs to learn the higher-level policy. However, this completely ignores the fact that the proposed method had access to a much larger corpus of transition since it uses the pre-trained value functions. In contrast, the baselines do not have access to the value functions of the skills. \n\n(2) I would like to see the method evaluated in much more environments. From the existing experiments, it is hard to judge how the method performs in different settings.\n\n# Needs clarification and general comments\n\n(1) How does this method deal with different scalings of the value estimates? Especially in the dense reward case, value estimates for different skills can differ significantly, i.e. because of reward shaping.\n\n(2) Paragraphs 2 and 3 in section 4 repeat more or less what is written in paragraph 4 of the introduction.\n\n(3) To me, it does not feel right to artificially restrict the embedding spaces of the baselines to the maximum size of the VFS embedding space. \nWhat is the justification for that? What are the best-case results for the baselines without that restriction?\n\n(4) In the case of the offline baselines: How many transitions were seen during training of the embedding space compared to the number of transitions used for learning the different skills/policies and their respective value functions?\nSimilarly, in the case of the online baselines: How many transitions were used in total (pre-training plus training of the higher-level policy) to train the proposed method compared to train the baselines (which do not have access to the value function estimates of the skills)?\n\n# Final decision due to rebuttal:\n\nThe authors addressed many of my concerns and provided new experimental results that led to interesting new insights. Hence, I am happy to raise my score from 5 to 6.", "summary_of_the_paper": "The paper proposes a state abstraction in hierarchical reinforcement learning (HRL), called Value Function Spaces (VFS), that is constructed from the value functions of task-conditioned lower-level policies (or skills). The idea is that the value functions capture affordances of the lower-level skills while ignoring task-irrelevant information.\nThe lower-level policies and the corresponding value functions are given a prior, and the state is constructed by concatenating the value estimate of each lower-level policy.\nThe state abstraction is evaluated in a model-free Q-learning and model-based MPC type scenario.", "summary_of_the_review": "The idea of using the value estimates of pre-trained policies is interesting and the empirical results look promising. However, as I pointed out in the weaknesses section, I do not think that the comparison with the baselines is entirely fair. Additionally, I would like to see the method evaluated in more environments. Because of that, I tend towards a weak reject, but I am willing to change my score if the authors can address these issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QNIqBl6D8u", "reviewer": "Reviewer_78KW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting approach by which one can get more out of trained skills; their primary contribution (the novel VFS representation) is easy to construct for skill-driven agents and could therefore serve as a relatively general representation to allow for performance improvement across a range of applications in which skills are already being used to make progress. Overall, though the idea is somewhat simple, it seems to me to be novel and has wide-ranging applications, including a number of applications demonstrated here.\n\nMy biggest concern is the lack of performance metrics beyond simply \"success rate\". Their omission is somewhat surprising and the authors should either include these additional performance metrics or discuss why they are unnecessary. The zero-shot generalization results are indeed promising, but it is hard to really evaluate how well the VFS agent performed compared to the HRL-Target agent (which is trained directly on this problem).\n\"Success-weighted path length\" (SPL), a metric used by the Vision-and-Language community, would be somewhat appropriate, though presenting the \"average performance/cost of successful trials\" would perhaps be more useful. While I am not sure that this addition is *essential* for publication, these results are rather important for understanding the value of the approach and so the authors should make an effort to include them. The authors should either add this metric or adequately justify its omission.\n\nRelatedly, it would be helpful to understand the nature of the failure cases: What does a failure look like? Some understanding of when and why the system fails is essential to understanding where future work is needed. For instance, in Sec 5.2, the agent fails in 32% of the more challenging environments: when it fails, is this a limitation of the low-level skills, of the ability of the agent to correctly predict the transition model, or both? Even if the authors do not readily have the answers to these questions, a few sentences commenting on the nature of experiments that would be necessary to answer them would be a welcome addition to the paper and help to improve the understanding of both readers and those that would build upon this approach. Similarly, what are the limitations of this approach? When would we expect that it would not be effective or fail? The Discussion section touches upon some of the potential directions for future improvement, but additional comments on the limitations of this version would help understanding. Additional experiments are not necessary here (nor are specific examples); a high-level discussion of these questions will suffice.\n\nQuestions and smaller suggestions:\n- [Sec 6.1] Where does the \"goal latent sate\" come from? My intuition is that there would a non-latent goal state provided to the planner, which would be subsequently \"encoded\" into its VFS form. Is that correct? Either way, this should be clarified in the paper (along with, perhaps, a more comprehensive overview of the planning procedure).\n- [Figure 1(b)] Though it is mostly clear from context, using an \"equal\" sign is not quite correct here, since the states themselves are in fact different. It is certainly possible to understand what the authors are going for, but changing the figure to reflect that applying the VFS encoding function makes all these states equal to one another would be a more appropriate way of communicating this knowledge, especially important since this figure is supposed to outline the core of the authors approach.\n- [Figure 6] What are the skills available to the agent in this environment? Adding a few examples would help the reader understand the clusters better, though the idea that \"the pose of objects is distractor information\" is already clear from the figure.", "summary_of_the_paper": "This paper presents Value Function Spaces (VFS), an abstract \"skill-centric\" representation for reinforcement learning and long-horizon planning. The core idea of VFSs is to leverage the learned value functions that are often trained alongside skills during skill learning to abstract knowledge about the world. Each element of this low-dimensional representation corresponds to the value estimated from one of the learned value function, so that the representation (by construction) tends to ignore distractor information unnecessary for accomplishing the skills. Using this representation on a set of pre-trained skills, the authors demonstrate improved performance on long-horizon planning tasks in grid-based maze and \"locked door\" environments, showing improved success probability on a number of such tasks compared to competitive baselines. They also show the ability of their abstraction to facilitate model-based goal-directed planning by learning a state-transition model (in VFS-space) that allows one to predict the outcome of executing a skill.", "summary_of_the_review": "This paper presents an interesting approach by which one can get more out of trained skills; their primary contribution (the novel VFS representation) is easy to construct for skill-driven agents and could therefore serve as a relatively general representation to allow for performance improvement across a range of applications in which skills are already being used to make progress. One of my primary concerns is the lack of inclusion of cost-based performance metrics for their experiments, which I feel are somewhat important for understanding the performance of the approach. Even without them, I think this work represents a (mostly) well-executed, interesting idea, and so I am generally in favor of publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PdT3MCjBOhA", "reviewer": "Reviewer_aM2S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------------\n\n- Well written paper: clear, to the point and nice to read\n- VFS is a very interesting, simple and elegant state representation method\n- VFS appears to work very well in practice (if the low-level skills are selected/trained by hand). I particularly liked the manipulation example: this is a challenging task and VFS seem to be quite effective\n\nWeaknesses\n------------------\n\n- In the HRL setting, the most challenging task is discovering and identifying the low-level skills automatically. This is not touched at all by the paper. Rather the authors pick well-selected and fully trained low-level policies. I would have liked to see one of the following:\n    - A setting where the low-level skills are not \"perfect\". A few examples consist: skills that overlap, skills that are not well-trained, skills that achieve the desired effect but in a sub-optimal way (e.g. make a big curve to reach a point), etc..\n    - Experiments on how to integrate VFS inside a full HRL pipeline that automatically discovers low-level skills and uses them for planning with VFS representation; this is would actually be the use case of VFS (since it is assuming a rather prior-free setting).\n\nWithout one of the two it is actually difficult to assess if VFS is actually useful.\n- \"*in Figure 1b, states with varying object or arm positions (i-iii), different background textures (iv), and distractor objects (v) are functionally equivalent for planning, and map to the same embedding in our representation*\" -> How is this true? Isn't $Z_t = [V_{o1}(s_t), ..., V_{ok}(s_t)]$? Passing the raw image state inside the VF learned by an RL algorithm will definitely affect the outcome and thus the high-level states will not necessarily be close to each-other. I'd like to see more discussion on this as this is quite crucial.\n- Similarly, \"*This representation captures positional information about the contents of the scene, preconditions for interactions, and the effects of executing a feasible skill, making it suitable for high-level planning*\" -> Although I understand what the authors want to convey, this sentence is overselling what VFS do. Or at least there is no real validation in the paper suggesting that it does all of those things.\n- By the way, especially in the MiniGrid world (that has a field of view of the agent as the raw observation) many (different) states will actually be encoded very similarly using VFS (see the Go-Explore paper {1} for a big discussion on this). How is the algorithm addressing this? What is the intuition behind the success in the MiniGrid world?\n- The success rate metric is nice but not the only interesting metric. E.g. in the manipulation task, the algorithm might be able to position the objects but make unnecessarily big curves. At least a cumulative reward metric should also be defined (even if this is not given to the algorithms for learning).\n- There is no mention of the neural network architectures for the VFs.\n\nReferences\n----------------\n\n{1}: Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K.O. and Clune, J., 2021. First return, then explore. Nature, 590(7847), pp.580-586.", "summary_of_the_paper": "The paper presents a novel state representation technique that is based on Value Functions (VFs). The core idea of the paper is to use VFs to construct a high-level space representation in an hierarchical reinforcement learning (RL) scenario. The contributions of the paper are:\n\n- Value Function Spaces (VFS): learned state representation\n- Practical algorithms with VFS both for model-free and model-based RL settings\n- Evaluation of VFS in 2 scenarios (MiniGrid and manipulation task)", "summary_of_the_review": "The main idea of the paper is interesting and as far as I can tell novel. The results indicate that it is working well. However, the most difficult part in an hierarchical RL setting is actually discovering the low-level skills and how to identify them automatically. The paper is not touching this at all and assumes access to pre-defined (well-selected) low-level skills.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is good but at a borderline. One reviewer increased the score during the discussions. However, no reviewer was in strong favor. So that this paper is still a borderline one, and it is up to the SAC to decide.
Decision: Accept (Poster)