Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning
Authors: Jan Achterhold
Markus Krimmel
Joerg Stueckler
Venue: CoRL 2022 Oral (2022)
Keywords: temporally extended skill learning
hierarchical reinforcement learning
diverse skill learning

Abstract: Problems which require both long-horizon planning and continuous control capabilities pose significant challenges to existing reinforcement learning agents. In this paper we introduce a novel hierarchical reinforcement learning agent which links temporally extended skills for continuous control with a forward model in a symbolic discrete abstraction of the environment's state for planning. We term our agent SEADS for Symbolic Effect-Aware Diverse Skills. We formulate an objective and corresponding algorithm which leads to unsupervised learning of a diverse set of skills through intrinsic motivation given a known state abstraction. The skills are jointly learned with the symbolic forward model which captures the effect of skill execution in the state abstraction. After training, we can leverage the skills as symbolic actions using the forward model for long-horizon planning and subsequently execute the plan using the learned continuous-action control skills. The proposed algorithm learns skills and forward models that can be used to solve complex tasks which require both continuous control and long-horizon planning capabilities with high success rate. It compares favorably with other flat and hierarchical reinforcement learning baseline agents and is successfully demonstrated with a real robot.

Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning
Markus Krimmel Joerg Stueckler Embodied Vision Group, Max Planck Institute for Intelligent Systems, Tübingen, Germany {jan achterholdcc,c..o. markus. krimmel, , joerg. stueckler
Jan Achterhold
mpg de
Abstract: Problems which require both long-horizon planning and continuous control capabilities pose significant challenges to existing reinforcement learning agents. In this paper we introduce a novel hierarchical reinforcement learning agent which links temporally extended skills for continuous control with forward model in a symbolic discrete abstraction of the environment' ' state for planning. We term agent SEADS for Symbolic Effect-Aware Diverse Skills. We formulate an objective and corresponding algorithm which leads to unsupervised learning of a diverse set of skills through intrinsic motivation given a known state abstraction. The skills are jointly learned with the symbolic forward model which captures the effect of skill execution in the state abstraction. After training, we can leverage the skills as symbolic actions using the forward model for long-horizon planning and subsequently execute the plan using the learned continuous-action control skills. The proposed algorithm learns skills and forward models that can be used to solve complex tasks which require both continuous control and long-horizon planning capabilities with high success rate. It compares favorably with other flat and hierarchical reinforcement learning baseline agents and is successfully demonstrated with a real robot. Project page: https: /tops://seeee //sead//saaaa iss tue tue.mpg.e........... de
temporally extended skill learning, hierarchical reinforcement learning, diverse skill learning
Keywords:
1 Introduction
Reinforcement learning (RL) agents have been applied difficult continuous control and discrete planning problems such as the DeepMind Control Suite [1], StarCraft [2], or Go [3] in recent years. Despite this tremendous success, tasks which require both continuous control capabilities and long-horizon discrete planning are classically approached with task and motion planning [4]. These problems still pose significant challenges to RL agents [5]. An exemplary class of environments which require both continuous-action control and long-horizon planning are physically embedded games as introduced by [5]. In these environments, a board game is embedded into a physical manipulation setting. A move in the board game can only be executed indirectly through controlling a physical manipulator such as a robotic arm. We simplify the setting of [5] and introduce physically embedded single-player board games which do not require to model the effect of an opponent. Our experiments support the findings of [5] that these environments are challenging to solve for existing flat and hierarchical RI. agents. In this paper, we propose novel hierarchical RL agent for such environments which learns skills and their effects a known symbolic abstraction of the environment.
As a concrete example for proposed embedded singlc-player board game we refer to the Lights- OutJaco environment (see Fig. 1). Pushing a field on the LightsOut board toggles the illumination state (on off) of the field and its non-diagonal neighboring fields. A field on the board can only be pushed by the end effector of the Jaco robotic arm. The goal is to reach a board state in which all fields are off. The above example also showcases the two concepts of state and action abstraction in decision making [6]. A state abstraction function ((xx) only retains information in state 8, which is relevant for a particular decision making task. In the LightsOut example, to decide which move to perform next (i.e., which field to push), only the illumination state of the board relevant.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Light:DutCCrrsor
T:leSvapCaraor
LightaOucRRachar
TileSvapRaachar
LightsOut.Jaco
PileSvaplacc
LightsOut (with on (red) and off (gray) fields) and TileSwap (fields in a rhombus are swapped if pushed inside) board games embedded into physical manipulation settings, move the board game can only indirectly be executed through controlling manipulator.
Figure I:
move can considered an action abstruction: skill, i.e. high-level action (e.g.. push top-left field), comprises a sequence of low-level actions required to control the robotic manipulator.
We introduce a two-layer hierarchical agent which assumes discrete state abstraction 2, (st) Z to be known and observable in the environment, which we in the following refer to as symbolic observation. In our approach, we assume that state abstractions can be defined manually for the environment. For LightsOut, the symbolic observation corresponds to the state of each field (on/off). We provide the state abstraction as prior knowledge about the environment and assume that skills induce changes of the abstract state. Our approach then learns a diverse set of skills for the given state abstraction as action abstractions and a corresponding forward model which predicts the effects of skills on abstract states. In board games, these abstract actions relate to moves. We jointly learn the predictive forward model qo and skill policies π (a 8t: k) for low-level control through an objective which maximizes the number of symbolic states reachable from any state of the environment (diversity) and the predictability of the effect of skill execution. Please see Fig. for an illustration of the introduced temporal and symbolic hierarchy. The forward model go can be leveraged to plan sequence of skills to reach a particular state of the board (e.g., all fields off), i.c. to solve tasks. We evaluate our approach using two single-player board games in environments with varying complexity in continuous control. We demonstrate that our agent learns skill policies and forward models suitable for solving the associated tasks with high success rate and compares favorably with other flat and hierarchical RL baseline agents. We also demonstrate our agent playing LightsOut with a real robot.
In summary, we contribute the following: (1) We formulate a novel RI. algorithm which, based on state abstraction of the environment and an cnoomatiion-thoreee objective, jointly learns a diverse set of continuous-action skills and a forward model capturing the temporally abstracted effect of skill exccution in symbolic states. (2) We demonstrate the superiority of our approach compared to other flat and hierarchical baseline agents in solving complex physically-embedded single-player games, requiring high-level planning and continuous control capabilities. We provide additional materials, including video and code, at https: /tctts://seeee //seads. is. tue mmg.mp.................... de.
2 Related work
Diverse skill learning and skill discovery. Discovering general skills to control the environment through exploration without task-specific supervision is a fundamental challenge in RL research. DIAYN [7] formulates skill discovery using an information-theorrtie objective as reward. The agent learns a skill-conditioned policy for which it receives reward if the target states can be well predicted from the skill. VALOR [8] proposes to condition the skill prediction model on the complete trajectory of visited states. Warde-Farley et [9] train a goal-conditioned policy to reach diverse states in the environment. Variational Intrinsic Control [10] proposes to use an information-theorel objective learn set of skills which can be identified from their initial and target states. Relative Variational Intrinsic Control [11] seeks to learn skills relative to their start state, aiming to avoid skill representations that merely tile the state space into goal state regions. Both approaches do not learn a forward model on the effect of skill execution like our approach. Sharma et al. [12] propose a model-based RL approach (DADS) which learns a set of diverse skills and their dynamics models using
based exploration. While DADS learns skill dynamics as immediate behavior g (8++11 (s,, k), we learn transition model on the effect of skills g( Fo k) in a symbolic abstraction, thereby featuring temporal abstraction.

---

xt rtae |sc, k)
(rooloo Ag)
stoelee,
#1
(s0)
(srr
((sn) gotara) 1u)) As)
go(ar 20,
go(=(1) I 4cp). Rx)
Sco;
2T 2(13
(a) Temporal abstraction induced by skills | k) with associ- ated forward forward model on symbolic observations 2(n))
(b) Symbolic abstraction & and tempo- ral skill abstraction demonstrated on the LightsOutJacc environment. The sym- bolic observation x represents the discrete state of the board while s contains both the board state and the state of the Jaco manipu- lator. Executing skill & by applying the skill policy at-1- k) until termination leads to change of the state of the board, which modeled by qe with a single action l..
single action & transforming the symbolic observation 2(x) + 2(1++1) denoted by the bracketed (-) time indices. Here, two skills (ki, k2) are executed subsequently on the environment until ter- mination, taking (Ti, T2) steps, respectively. The temporally abstract effect of skill execution the symbolic observation captured by the forward model qo.
Exccuting skill until termination can be interpreted
Figure 2: We aim to learn skills with associated policies T ( 1t; k) which lead to diverse and predictable (by the forward model qæ) transitions in a symbolic abstraction 2 = (b(s of the state s.
Hierarchical RL. Hierarchical RL can overcome sparse reward settings and time extended tasks by breaking the task down into subtasks. Some approaches such as methods based on MAXQ [13, 14] assume prior knowledge on the task-subtask decomposition, In SAC-X [15], auxiliary tasks assist the agent in learning sparse reward tasks and hierarchical learning involves choosing between tasks. Florensa et al. [16] propose to learn a span of skills using stochastic neural networks for representing policies. The policies are trained in task-agnostic way using a measure of skill diversity based on mutual information. Specific tasks are then tackled by training an RL agent based on the discovered skills. Feudal approaches [17] such as HIRO [18] and HAC [19] train high-level policy to provide subgoals for a low-level policy. In our method, we impose that a discrete state-action representation exists in which learned skills are discrete actions, and train the discrete forward model and the continuous skill policies jointly. Several approaches to hicrarchical RL are based on the options framework [20] which learns policies for temporally extended actions in two-layer hierarchy. Learning in the options framework is usually driven by task rewards. Recent works extend the framework to continuous spaces and discovery of options (e.g. [21, 221). HiPPO [23] develop an approximate policy gradient method for hierarchies of actions. HIDIO [24] learns task-agnostic options using a measure of diversity of the skills. In our approach, given state abstraction) hierarchical representations using a measure of intrinsic motivation. However, an important difference is that we do not learn high-level policies over options using task rewards, but learn lkill-conditions forward model suitable for planning to reach a symbolic goal state. Jointly, continuous policies are learned which implement the skills. Several approaches combine symbolic planning in a given domain description (state and action abstractions) with RL to execute the symbolic actions [25, 26, 27, 28, 29]. Similar to our approach, the work in Guan et al. [29] learns low-level skill policies using an
also learn task-agnostic (for the
diversity measure which implement known symbolic actions. Differently, we learn the action abstraction and low-level skills given the state abstraction.
Representation learning for symbolic planning, Some research has been devoted to learning representations for symbolic planning. Konidaris al. [30] propose method for acquiring symbolic planning domain from a set of low-level options which implement abstract symbolic actions. In [31] the approach extended to learning symbolic representations for families of SMDPs which describe options in a variety of tasks. Our approach learns action abstractions as a set of diverse skills given a known state abstraction and a termination condition which requires abstract actions change abstract states. Toro Icarte et al. [32] learn structure and transition models of finite state machines through RL. Ugur and Piater [33] acquire symbolic forward models for predefined low-level action repertoire in a robotic manipulation context. Chitnis et al. [34] concurrently learn transition models on the symbolic and low levels from demonstrations provided the form hand-designed policies, and use the learned models for bilevel task and motion planning. The approach also assumes the state abstraction function to be known. In [35] different setting is considered which the symbolic transition model is additionally assumed known and skill policies that execute symbolic actions
3

---

are learned from demonstrations. Other approaches such as DeepSym [36] or LatPlan [37] learn mappings of images to symbolic states and learn contiioconiiiinnll forward models. In [37] symbolic state-action representations are learned from image observations of discrete abstract actions (e.g. moving puzzle tiles to discrete locations) which already encode the planning problem. Our approach concurrently learns a diverse set of skills (discrete actions) based on an information theoorrii intrinsic reward and the symbolic forward model. Differently, in our approach low-level actions are continuous.
3
Method
Our goal is to learn a hierarchical RL agent which (i) enables high-level, temporally abstract planning to reach a particular goal configuration of the environment (as given by a symbolic observation) and (ii) features continuous control policies to execute the high-level plan. Let S, AA denote the state and action space of an environment, respectively. In general. by Z = {0. 1) P we denote the space of discrete symbolic environment observations E Z and assume the existence of a state abstraction :SS- - Z. The dimensionality of the symbolic observation D is environment-dependen. For the LightsOutlaco environment, the state [g. q. 2] E S contains the robot arms' joint positions and velocities (g. q) and binary representation of the board 2 E {0. 1],x6, The action space A is equivalent to the action space of the robotic manipulator. In the LightsOutJacr example, it contains the target velocity of all actuable joints. The discrete variable & K = {1, K} refers to a particular skill, which we will detail in the following. The number of skills K needs to be set in advance, but can be chosen larger than the number of actual skills.
We equip our agent with symbolic planning and plan execution capabilities through two components: First, a forward model å =fz, symbolic successor states & of the current symbolic state by iterating over the discrete variable k. This allows for node expansion in symbolic planners. Second, a family of discretely indexed policies π A S x K + R. a ~ <(a St. k) aims to steer the environment into a target state ST for which holds that ((s)) = 2, given that ((sg) = 2 and
argmax_ 90 2, k) allows to enumerate all possible
argmax./ 90 (a') 2 k) (see Fig. 2). We can relate this discretely indexed family of policies to set of K options [20]. An option is formally defined as a triple o% (Ik, Br, T&) where I C is the set of states in which option l is applicable, Ba S x S - [0, 1). (x 8x) parametrizes Bernoulli probability of termination in state St when starting in so (f.c. when detecting an abstract state change) and T& (a st): S (AA) is the option policy
the action space A. We will refer to the option policy as skill policy in the following. We assume that all options are applicable in all states, i.c., 4x = S. An option terminates if the symbolic state has changed between so and 8t a timeout is reached, i.e. 3& (Su, st) = 1 [ ($s) * ((s)) Vt= f.ax]. To this end, we append a normalized counter t/funax to the state st. We define the operator apply sT apply(E, π,, 8o, k) which applies the skill policy [(a st, k) until termination on environment E starting from initial state Su and returns the terminal state 8T. We also introduce bracketed time notation which abstracts the effect of skill execution from the number of steps T taken until termination 8(x) = apply (E, म. ((n-1); k) with n E No (see Fig. 2a). The apply operator can thus be rewritten as $(1) = apply(E, म, s(o) k) with S(o) = 50, $(1) = ST. The symbolic forward model qo (2T | zg, k) aims to capture the relation of 20, ic and 2T for ST apply [E, ग. so. k) with 2g = ((s)). 27 = ( The model factorizes over the symbolic observation as (0(2r) k. 20) = -T2 ( k. an) = No Bernoulli(arr]] la). The Bernoulli probabilities ar: : 2 x XC + (0, 1) are predicted by a neural component. We use a multilayer perceptron (MLP) so which predicts the probability ptip of binary variables in zg toggle Ptip = fo(2o. k).. The index operator [xr]aa returns the th element of vector x.
Objective For any state 8o E S with associated symbolic state =p = ((8)) we aim to learn K skills T(a| st, k) which maximize the diversity in the set of reachable successor states {24 = (apply(EE, म. Sp, k)]lk E K.. Jointly, we aim to model the effect of skill exccution with the forward model qo (2r 20. k)). Inspired by Variational Intrinsic Control [10] we take an information-theoree perspective and maximize the mutual information I(2r, k 20) between the skill index & and the symbolic observation 2T at skill termination given the symbolic observation 2g at skill initiation, i.c.,
max I(ET, A 20) = max H (ZT | 20) H (2TT) zo. k)). The intuition behind this objective function is that we encourage the agent to (i) reach diverse set of terminal observations 27 from an initial
observation 2g (by maximizing the conditional entropy (2T =0)) and (ii) behave predictably such
that the terminal observation 27 is idcally fully determined by the initial observation 20 and skill

---

index k (by minimizing (2r I 20, k)). We reformulate the objective as an expectation over tuples (so, k ST) by employing the mapping function & as [[ZT, k =0) = (r.r.k,0))~P log p(ar so.k) with 2T := ((TT), 2o := 2(50) and replay buffer P. Similar to [12] we derive a lower bound on the mutual information, which is maximized through the interplay of RL problem and maximum
plær fo)
likelihood estimation. To this end, we first introduce a variational approximation (0 20, k) to the
transition probability p(2r | 20, k), which model by neural component. We decompose
I(Zr, k z0)= (e log (= 20,k) k)
p(2r =0, k)
+ (( log
p(er 20)
qo (2T 20, k))
DDKL k) (or so,
20(er so.k) whose maximization giving rise to the lower bound I(2r, k | z0) 2 16 (s0 [log
p(ar Ra) go(er Ro. We
can be interpreted as sparse-reward RL problem with reward RT(K) log
p(ar =o)
approximate p(2r | zo) as p(2r z0) ~ Σx go(2r z0, k') H))p(k' 20) and assume k uniformly
distributed and independent of zo. i.e. p(k' zo) = K. This yields a tractable reward
qa(2T z0, 20,k)
RT = log E. (0
+ log K.
(1)
zo; =0)
In Sec. 3 we describe modifications we apply to the intrinsic reward Ry which improve the perfor- mance of our proposed algorithm. To tighten the lower bound, the KL. divergence term in eq. (1) has to be minimized. Minimizing the KI. divergence term corresponds to "training" the symbolic forward model go by maximum likelihood estimation of the parameters 0.
Training procedure In each epoch of training, we first collect skill trajectories on the environment using the skill policy æ. For each episode i E {1, (1.....] N we reset the environment and obtain an initial state sy. Next, we uniformly sample skills k² ~ (niffrmm[,.... K.. By iteratively applying the skill policy π (-),, k*) we obtain resulting states sf. and actions ag...ar; 1- A skill rollout terminates step-limit is reached or when a change in the symbolic observation 2, F 2g observed. We append the rollouts to limited-size buffer B. In each training epoch we sample two sets of episodes SRL, SPM from the buffer for training the policy æ and symbolic forward model qu. Both episode sets are relabelled as described in Sec. Let E {1. [1.....] .M now refer the episode index in the set SRI. From SRI. we sample transition tuples ([st, [s k*),, af, ri- (k*))
either if an environment-depeen
which are used to update the policy и using the soft actor-critic (SAC) algorithm [38]. To condition
the policy on skill & we concatenate k to the state as denoted by [. We set the intrinsic reward to
zero ri+1 = O except for the last transition in an episode (t + = T2) in which (k*) = R(k)).
From the episodes in SpM we form tuples (zj = ((s)), k., 2 = (ss;:) which are used to train
the symbolic forward model using gradient descent.
Relabelling Early in training, the symbolic transitions caused by skill executions mismatch the predictions of the symbolic forward model. We can in hindsight increase the match between skill transitions and forward model by replacing the actual ked which was used to collect the episode i by different kt in all transition tuples ([st, kg), [si+1: kt], at. r (ktt) and (z), hi, =1,) ) of episode i. In particular, we aim to replace k by k which has highest probability k = max; go(k: 2r, z0). However, this may lead to an unbalanced distribution over K. after relabelling which is no longer uniform. To this end, we introduce constrained relabelling scheme as follows. We consider set of episodes indexed by i E (1,..., N} and compute skill log-probabilities for each episode which ga(=t, k))a))
we denote by Q log (k) =t 24:)) where (k) = = EE qu(=), 25) We find relabeled
skill for each episode (k., RNN) which maximizes the scoring max(u!...... E, Qi under the constraint that the counts of re-assigned skills (k]...,k?") and original skills (A* ...., k- match, i.e. #N [k] = k] # [k = k] Yk E (1...., K} which is to ensure that after relabelling no skill is over- or underrepresented. The count operator #I-1 counts the number of positive (true) evaluations of its argument in square brackets. This problem can be formulated as a linear sum assignment problem which we solve using the Hungarian method [39,40] 40]. While we pass all episodes in SPM to the relabelling module, only a subset (50%) of episodes in SRI. can potentially be relabeled to retain negative examples for the SAC agent. Relabelling experience in hindsight to improve sample efficiency is common approach in goal-conditioned [41] and hierarchical [19] RL.
5

---

LightsOurCursor
TileSwapCursor
LightsOutReacher
TileSwapReacher
LaghtsOut.aco
TileSwap.lace
Average unique moves
25
25
0.5M
0.5M
5M
10M
5M
10M
5M
10M
5M
10M
Env. steps
Env. steps
Env. steps
Env. steps
Env. steps
Env. steps
SEADS
norm.
More skillss
No relabelling
nov. banus
LighrSOutCuroor
TileSwapCursor
Leghte
LightsOutllaco
TileSwap.aac
Success rate (%)
100
0.5M
0.5M
5M
5M
10M
5M
10M
5M
10M
Env. steps
Env. steps
Env. steps
Env. steps
Env. steps
Env. steps
SEADS (replan.)
SEADS (no replan.)
SAC
HAC
Top row: Number of learned unique game moves for ablations of SEADS. Bottom row: Success rate of the proposed SEADS agent and baseline methods LightsOut, TileSwap games embedded in Cursor, Reacher, Jaco environments. SEADS performs comparably or outperforms the baselines on all tasks. The solid line depicts the mean, shaded area min. and max. of 10 (SEADS, SAC on Cursor) / 5 (HAC, SAC on Reacher, Jaco) independently trained agents.
Figure 3:
Reward improvements The reward in eq. (1) can be denoted as R(k) log qo (k:) zo, 2T) log K. For numerical stability. we define a lower bounded term Qk = clip(log go 2o.2T) min = loo)(K)) and write (k) Qx + log K. In our experiments, agent is stuck in a local minimum in which (i) the learned skills are not unique, i.c., two or more skills k KC cause the same symbolic transition 20 27. In addition, (ii), occasionally, not all possible symbolic transitions are discovered by the agent. To tackle (i) we reinforce the policy π with a positive reward if and only if other skill k'' better fits the symbolic transition (zo - 2r) generated by apply(E, E, π. So, k), i.e., Rnorm (k) = Qx to which we call second-best normalization. The operator top2gr selects the second-highest value of its argument for k' K. We define Rbase(k) Ruorm (k) except for the "No second-best norm." ablation where Rbase (k) R2" (k). To improve (ii) the agent obtains novelty bonus for transitions (zo + 2T) which
observed that occasionally the
modeled by the symbolic forward model for any k' by R(k) Rbase (k) max log (2 20, k). If the symbolic state does not change (2T = 20), we set R(k) -2 log (K) (minimal attainable reward).
Planning and skill execution A task presented to our agent as an initial state of the environ- ment So with associated symbolic observation 20 and symbolic goal 2². First, we leverage learned symbolic forward model go to plan a sequence of skills ki,... KN from 20 to z using breadth-first search (BFS). We use the mode of the distribution over =' for node expansion in BFS: successor qe (z, k) = argmaxy'ez (=') k)). After planning, the sequence of skills [K1, kx is iteratively applied the environment through (n)) apply(Ei ग,, (n k2). Inaccuracies of skill execution (leading to different symbolic observations than predicted) can be coped with by replanning after each skill execution. Both single-outcome (mode) determinisation and replanning are common approaches to probabilistic planning [42]. We provide further details in the supplementary material.
4 Experiments
We evaluate our proposed agent on set of physically-embeddee game environments. We follow ideas from [5] but consider single-player games which in principle enable full control over the environment without the existence of an opponent. We chose LightsOut and TileSwap as board games which are embedded in physical manipulation scenario with Cursor, Reacher or Jaco manipulators (see Fig. 1). The LightsOut game (see Figure 1) consists of 5 board of fields. Each field has a binary illumination state of on or off. By pushing field, its illumination state and the state of (non-diagonally) adjacent fields toggles. At the beginning of the game, the player is presented a board where some fields are and the others are off. The task of the player is to determine a set of fields to push to obtain board where all fields are off. The symbolic observation in all LightsOut environments represents the illumination state of all 25 fields on the board 2 {0, 1 136x5

---

In TileSwap (see Fig. 1) 3 x 3 board is covered by chips numbered from O to 8 (each field contains exactly one chip). Initially, the chips are randomly assigned to fields. Two chips can be swapped if they are placed on (non-diagonally) adjacent fields. The game is successfully finished after a number of swap operations if the chips are placed on the board in ascending order. In all TileSwap environments the symbolic observation represents whether the i-th chip is located on the j-th field 2 = {0, 1 19 To ensure feasibility, we apply a number of random moves (pushes/swaps) to the goal board configuration of the respective game. We quantify the difficulty of particular board configuration by the number of moves required to solve the game, We ensure disjointness of board configurations used for training and testing through hashing algorithm (see supp. material). A board game move ("push" " in LightsOut, "swap" in TileSwap) is triggered by the manipulator's 's end effector touching particular position on the board. We use three manipulators of different complexity (see Fig. 1). The Cursor manipulator can be navigated on the 21D game board by commanding x and displacements. The board coordinates are x. y [0. 1], the maximum displacement per timestep is Ar. Ay = 0.2. A third action triggers a push (LightsOut) or swap (TileSwap) at the current position of the cursor. The Reacher [1] manipulator consists of two-link arm with two rotary joints. The position of the end effector in the 2D plane can be controlled by external torques applied to the two rotary joints. As for the Cursor manipulator an additional action triggers game move at the current end effector coordinates. The Jaco manipulator [43] is 9-DoF robotic arm whose joints are velocity-controlle underlying board to trigger game moves. The arm is reset to random configuration above the board around the board's center after a game move (details in supplementary material). By combining the games of LightsOut and TileSwap with the Cursor, Reacher and Jaco manipulators we obtain six environments. For the step limit for skill execution we set 10 steps on Cursor and 50 steps in Reacher and Jaco environments. With our experiments we aim at answering the following research questions: R1: How many distinct skills are learned by SEADS? Does SEADS learn all 25 (12) possible moves in LightsOut (TileSwap) reliably? R2: How do our design choices contribute to the performance of SEADS? R3: How well does SEADS perform in solving the posed tasks in comparison to other flat and hierarchical RL approaches? R4: Can our SEADS also be trained and applied on a real robot?
at 10Hz. has an end-effector with three "fingers" which can touch the
Skill learning evaluation To address RI we investigate how many distinct skills are learned by SEADS. If not all possible moves within the board games are learned as skills (25 for LightsOut, 12 for TileSwap), some initial configurations can become unsolvable for the agent, negatively impacting task performance. To count the number of learned skills we apply each skill k E {L...., K) on a fixed initial state So of the environment E until termination (i.c., apply 80, n, k)). Among these K skill executions we count the number of unique game moves being triggered. We report the average number of unique game moves for N = 100 distinct initial states sy. On the Cursor environments SEADS detects nearly all possible game moves (avg. approx. 24.9 of 25 possible in LightsOutCursoor, 12 of 12 in TileSwapCursor). For Reacher almost all moves are found (4..3/1..8.. In the Jaco environments some moves are missing occasionally 23.6/11.5). We demonstrate superior performance compared to a baseline skill discovery method ( (Variational Intrinsic Control, Gregor al. [10] in the supplementary material. We substantiate our agent design decisions through an ablation study (R2) in which we compare the number of unique skills (game moves) detected for several variants of SEADS (see Fig. 3). In a first study, we remove single parts from our agent to quantify their impact on performance. This includes training SEADS without the proposed relabelling, second-best normalization and novelty bonus. We found all of these innovations to be important for the performance of SEADS, with the difference to the full SEADS agent being most prominent in the LightsOut.Jaco environment. Learning with more skills (15 for TileSwap, 30 for LightsOut) than actually needed does not harm performance.
Task performance evaluation To evaluate the task performance of our agent and baseline agents (R3) we initialize the environments such that the underlying board game requires at maximum 5 moves (pushes in LightsOut, swaps in TileSwap, limited due to branching factor and BFS) to be solved. We evaluate each agent on 20 examples for each number of moves in (1. 5) required to solve the game. We consider a task to be successfully solved if the target board configuration was reached (all fields off in LightsOut, ordered field in TileSwap). For SEADS we additionally count tasks as "failed" if planning exceeds a wall time limit of 60 seconds. We evaluate both planning variants with and without replan- ning. As an instance of flat hinn-hierarhhical) agent we evaluate the performance of Soft Actor- Critic (SAC [38]. The SAC agent receives the full environment state & S which includes the symbolic observation (board state). It obtains reward of 1 if it successfully solved the game and 0 otherwise.
7

---

In contrast to the Soft Actor-Critic agent the SEADS agent leverages the decomposition of state s S and symbolic observation E Z. For fair comparison to a hierarchical agent, we consider Hierarchical Actor-Critio (HAC, Levy et al. [19]), which, similar to SEADS, can also leverage the decomposition of and 2. We employ two-level hierarchy in which the high-level policy sets symbolic subgoals 2 E Z to the low-level policy, thereby leveraging the access to the symbolic observation. We refer to the supplementary material for implementation and training details of SAC and HAC. Fig. 3 visualizes the performance of SEADS and the baselines. On all environments SEADS performs similar or outperforms the baselines. with the performance difference being most pronounced on the Jaco environments on which SAC and HAC do not make any progress. On the Cursor environments SEADS achieves a success rate of 100% after 10 environment steps. On the remaining environments, the average success rate (with replanning) is 95.8% (LightsOutReachee (LightsOut.Jacoo, 98.8% (TileSwapJaco) after 10" steps.
Figure 4: Robot setup.
95.5% (Til (TileSvapReacher) 94.9%
Robot experiment To evaluate the applicability of our approach on a real-world robotic system (R4) set up testbed with a uArm Swift Pro robotic arm which interacts with tablet using capacitive pen (see Fig. 4). The SEADS agent commands a displacement |Ar), | A S 0.2 and an optional pushing command as in the Cursor environments. The board state is communicated the agent through the tablet's USB interface. We manually reset the board once at the beginning of training, and do not interfere in the further process. After training for 160k interactions (+e 43.5 hours) the agent successfully solves all boards in a test set of 25 board configurations (5 per solution depth in f1.... 5]). We refer to the supplementary material for details and video.
Assumptions and Limitations
Our approach assumes that the state abstraction is known, the symbolic observation is provided by the environment, and that the continuous state is fully observable. Learning the state abstraction too is an interesting direction for future research. The breadth-first search planner we use for planning on the symbolic level exhibits scaling issues for large solution depths; e.g., for LightsOut it exceeds a 5 5-minute threshold for solution depths (number of initial board perturbations) 2 9. In future work, more efficient heuristic or probabilistic planners could be explored. Currently, our BFS planner produces plans which optimal with respect to the number of skills executed. Means for predicting and taking the skill execution cost into account for planning could be pursued in future work. In the more complex environments (Reacher, Jaco) observe agent to not learn all possible skills reliably, in particular for skills for which no transitions exist in the replay buffer. In future work one could integrate additional exploration objectives which incentivize visit unseen regions of the state space. Also, the approach is currently limited to settings such as in board games, where all symbolic state transitions should be mapped skills. It is an open research question how our skill exploration objective could be combined with demonstrations or task-specific objectives to guide the search for symbolic actions and limit the search space in more complex environments.
Conclusion
We present agent which, in an unsupervised way, learns diverse skills in complex physically embedded board game environments which relate to moves the particular games. We assume a state abstraction from continuous states to symbolic states known and observable to the agent as prior information, and that skills lead to changes in the symbolic state. The jointly learned forward model captures the temporally extended effects of skill execution. We leverage this forward model to plan over a sequence of skills (moves) to solve a particular task, i.e., bring the game board to a desired state. We demonstrate that with this formulation games with high success rate, that our approach compares favorably with other flat and hierarchical RL algorithms, and also transfers to a real robot. Our approach provides an unsupervised learning alternative to prescribing the action abstraction and pretraining each skill individually before learning forward model from skill executions. In future research, our approach could be combined with state abstraction learning to leverage its full potential.
solve complex physically embedded
8

---

Acknowledgments
This work was supported by Max Planck Society and Cyber Valley. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Jan Achterhold.
References
[1] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller. DeepMind control suite. CoRR, abs/180 1011880.00000 $001006900 2018. URL http://arxiv. org/ abs/a/////// 1801 00690.
[2] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine. Ç. Gülçehre, Z. Wang. T. Pfaff, Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith. T. Schaul, T. P. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):3003354 2019. doi: 1011001455886-0099 1724-7. org/o0..//../. wtt.://..rr......1//111111. 00..00///11118019--977779
URL https///:///d http::////ooooo.....o1/1
[3] D. Silver, A. Huang, C. Maddison, A. Guez, I.. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. Lillicrap. M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and search. Nature, 529(7587): 484-489, Jan. 2016. doi: 10t:10000//atuurr 16961.
[4] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano- Pérez. Integrated task and motion planning. Annual Review of Control, Robotics, and Autonomous Systems, 4(1):265-293, 2021.
URL https://doi. org.oo..//... /0/ 1146/
amurev-control--911000 084139.
[5] M. Mirza, A. Jacgle, J. Hunt, A. Guez, S. Tunyasuvunakool, A. Muldal, T. Weber, P. Karkus, Racanière, L. Buesing, T. Lillicrap, and N. Heess. Physically embedded planning problems: New challenges for reinforcement learning. CoRR, abs/2009.05524, 2020. URL https: hwt.:/gggg///////eeeee abs/ 05524.
// //arxiv. org.v...rr/aaao
[6] G. Konidaris. On the necessity of abstraction, Current Opinion in Behavioral Sciences, 29:1-7,
2019. doi.https://doi.oogg 00gg/..///////
wttp:/ooo///...0/.oooooooooo 1016/10 watobeehaa202..1111 2018.11.008 Artificial Intelligence.
[7] B. Eysenbach, A. Gupta,
Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function. In 7th International Conference on Learning Representations (ICLR),
2019. URL https: / hptp//vieeeeeeeeeeeeeeeeeee net/ fed//foruum?d==5)633]j
[8] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. CoRR, 102.19997. 2018. URL
wttp:/i..rgg/////// /1007. 10299.
[9] D. Warde-Farley, T. V. de Wiele, T. D. Kulkarni, C. Ionescu, S. Hansen, and V. Mnih. Unsuper- vised control through non non-parametric discriminative rewards. In 7th International Conference on Learning Representations (ICLR), 2019. URL https:// httpa:///pppeevveeeee http///nrrreee..e/fffrrrmrr/ net/formm?id r1eVMnA9K7.
[10] K. Gregor, J. Rezende, and D. Wierstra. Variational intrinsic control. International Conference on Learning Representations (ICLR), Workshop Track Proceedings, 2017. URL http::///ppenreviee..eeeee/orers hitteeeettttetmoodddttteeeeeee Fo4Yg.
[11] K. Baumli, D. Warde-Farley, S. Hansen, and V. Mnih. Relative variational intrinsic control. In AAAI Conference Artificial Intelligence (AAAI), pages 6732-6740, 2021. URL https: / //ojs. aaai. wwwaa..or//iii org/index http:/innx.pppppppppp. phpp AAAI/article///eeaaaaaaaaaaaaaaa /vie///www///2///
[12] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. In 8th International Conference on Learning Representations (ICLR), 2020. URL https: /ttp::///ppeeeeeeeee wwtp:/oonenndntttniann.

---

[13] T. G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function de- composition. J. Artif. Intell. Res., 13:227-303, 2000, doi: doi:10.1613:0 101.0...////ai..33.. URL https: //doi.org/10 11113/jaiz 639.
[14] Z. Li, A. Narayan, and T. Leong. An efficient approach to mrodel-based hierarchical reinforce- ment learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pages 3583-3589, 2017. URL htt:: //aaai org/ ocs/ /et///inxx....///// php/AAAI/ /AAI/ AAAI17/
paper/viex/1477..
[15] M. A. Riedmiller, R. Hafner, T. Lampe, M. Neunert, Degrave, T. V. de Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing solving sparse reward tasks from scratch. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages 4341-4350. PMLR, 2018. html.
URL http: / htt: http///prceeen...../ wwwpeediinnsm........mmm warres//800riedmiiin
[16] C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In 5th International Conference on Learning Representations (ICLR), 2017. URL https://openreeiess .wt. http/fooummiddee Borumrid-BoooBBoxx..
[17] Dayan and G. E. Hinton.
Feudal reinforcement learning. In S. J. Hanson, D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems (NeurIPS), pages 271-278, 1992. URL http: htap:://ppprs...../ nips.
[18] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforce- ment learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 3307-3317, 2018. URL https: abb.abbbeeeceebbbbbbeeboeoottttd hhmml.
[19] A. Levy, G. D. Konidaris, R. P. Jr., and K. Saenko. Learning multi-level hierarchies with hindsight. In 7th International Conference on Learning Representations (ICLR), 2019. URL https://openrevier: nwt.feeeffeeoo orum? iidrryECoocc.T...
[20] R. S. Sutton, D. Precup, and S. P. Singh. Between MDPs and Semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2):811211, 1999. 1oi:
[21] P. Bacon, J. Harb, and D. Precup. The option-critic architecture. In S. Singh and S. Markovitch, editors, Proceedings of the Thirty-First AAAI Conference Artificial Intelligence (AAAI), pages 1726-1734, 2017. URL attp:://aase http:://aaaa.oo/ocs///////////// benpcs//iidx../eeeeeee ponudex..pp//ae php/ //////AAI//AAAAA /aaper/////p/////////////// view/14855
[22] A. Bagaria and G. Konidaris. Option discovery using deep skill chaining. In 8th International Conference on Learning Representations (ICLR), 2020. URL httaas://operrviiee nnt// forum?id-Bigqipe
[23] A. C. Li, C. Florensa, I. Clavera, and P. Abbeel. Sub-policy adaptation for hierarchical reinforcement learning. In 8th International Conference on Learning Representations (ICLR), 2020. URL https: //tp///eep///eeee/////e//e wtt.://ooouu...ddaaaaaaaaauuuu
[24] J. Zhang, H. Yu, and W. Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations (ICLR), 2021.
[25] M. R. K. Ryan. Using abstract models of behaviours to automatically generate reinforcement learning hierarchies. In International Conference Machine Learning (ICML), pages 522-529, 2002.
[26) D. Lyu, F. Yang, B. Liu, and S. Gustafson. SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In The Thirty-Thine AAAI Conference on Artificial Intelligence (AAAI), pages 2970-2977, 2019.
[27] L. Illanes, X. Yan, R. T. Icarte, and S. A. Mcllraith. Symbolic plans as high-level instructions for reinforcement learning. In C. Beck, O. Buffet, J. Hoffmann, E. Karpas, and S. Sohrabi, editors, Proceedings of the Thirtieth International Conference on Automated Planning and Scheduling (ICAPS), pages 540-550. AAAI Press, 2020.
10

---

[28] H. Kokel, A. Manoharan, S. Natarajan, B. Ravindran, and P. Tadepalli. RePReL: Integrating relational planning and reinforcement learning for effective abstraction. In Proceedings of the T'hirty-Frssl International Conference on Automated Planning and Scheduling (ICAPS), pages 533-541, 2021.
[29] L. Guan, S. Sreedharan, and S. Kambhampati. Leveraging approximate symbolic models for reinforcement learning via skill diversity. In International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 1949-7967. PMLR, 2022. URL https: httpss////oooeeeepeeeeeeee .mr..............m.........m.... w.l...rees///eeo.......... /tees/vv62// guwn//c.ann..cc. html.
[30] G. Konidaris, L. P. Kaclbling, and T. Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. J. Artif. Int. Res., 61(1):215-289, Jan. 2018.
[31] S. James, B. Rosman, and G. Konidaris. Learning portable representations for high-level planning. In Proceedings of the 37th International Conference Machine Learning, volume 119 of Pruceedings of Machine Learning Research, pages 4682-4691. PMLR, 13-18 Jul 2020. URL https: //tp:::///rroeee htttps://proceedingg mlr. pteas// //1//11111111 1119/ james200.......mmmm html.
[32] R. Toro Icarte, E. Waldie, T. Klassen, R. Valenzano, M. Castro, and S. Mcllraith. Learning reward machines for partially observable reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. uttpstps: URthhtts::/prooceettt http////ooeeedneeeeeeeeee neurips.cc/ http://pppppp../..///////////aaa
[33] E. Ugur and J. Piater. Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative exploration to symbolic planning. IEEE International Conference on Robotics and Automation (ICRA), pages 2627-2633, 2015. doi: 1009//A.. 2015.7139553.
[34] R. Chitnis, T. Silver, J. B. Tenenbaum, T. Lozano-Perez, and L. P. Kaelbling, Learning neuro- symbolic relational transition models for bilevel planning. In /EEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2022.
[35] T. Silver, A. Athalye, J. B. Tenenbaum, T. Lozano-Perez, and L. P. Kaelbling. Learning neuro- symbolic skills for bilevel planning. In Conference on Robot Learning (CoRL), 2022. URL /ttps://openrevies http://otfiddd-IIIRRoo.....
[36] A. Ahmetoglu, M. Y. Seker, A. Sayin, S. Bugur, J. H. Piater, E. Öztop, and E. Ugur. Deepsym: Deep symbol generation and rule learning from unsupervised continuous robot interaction for planning. CORR, abs/2012.02532,, 2020. URL /ttts::///raaet wtt. 2233..
[37] M. Asai and A. Fukunaga. Classical planning in deep latent space: Bridging the subsymbolic- symbolic boundary. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intel- ligence (AAAI). pages 6094-6101, 2018. URL https:// aaai. org/ ocs/ TAL1//AAT11//apee///a
//s///ccexnnn//////eee//ee/eeeee /idd/iddxxx php/
[38] T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (JCML), volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018. URL http: http://prroceedings wmm.eeeeeinnnmm..........mm.mm.m wwtpppswww......eeeeeeeeemmereee press/v80/ haarnoja18b html.
[39] H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83 1955.
[40] Munkres. Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1):32-38 -3,, 1957.
[41] M. Andrychowiez. D. Crow, A. Ray, Schneider, R. Fong, P. Welinder, B. McGrew, Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neurul Information Processing Systems (NeurIPS), pages 5048-5055 2017. URL. aattp:/ppapet nips.cc/ cc/ paper/7090 paper/r090-bindsightt- hiiddiigtt-eppprreeee---e--e
11

---

[42] S. W. Yoon, A. Fern, and R. Givan. Fe-Replan: A baseline for probabilistic planning. In Proceedings of the Seventeenth International Conference on Automated Planning and Scheduling (ICAPS), page 352, 2007. htI.htpp: //ww///wwwwww/// aaai. otgg/L/baarryI/CS///000 / icaps07-045.. php.
[43] A. Campeau-Lecours H. Lamontagne, S. Latour, Fauteux, Maheu, F. Boucher, C. Deguire, and L. C. L'Ecuyer. Kinova modular robot arms for service robotics applications. Int. J. Robotics Appl. Technol.. 5(2):49-71, 2017. doi: 000:444000// URL https: / //doi. org/10 IIJAT 2017070104.
12


Reviews:
[{"review_id": "tapMEOkjxnGB", "reviewer": "Reviewer_w2Y9", "summary": "This work proposes a skill discovery method given a set of symbolic states (state abstractions). The method consists of two parts. In the first part, a low-level controller, a = pi(s_0, k), which can be regarded as an option, outputs continuous actions to move the robot to the terminal state, s_T, given the initial state s_0, and the option index k. Here, s_0 and s_T correspond to symbolic states z_0 and z_T, respectively. In the second part, there is a symbolic forward model, z_T = q(z_0, k), which outputs the next symbolic state z_T given the current symbol z_0 and the option index k. By maximizing the mutual information between z_T and k, the robot learns a set of options that are both diverse and predictable given the initial symbol z_0. The low-level controller is trained by an RL algorithm (SAC), and the symbolic forward model is trained in a supervised manner.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method is a novel contribution for discovering skills given state abstractions. Although the assumption of state abstractions is strong, these abstractions can be indeed learned via symbol learning methods also mentioned in the text [24, 25, 27, 28, 29]. The idea of concatenating an option index k to the state, and maximizing the mutual information between k and z_T is very interesting. The paper is nicely written, explaining each definition thoroughly. Experiments are well-designed and there is an ablation study for each part of the method.\n\nOne slight downside might be that the method is only tested in 2d environments; it would have been good to see the performance of the method in a scaled environment, say, a 3d version of the lights out, since the exploration of the environment might not be as easy as in 2d. Of course, this is more related to exploration, but it would still be interesting to see the limits of the method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe the contribution of the paper is very novel and significant. Experiments show the applicability of the method both in simulated and real-world environments. The paper is well-written, and the accompanying video also helps assessing experiments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Although the related work section is quite extensive and good read, the  authors missed a very relevant literature. A number of recent work (Silver et al., 2021&2022; Chitnis et al., 2021) also works on skill learning where skills are represented with both symbolic and low-level components and hierarchical learning is performed. These studies use demonstrations rather than Reinforcement Learning, but otherwise have very similar assumptions, approaches and goals. The authors need to include one or more of these studies in their paper.\n\nSilver, Tom, et al. \"Learning Neuro-Symbolic Skills for Bilevel Planning.\" arXiv preprint arXiv:2206.10680 (2022).\nChitnis, Rohan, et al. \"Learning neuro-symbolic relational transition models for bilevel planning.\" arXiv preprint arXiv:2105.14074 (2021).\n\n- line 132: please define k before its use.\n- line 152: d and are is mixed up.\n- According to line 184, a change in state might not be observed in an episode. Then, how sound is it to include such an episode for FM learning (line 192)?\n- BFS does not give optimal solution if the actions costs are not same. Because the skills are discovered by the robot, each skill might have different length or cost. Therefore, the solutions might not be optimal. Please comment.\n- line 260: In 2D simulated environment, is the robot really torque controlled?\n- Although the notation is consistent and sound in Section 3, the use of numbers/labels/brackets in the subsecripts is a bit confusing. Is it possible to improve?\n- In task performance evaluation in Section 4, do environment steps refer to the number of optimization steps or the number of data points (i.e., environment interactions)? This is not clear because 10m steps are needed for experiments in the simulation while it is only 5000 interactions in the real world.\n- It would be nice to show what kind of skills are discovered after training (i.e., a scatter-plot of executions from an initial point for option k), and the average number of low-level actions for each skill. Regarding the statement, \"The board coordinates are x, y ∈ [0, 1], the maximum displacement per timestep is ∆x, ∆y = 0.2.\", if each displacements after each action become very high, then it would be enough to output only 3 or 4 low-level actions.\n- Moreover, for the reward improvement, I think having multiple high-level actions that moves to the same terminal state from the same initial state (i.e., not unique) might be a feature in other settings in which the robot learns two modes of operations and selects one at different conditions.\n"}}, {"review_id": "nW5gJ9hvfwm", "reviewer": "Reviewer_XcfM", "summary": "This work proposes to use a given symbolic state abstraction to learn and plan with temporally extended skills. The skills are learned by reinforcement learning, with an objective that combines diversity and predictability in terms of reachable symbolic states. Additional tricks are proposed to help avoid local minima. Experiments are in physically embodied grid-based games.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* The writing is overall very clear. I did not find many grammatical issues and the organization is good.\n* It is clear how the proposed approach improves on less structured and more model-free approaches like SAC, VIC, and HAC, and it is good to see those baselines in the experiments.\n* The ablations in Figure 3 and the appendix are very helpful for understanding the relative contributions of the proposed approach.\n* It’s always impressive to see results with reinforcement learning on a real robot, and the video does a good job visualizing that process.\n* The discussion of limitations is helpful and thorough.\n\n\n### Weaknesses\n\n* The environments used in the experiments are a bit contrived, and it is unclear to what extent the approach proposed in this paper would work beyond these environments.\n   * It is unclear whether the robot is really adding anything in these environments. In other words, one could argue that the robot is added just to make the demos more flashy, when in reality, all of the technical difficulty resides in the original board games.\n   * If these environments were actually ones that we wish to solve in a real application, there would be much easier ways to solve them, and those ways would probably not involve machine learning. For example, use a symbolic planner to solve the game boards, and use inverse kinematics and a hardcoded press-down motion for the temporally extended actions. (This would be a simple version of task and motion planning, but where the task and motion planning aspects are completely decoupled.)\n   * Given that the symbolic state space is highly structured, and it is known what the ideal abstract actions would be (pressing each grid cell), a much more direct RL approach would (1) create one skill for each grid cell; and (2) directly associate skills to cells, and give rewards accordingly.\n* The fact that the number of skills needs to be manually specified is a big limitation, and in experiments, the direct correspondence between the number of skills and the symbolic action space is a big injection of knowledge that is not emphasized.\n* There are several related works in the AI planning community (see Issues below) that call into question the novelty of this work, but this paper is the first that I have seen with (real) robotics results.\n* If the objective of this work is to learn temporally extended skills, the skills learned in the experiments are all essentially the same at the policy level: move and press. It would be more interesting to see a variety of skills learned.\n* I’m not completely convinced that the “second-best normalization” idea makes sense. Intuitively, it seems redundant with the diversity objective, but I can understand how optimizing that objective directly could lead to local minima. Moreover, the empirical results do not show a significant improvement vs. the “no sec-best norm” ablation.\n* It is not clear to what extent generalization is evaluated in the experiments. L250 states that board configurations are disjoint in training and testing, which is a good start, but I would be curious to know: in the course of a testing episode, how many of the symbolic states encountered are novel with respect to the training distribution? In addition to measuring this directly, it would be informative to see an ablation where the learned forward model is replaced with a tabular model.\n* As the paper mentions, planning with breadth first search is very limiting. Given the symbolic states and the existence of powerful AI planners, it would be more impressive if the overall approach were able to use those planners. For example, the works by Asai et al. and Konidaris et al. use PDDL planners and can scale to much longer horizons than the ones in this work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "*Post-rebuttal:* Thanks to the authors for addressing my questions and concerns. I am raising my recommendation to Weak Accept.\n\n*Pre-rebuttal:*\nThe work is overall performed with a high level of technical sophistication, and the paper is well-written, but I am concerned about novelty and about the limitations of the experimental environments.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* A “more skills” variation of the main method is briefly mentioned in S.5, with results in Figure S.4, but the variation is never explained (e.g., how many more skills are there?) I was going to ask to see results for a variation like this, so I am interested to know the details.\n* It’s not clear why that “more skills” variation and the VIC results are in the appendix, rather than in the main paper. (Maybe they were generated in between the paper deadline and the supplementary deadline?)\n* Other references:\n   * “Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity” Guan et al. 2022\n   * “Reprel: Integrating relational planning and reinforcement learning for effective abstraction” Kokel et al. 2021\n   * “Symbolic Plans as High-Level Instructions for Reinforcement Learning” Illanes et al. 2020\n   * “SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning” Lyu et al. 2019\n   * “Using Abstract Models of Behaviours to Automatically Generate Reinforcement Learning Hierarchies” Ryan 2002\n   * “Integrated Task and Motion Planning” Garrett et al. 2021\n\n### Minor\n* $\\mathcal{S}_0$ is not defined on L139, and it seems like just $\\mathcal{S}$ is used instead starting on L143.\n* It’s not immediately obvious why $I$ and $\\beta$ are defined over tuples of states rather than individual states. In the original options paper, they’re just defined over individual states. My understanding is that they are defined in this way because the termination function needs to “remember” the original abstract state so that it can terminate when the abstract state changes. It would be helpful to spell this out. \n* L141 grammar: “for the option policy holds”\n* L218: remove period before “which”\n* L246 grammar: “As step limit for skill execution”\n* There are a number of places where the open quotation marks are backwards\n\n"}}, {"review_id": "UP_4tJE6lpIE", "reviewer": "Reviewer_rgwq", "summary": "This paper presents a method that jointly learns a predictive model in the symbolic level and a skill-conditioned policy.  After training, the learned predictive model allows the authors to do long-horizon planning in the high level, and execute those plans using the learned policy. The method is both evaluated in simulation and on a real robot.  ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nAutomatically learning a discrete set of skills as well as a transition function in high level is extremely interesting. The paper shows interesting results that in many cases, the method can recover all the discrete skills needed to solve the task. \n\n## Weaknesses and questions\n-  about the assumption\n\nThe authors have made it clear that there is one assumption of this work, which is that the symbol definition and the mapping from state to symbol are given. While I consider this assumption reasonable, it dramatically simplified the problem, thus posing an issue of applying this method to other tasks. \n\n-  about the efficiency\n\nFrom the experiments, we can see as the task becomes more complex, recovering all skills becomes harder. It seems to me that the method in this paper is still not efficient enough to be scaled up to more interesting robotic tasks (with a magnitude of more symbolic states and skills). I'm curious about the authors' view on learning efficiency. What bottlenecks, which prevent the system to discover skills more efficiently, does this algorithm have? One reason, as mentioned in the relabelling section, is that the coupled nature of the forward model and the policy make the learning in early stage hard. Another reason I can think of is the exploration issue. Does the method have trouble in encouraging exploration? \n\n-  about the learned skills\n\nThe training objective encourages the skill to reach diverse successor states, and be consistent with the forward model. Are there any constraints on the skills being consistent over different starting states?  For example, in two different starting states where different lights are on, if one skill is learned in the first starting state to be pressing the up-left button, will this skill also be pressing the same button if the initial lights and robot state are different? Figure S1 suggests that the skills are consistent. But I wonder how this is achieved. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper bridges the gap between the neural network based reinforcement learning and symbolic planning. This could potentially have a large impact on the direction of learning more robust and interpretable models. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "No issues. ", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "1ycwmYeSQjG", "reviewer": "Reviewer_a18m", "summary": "This paper proposes ‘SEADS’, a novel hierarchical approach to learning diverse skills as well as a forward model over these skills that will be useful for planning to solve long-horizon tasks. The proposed method assumes that a discrete, symbolic abstraction of the low-level continuous environment is provided as input. Given this, it utilizes an unsupervised exploration reward to collect data via environment interaction, and then learns skills and the corresponding forward model based on these data. Experiments show that the proposed approach is capable of solving two different physically-embedded games (LightsOut and TileSwap) under different settings more efficiently than other baseline hierarchical RL approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed method seems novel and is intuitive. I thought the overall idea to learn skills and the forward model jointly was interesting. Moreover, the derivation of the exploration reward approximation, as well as the relabelling were both clever and creative ideas.\n- The paper is generally well-written, clear and easy-to-follow.\n\n**Weaknesses**\n\n- The proposed approach applies only to a rather limited setting. To me, it seems that a key limitation of this work is the set of assumptions that (1) the abstract symbolic state space is small, and (2) getting skills to reach the different possible symbolic state changes is desirable. These are not practical in most robotic domains, where even the symbolic state space might be extremely large and only a small subset of symbolic state changes might be important for useful skills. Thus, it seems difficult if not impossible to apply this work to continuous, long-horizon problems beyond the relatively simple setting (physically-embedded games with small and simple symbolic state spaces) that the authors have chosen.\n- The task performance evaluation is limited to tasks that only require up to 5 moves. This seems like an unjustified choice that simplifies the task and significantly reduces the horizon. Given that the approach was trained to generalize to settings regardless of horizon, why was the horizon limited?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think this is an interesting, well-written paper that provides an interesting new direction for hierarchical RL. However, the limited applicability and setting make it seem rather narrow in scope compared to existing hierarchical RL approaches. I think it is valuable and interesting to the community, but could be made significantly better if the authors clearly address and point out the applicability of their method in the writing of the paper or via additional environments/domains in the experimental evaluation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- As mentioned in the “weaknesses” section, I find the approach’s applicability to be rather limited to physical games such as LightsOut and TileSwap. Do the authors agree with this assessment? If so, I think it is important to mention this in the Limitations section. Additionally, I think it is important to highlight the assumptions regarding the size of the state space and desirability of reaching all of it in Section 3.\n- As mentioned in the “weaknesses” section, I don’t understand why the evaluation was restricted to tasks that require only up to 5 moves. I think it is important to justify this choice in the text.\n- When reading through the paper, it took me a while to understand that the number *K* is being provided as input to the approach. I think it’s important to mention this explicitly at the start of Section 3."}}]

Author rebuttal:

Decision: Accept (oral)