Learning Options via Compression
Authors: Yiding Jiang
Evan Zheran Liu
Benjamin Eysenbach
J Zico Kolter
Chelsea Finn
Venue: NeurIPS 2022 Accept (2022)
Keywords: hierarchical reinforcement learning
skill learning

Abstract: Identifying statistical regularities in solutions to some tasks in multi-task reinforcement learning can accelerate the learning of new tasks.
Skill learning offers one way of identifying these regularities by decomposing pre-collected experiences into a sequence of skills.
A popular approach to skill learning is maximizing the likelihood of the pre-collected experience with latent variable models,
where the latent variables represent the skills. However, there are often many solutions that maximize the likelihood equally well, including degenerate solutions. To address this underspecification, we propose a new objective that combines the maximum likelihood objective with a penalty on the description length of the skills. This penalty incentivizes the skills to maximally extract common structures from the experiences. Empirically, our objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood. Further, while most prior works in the offline multi-task setting focus on tasks with low-dimensional observations, our objective can scale to challenging tasks with high-dimensional image observations.

Learning Options via Compression
Yiding Jiang"
Evan Zheran Liu" Stanford University evanliuucs stanf stanford.co edu
Carnegie Mellon University yidingji@ee i@cs. cmu. edu
Benjamin Eysenbach Carnegie Mellon University beysenba@cs cmu. edu
J. Zico Kolter Carnegie Mellon University zkolterêcs cmu. edu
Chelsea Finn Stanford University
cbfinnecs stanford.edu edu
Abstract
Identifying statistical regularities in solutions to some tasks in multi-task reinforce- ment learning can accelerate learning of new tasks. Skill learning offers one way of identifying these regularities by decomposing pre-collected experiences into a sequence skills. popular approach to skill learning is maximizing the likelihood of the pre-collected experience with latent variable models, where the latent variables represent the skills. However, there are often many solutions that maximize the likelihood equally well, including degenerate solutions. To address this underspecification, we propose new objective that combines the maximum likelihood objective with penalty the description length of the skills. This penalty incentivizes the skills to maximally extract common structures from the experiences. Empirically, objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood. Further, while most prior works in the offline multi-task setting focus tasks with low-dimensional observations, our objective can scale to challenging tasks with high-dimensional image observations.
Introduction
While learning tasks from scratch with reinforcement learning (RL) often sample inefficient [30, 8]. leveraging datasets of pre-collected experience from various tasks can accelerate the learning of new
tasks. This experience can used in numerous ways, including learn dynamics model [18, 38], to learn compact representations of observations [87], to learn behavioral priors [77], or to extract meaningful skills or options [79, 42, 2, 59] for hierarchical reinforcement learning (HRL) [6]. Our work studies this last approach. The central idea is to extract commonly occurring behaviors from the pre-collected experience as skills, which can then be used in place of primitive low-level actions to accelerate learning new tasks via planning [70, 51] or RL [55, 47, 52]. For example, in navigation domain, learned skills may correspond to navigating to different rooms or objects.
Prior methods learn skills by maximizing the likelihood of the pre-ccllected experience [43, 42, 2, 90]. However, this maximum likelihood objective (or the lower bounds on it) is underspecified: it often admits many solutions, only some of which would help learning new tasks (see Figure 1). For example, one degenerate solution is to learn a single skill for each entire trajectory; another degenerate solution is to learn skills that operate for single timestep, equivalent to the original action space. Both of these decompositions can perfectly reconstruct the pre-collected experiences (and, hence, maximize likelihood), but they of limited use for learning to solve new tasks. Overall, the maximum likelihood objective cannot distinguish between such dccompositions and potentially more useful
Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Skills that arbitrarily segment the trajectories
Skills that navigate to objects
Figure 1: Skill learning via maximizing likelihood is underspecified: the skills (represented by the different colored segments) on both the left and the right maximize the likelihood (i.., encode the data), but the skills on the right more likely to be useful for new task. Our approach factors out common temporal structures, yiclding the skills on the right. The visualized tasks are from Kipf al. [42].
decompositions, and we find that this underspecification problem can empirically occur even on simple tasks in experiments.
To address this underspecification problem, we propose to introduce an additional objective which biases skill learning to acquire skills that can accelerate the learning of new tasks. To construct such an objective, we make two key observations:
Skills must extract reusable lemporally-extended structure from the pre-collected experiences to accelerate learning of new tasks.
Compressing data also requires identifying and extracting common structure.
Hence, hypothesize that compression is a good objective for skill learning. We formalize this notion of compression with the principle of minimum description length (MDL) [69]. Concretely, combine the maximum likelihood objective (used in prior works) with new term to minimize the number of bits required to represent the pre-collected experience with skills, which incentivizes the skills to find common structure. Additionally, while prior compression-based methods typically involve discrete optimization and hence are not differentiable, we also provide a method to minimize our compression objective via gradient descent (Section 4), which enables optimizing our objective with neural networks.
Overall, our main contribution is an objective for extracting skills from offline experience, which addresses the underspecification problem with the maximum likelihood objective. We call the resulting approach LOVE (Learning Options Via comprEssion). Using multi-task benchmarks from prior work [42], we find that LOVE can learn skills that enable faster RL and are more semantically meaningful compared skills learned with prior methods. We also find that LOVE can scale to tasks with high-dimensional image observations, whereas most prior works focus on tasks with low-dimensional observations.
2 Related Works
We study the problem of using offline experience from one of tasks to quickly learn new tasks. While we use the experience to learn skills, prior works also consider other approaches of leveraging the offline experience, including to learn a dynamics model [38], to learn a compact representation of observations [87], and to learn behavioral priors for exploring new tasks [77].
We build rich literature extracting skills [79] from offline experience [62, 43, 74, 42, 66, 92, 91, 46, 2, 72, 59, 90, 53, 93, 68, 80, 88). These works predominantly learn skills using a latent variable model, where the latent variables partition the experience into skills, and the overall model is learned by maximizing (a lower bound on) the likelihood of the experiences. This approach structurally similar
long line of work in hierarchical Bayesian modeling [24, 20, 49, 37, 13, 48, 39, 28, 15J. We also follow this approach and build off of VTA [39] for the latent variable model, but differ by introducing new compression term to address the underspecification problem with maximizing likelihood. Several prior approaches [90, 91, 23] also use compression, but yield open-loop skills, whereas we aim learn closed-loop skills that can react to the state. Additionally, Zhang et al. [90] show that maximizing likelihood with variational inference can itself be seen as form of compression, but this work compresses the latent variables at each time step, which does not in general yield optimal compression our objective does.

---

Beyond learning skills from offline experience, prior works also consider learning skills with addi-
tional supervision [3, 63, 75, 35. 78], from online interaction without reward labels [25, 19, 17, 73, 5], and from online interaction with reward labels [81, 44, 4, 84, 29, 58). Additionally, large body of work on meta-reinforcement learning [16, 85, 67, 94, 50] also leverages prior experience to quickly
learn new tasks, though not necessarily through skill learning.
3 Preliminaries
Problem setting. We consider the problem of using offline dataset of experience quickly solve new RI. tasks, where each task is identified by reward function. The offline dataset is a set of trajectories D (n) where each trajectory is sequence of states E X' and actions = [[x1, a1); (I2, a ... Each trajectory is collected by some unknown policy interacting with Markov decision process (MDP) with dynamics P(æ++1 Ør, Qe). Following prior work [18, I, 14], we do not assume access to the rewards (i.e., the task) of the offline trajectories.
A:
Using this dataset, our aim is to quickly solve the same MDP as the data collection policy in the offline dataset, except with a new reward function R(x,a) a). The objective is to learn policy that maximizes the expected returns in few numbers of environment interactions as possible.
task. The new task involves interacting with
Variational Temporal Abstraction. Our method builds upon the graphical model from variational temporal abstraction (VTA) [39], which is method for decomposing non-control sequential data (i.e., data without actions) into subsequences. We overview VTA below and extend it to handle actions it Section 4.
VTA assumes that each observation D, is associated with an abstract representation 8; and a subsequence descriptor 2t. An additional, binary random variable m, indicates whether a new subsequence begins at the current observation. VTA uses the following generative model:
x
444
1. Determine whether to begin new subsequence:
m, ~ p(m; $;-1). A new subsequence always begins on the first time step. i.e., mi
21
2. Sample the descriptor: 21 p(zt <ct, m2). If
The VTA graphical model (adapted from Kim et al. [39]). The model decomposes the data Fi.r into subse- quences demarcated by when the boundary variables mn, = 1 (highlighted in yellow). Each subsequence is assigned descriptor z, where all 2, within subsequence are the same (outlined in dashed lines).
Figure 2:
m: = 0, the previous descriptor is copied, i.c..
ze Zc-1-
3. Sample the next state abstract representation:
$t ~ >(s; $<1, 2t, m.).
4. Sample the observation: ot ~ p(æ; | st).
We visualize this sampling procedure in Figure 2. Formally, we can write this generative model as:
T
p(Z1: 2I.T, 81.7, m.:r = II
p(#, st)p(m, sc-1)p(ss) | 8.t, 2t, m.)p(ze Z m.).
t=1
VTA maximizes the likelihood of the observed data under this latent variable model using the standard evidence lower bound (ELBO):
logr())))) N Σ
J MI.T. #1:T
(z 81. Warr mirr) log (Z &1:7, $1.7, Mirr) qe (Zi.T, S1.7, mir (i:r)
This lower bound holds for any choice of qo. VTA chooses factor this distribution as:
(ir) = II (m, xi.e) go (2 Æt-1, m., xirr) go(s: | #;, m. x1.))
Mir, Mir
As discussed above and as we observe in Section 6, the maximum likelihood objective under- specified and may yield degenerate or unhelpful solutions, which we address in the next section. Specifically, we use the graphical model from VTA, but introduce new objective that compresses the latent variables z, while also maximizing the ELBO.
3

---

4 LOVE: Learning Options via Compression
In this section, we describe our method for learning skills from pre-collected experience. We first extend the VTA graphical model to handle experience labeled with actions, and then introduce compression objective that encourages extracting common structure from the experience.
4.1 A Graphical Model for Interaction Data
We now extend the VTA graphical model [39] to handle sequential data labeled with actions, where de- scriptors 2 now represent skills. From high level, the model partitions trajectory into subsequences with the boundary variables m and labels each subsequence as a skill z.
We wish to learn state-conditional policy rather than the joint distribution of the whole trajectory. To do this, we write new generative model for the actions a1.7 conditional on the state æ1:2t
p(Z1,T, 1.1TT88111 BUT, mI.T, anT (I:T II p(a, 8t)p(m, sc-1)p(se wt, zt)p(Zz æt, Zc- 1, m.-1).
This differs from the original VTA generative model in two ways: (I) we introduce p(a; 8.) term that indicates how actions are sampled; and (2) the distributions over 2, and 8, do not depend on all previous i1 and $1z to encode the Markov property. We then augment the variational distribution such that the posterior over skills z also depends on actions:
qo (2 $1.7, mar 011T air) = II
go (m: x..e) [0( ž1 m., $1.77 a1.)) go (3t 2 2c:)
termination poliey
skill postesior
abstraction posterior
Overall, this yields model with 3 learned components:
A state abstruction posterior go ft | Σt, æt.) and decoder po(a; | 8t), which together form a distribution over actions conditioned on the skill and current state 24. We refer to these jointly as the skill policy (a(a, 21, ..)) (ee.. (po(a; +:)]-
termination policy qo(m,, (.:)), which determines whether the previous skill terminates.
A skill posterior go(z; 1, m,, #1.7, Q1.T), which outputs the current skill conditioned
on all states æ1.7 and actions a1.T. This depends on m, and 2,-1, since the boundary variables determine whether the previous skill z1-1 terminates: when m. O, the previous skill does not terminate and 2; = 2(-1-
Once this model is learned, the skill policy and termination policy represent the skills, without need for the skill posterior: Given skill variable z, the skill policy encodes what actions the skill takes, and the termination policy determines when the skill stops. In the next section, we describe our objective for learning this model. Then, Section 5, we describe how we use the skill policy and termination policy to learn new tasks.
4.2 Discovering Structure via Compression
As previously discussed, the maximum likelihood objective is underspecified for skill learning, because many skills can maximize likelihood, independent of whether they are useful for learning new tasks. In this section, we address this with a new objective that attempts to measure how useful skills will be for learning new tasks in terms of compression. Our objective is based the intuition that effectively compressing a sequence of data requires factoring out common structure, and factoring out common structure is critical for learning useful skills.
We measure the complexity municate the sequence of skills that encode the pre-collected experience. The latent variable model introduced in the previous section encodes each trajectory as a sequence of skills 1:77 boundaries m1.7. However, using the skill and termination policies, it is possible to recover each trajectory from only the skill variables at the boundary points: i.e., the time steps {1 E [T) m. = 1). For prior on skills Pz. optimal code requires log (z,) bits to send skill 2, [12, Chapter 5.2). Hence,
skill decomposition as the amount of information required to com-

---

the expected code length of communicating a trajectory 71:T = {(11,a))... ai)...., (ær, ar)) is:
IN)) = -E T.
Σ Po
where the expectation is under trajectories T::T from the pre-collected experience and sampling 1.T from the skill posterior and mi.T from the termination policy. The choice of prior that minimizes the average code length is one that equals the empirical distribution of skills under the pre-collected experience [12, Chapter 5.3]:
T Σ
p2 - arg min (NFOCOST(@::P). where (z) = E Ti 71g
8(zz = 2) z)m²
Ps
TAL #1:T
and ng BriT -== m In Appendix A.I, show that this marginal pz is proper
density for both continuous and discrete z. Substituting in this optimal choice of prior, we can show that the code length can be expressed the marginal entropy Hyp [z] times the number of skills per trajectory n, (see Appendix A.2 for proof):
T
CcI. A min INFoCosT(O:P(2.)) EE TI Σ p(ac)
log log (Z n.,
n. Hoppppp)) [z]
(1)
Intuitively, this is equal to the average code length skill multiplied by the the average number of skills per trajectory. Note that compression is only beneficial if the model also achieves high likelihood of the data. We capture this by solving the following constrained optimization problem:
min (0) s.t. CELBO (0) S C,
(2)
where (f) negated evidence lower bound on the log-likelihood (detailed in Appendix B).
Remarks. We discuss connection between this objective and variational inference in Ap- pendix A.3. Additionally, while this work focuses on the RL setting, our objective generally applies sequential modeling problems. We believe that it could be useful for many applications beyond option learning.
4.3 Connections to Minimal Description Length
Our approach closely relates to the minimum description length (MDL) principle [69]. This principle equates learning with finding regularity in data, which can be used to compress the data. Informally, the best model is the one that encodes the data with the lowest description length. Given a model O that encodes the data into some message, one way to formalize the description length of the data L(D) is with crude two-part code [26]. This decomposes the description length of the data the length of the message plus the length of the model:
L(D)= L(D
L(
(3)
message longth
model longth
In our case, the message is the latent variables z, the boundary points {t [7] m. 1}. which can be decoded into the data with the skill and termination policies (representing the model). Hence, approach can be seen our objective in Equation is equivalent to minimizing the message length L(D @). While we do not directly attempt to minimize the model length term L(O),, many works [60, 82) indicate that deep learning has implicil regularization, which biases optimization toward low complexity solutions without an explicit regularizer. In general, computing the true description length or appropriate notion of complexity for neural networks is a tall order [61, 89, of methods be explored here and we therefore leave the extension of our approach to directly minimizing the model length term for future work.
instance of minimizing the description length L(D). Optimizing
89,36), 36]; however, there is a rich space
4.4 A Practical Implementation
Model. We instantiate our model by defining discrete skills E [K], state abstractions 8 E Rd and binary boundary variables m E [0, 1]. We parameterize all components of our model as neural networks. See Appendix D for architecture details.
5

---

Optimization. We apply Gumbel-softtax [34, 54] to optimize over the discrete random variables and m. We rewrite the compression objective Ca. as product of n, and Hp; [x,] in Equation 1 to improve the stability of optimization. This rewriting allows computing gradient in terms of distribution over the skill variables ph. rather than samples 2t, which yields a more accurate finite sample gradient. Empirically, this leads to stabler optimization and convergence to better solutions.
We approximate the optimal skill prior pz using our skill posterior as:
(5(2)
T
m. (z = z Z mc: mI:T (IT) my
T- m
E
miir
where E denotes the empirical expectation over minibatches of a1:T from D and sampling my.r from the termination policy. We solve the constrained optimization problem (Equation 2) with dual gradient descent on the Lagrangian. In Appendix G, we summarize the overall training procedure in Algorithm and report details about the Lagrangian.
Enforcing minimum skill length. Though degenerate solutions such as skills that only take single action score poorly in our compression objective, empirically, such solutions create local optima that are difficult to escape. To avoid this, mask the boundary variables during training to ensure that each skill & operates for at least Tiin = 3 time steps. We find that when the skills are at least minimally temporally extended, optimization of the compression objective appears to be stabler and achieves better values. We remove these masks at test time, when we learn task with our skills.
5 Using the Learned Skills for Hierarchical RL
We now describe how we quickly learn new tasks, given the skills learned from the pre-collected experience. Overall, we simply augment the agent's action space with the learned skills and learn new policy that can take either low-level actions or learned skills, similar to Kipf et al. [42]. However, our compression objective may result in unused skills, since this decreases the encoding cost of the used skills. Therefore, we first filter down to only the skills that are used to compress the pre-collected experience by selecting the skills where the marginal is over some threshold a:
2 {k E [K]
F > a}.
Then, new task with action space A. train an agent using the augmented action space A = A U Z. When the agent selects a skill x e Z, we follow the procedure in Algorithm in the Appendix. We take actions following the skill policy aa(a, x, xx) (lines 2-3), until the termination policy qo (m, (ia) outputs a termination m, = (line 5). At that point, the agent observes the next state 2++1 and selects the next action skill.
6 A Didactic Experiment: Frame Prediction
Before studying the RL setting, we first illustrate the effect of LOVE in the simpler setting of sequential data without actions. In this setting, use the original VTA model, which partitions sequence Fir into contiguous subsequences with the boundary variables m and labels each subsequence with descriptor 2, as described in Section 3. Here, VTA's objective is to maximize the likelihood of the sequence I1.7, and our compression objective applies to this setting completely unmodified. We compare LOVE and only maximizing likelihood, i.e., VTA [39], by measuring how well the learned subsequences correspond to underlying patterns in the data. To isolate the effect of the compression objective, do not enforce minimum skill length.
Dataset. We consider two simple datasets Simple Colors and Conditional Colors, which consist of sequences of 32 32 monochromatie frames, with repeating underlying patterns. Simple Colors consists of 3 patterns: 3 consecutive yellow frames, consecutive blue frames and 3 consecutive green frames. The dataset is generated by sampling these patterns with probability 0.4, 0.4, and 0.2 respectively and concatenating the results. Each pattern is sampled independent of history. The dataset is then divided into sequences of 6 patterns, equal to 6 x 18 frames.
In contrast to Simple Colors, Conditional Colors tests learning patterns that depend on history. consists of 4 patterns: the 3 patterns from Simple Colors with an additional pattern of 3 consecutive

---

LOVE
Table 1: Effect of LOVE on the Simple Colors and Condi-
fional Colors datasets (5 seeds). All approaches achieve similar values for the likelihood objective, but LOVE better recovers the correct boundaries.
2-1
Simple Colors
Conditional Colors
5-5
1-0
VTA
LOVE
VTA
LOVE
CaLmO
2868 :
2838 LM
2832 1:7
2827 1.9
Figure 3: Learned boundaries and descriptors Simple Colors. LOVE recovers the patterns, while VTA learns boundaries that break up pat- terns or span multiple patterns (outlined in red).
Precision Recall
0.87 0.11
0.99 + 0.00
0.81 0.22
0.99 + 0.01
0.79 0.82 ta
0.85
0.82
0.83
Fl
0.91 10:0
0.83 0.118
0.90 a 0.00
Code Length
7.58
6.34+o.m
9.17
6.83
purple frames. The dataset generated by uniformly sampling the patterns from Simple Colors. To make the current timestep pattern dependent on the previous timestep, the yellow frames are re-colored to purple, if the previous pattern was blue or yellow. As in Simple Colors, the dataset is divided into sequences of 6 patterns.
In both datasets, the optimal encoding strategy learn subsequence descriptors (i.e., skills), one for each pattern in Simple Colors. In Conditional Colors, the descriptor corresponding to yellow in Simple Colors either outputs yellow or purple, depending on the history. This encoding strategy achieves an expected code length of 6.32 nats in Simple Colors and 6.59 nats in Conditional Colors.
Results. We visualize the learned descriptors and boundaries in Figure 3. LOVE successfully segments the sequences into the patterns and assigns consistent descriptor to each pattern. In contrast, despite the simple structure of this problem, VTA learns subsequences that span over multiple patterns last for fewer timesteps than single pattern.
Quantitatively, we measure the (1) precision, recall, and F1 scores of the boundary prediction, (2) the ELBO of the maximum likelihood objective and (3) the average code length CaL in Table 1. While both VTA and LOVE achieve approximately the same negated ELBO (lower is better), LOVE recovers the correct boundaries with much higher precision, recall, and F1 scores. This illustrates that the underspecification problem can occur even in simple sequential data. Additionally, we find that LOvE achieves an encoding cost close to the optimal value. One interesting, though rare, failure mode is that LOVE sometimes even achieves a lower encoding cost than the optimal value by over-weighting the compression term and imperfectly reconstructing the data. In Appendix C, conduct ablation study on the weights of LCL in optimization.
Experiments
We aim to answer three questions: (1) Does LOVE learn se- mantically meaningful skills? (2) Do skills learned with LOVE accelerate learning for new tasks? (3) Can LOVE scale to high-dim pixel observations? To answer these questions, we learn skills from demonstrations and use these skills learn new tasks, first on 2D multi-task domain, and later on 3D variant.
Multi-task domain. We consider the multi-task 10 x 10 grid world introduced by Kipf et al. [42], a representative skill learning approach (Figure 4). This domain features challenging control problem that requires the agent to collect up to 5 objects, and we later add perception challenge to it with the 3D variant. We use our own custom implementation, since the code from Kipf et al. [42] is not publicly available. In each task, 6 objects are randomly sampled from a set of Novj = 10 different objects and placed at random positions in the grid. Additionally, impassable walls randomly added to the grid, and the agent is also placed at a random initial grid cell. Each task also includes an instruction list of Npick 3 or 5 objects that the agent must pick up in order.
Figure 4: Multi-task grid world instance from Kipf et al. [42]. The agent must pick up sequence of objects in speci- fied order.
Within each task, the agent's actions are to move up, down, left, right, or to pick up an item at the agent's 's grid cell. The state consists of two parts: (1) 10 x 10 x (Nob) + 2) grid observation,
7

---

indicating if the agent, a wall, or any of the Noby different object types is present at each of the 10 x 10 grid cells; (2) the instruction corresponding to the next object pick up, encoded as a one-hot integer. Following Kipf et al. [42], we consider two reward variants: In the sparse reward variant, the agent only receives +1 reward after picking up all Npick objects the correct order. In the dense reward variant, the agent receives +1 reward after picking up each specified object in the correct order. Our dense reward variant is slightly harder than the variant in Kipf et al. [42], which gives the agent +1 reward for picking up objects in any order. The agent receives 0 reward in all other time steps. The episode ends when the agent has picked up all the objects in the correct order, or after 50 timesteps.
Pre-collected experience. We follow the setting in Kipf et al. [42]. We set the pre-collected experience to be 2000 demonstrations generated via breadth- first search on randomly generated tasks with only Npick = 3 and test if the agent can generalize to Npicc = when learning a new task. These demonstrations are not labeled with rewards and also do not contain the instruction list observations.
Points of comparison. To study our compression objective, we compare with two representatives of learning skills via the maximum likelihood objective:
VTA [39], modified to handle interaction data as in Section 4.1.
Discovery of deep options (DDO) [21]. We implement DDO's maximum likelihood objective and graphical model on top of VTA's variational inference optimization procedure, instead of expectation graddinns used in the original paper.
For fairness, we also compare with variants of these that implement the minimum skill length constraint from Section 4.4. ComplI.R [42] is another approach that learns skill by maximizing likelihood and was introduced in the same paper as this grid world. Because the implementation unavailable, we do not compare with it. However, we note that CompILE requires additional supervision that LOVE, VTA, and DDO do not: namely, it requires knowing how many skills each demonstration is composed of. This supervision can be challenging obtain without already knowing what the skills should be.
Since latent variable models prone to local optima [45], it is common learn such models with multiple restarts [57]. We therefore run each method with 3 random initializations and pick the best model according to the compression objective La. for LOVE and according to the ELBO of the maximum likelihood objective LELBO for the others. Notably, this does not require any additional demonstrations or experience.
We begin by analyzing the skills learned from the pre-collected experience before analyzing the utility of these skills for learning downstream tasks in the next sections.
7.1 Analyzing the Learned Skills
We analyze the learned skills by comparing them to a natural decomposition that parti- tions the demonstration Npick sequences of moving to and picking up an object. Specifically, we measure the precision and recall of each method in outputting the cor- rect boundaries of these Npick sequences.
Table 2: Precision and recall of outputting the boundarics between picking up different objects. Only LOVE learns skills that move and pick up an object.
Precision
Recall
F1
DDO [21]
0.19
1
0.32
DDO + min. skill length
0.26
0.53
0.35
VTA [39]
0.19
0,99
0.32
We find that only LOVE recovers the correct boundaries with both high precision and recall (Table 2). Visually examining the skills learned with LOVE shows that each skill moves to an object and picks it up. We visualize LOVE'S ' skills in Appendix H. In contrast, maximizing likelihood with either DDO or VTA learns degenerate skills that output only a single action, leading to high recall, but low precision. Adding the minimum skill length constraint to these approaches prevents such degenerate solutions, but still does not lead to the correct boundaries. Skills learned by VTA with the minimum skill length constraint exhibit no apparent structure, but skills learned by DDO with the minimum skill length constraint appear to move toward objects, though they terminate at seemingly random places. These results further illustrate the underspecification problem. All approaches successfully learn to reconstruct the
VTA + min. skill length
0.27
0.53
0.36
LOVE (ours)
0.90
0.94
0.92
8

---

Nara (Dense)
Nack (Sparse)
Pick (Dense)
Nask (Sparse)
Average Rearms
0.5
0.5
0.5
L..
1.0-
1145
008-
0.0
903
2000
1002
2000
30:00
2003
2003
aice
2000
3000
1003
2000
3000
4200
1000
Timesteps (1e3)
Timestepo (1e3)
Timesteps (1e8)
Timesteps (1e3)
LovE
VIA
No skills
DUO
Optimal
VTA (min. skill length)
No skills (behavior cloning)
DDO (min. skill length)
Figure 5: Sample efficient learning. We plot returns vs. timesteps of environment interactions for 4 settings in the grid world with 1-stddev error bars (5 seeds). Only LOVE achieves high returns across all 4 settings.
demonstrations and achieve high likelihood, but only LOVE and DDO with the minimum skill length constraint learn skills that appear to be semantically meaningful.
7.2 Learning New Tasks
Next, we evaluate the utility of the skills for learning new tasks. To evaluate the skills, sample new task in each of 4 settings: with sparse or dense rewards and with Npick = 3 or Npick = 5. Then we learn the new tasks following the procedure in Section 5: we augment the action space with the skills and train an agent over the augmented action space. To understand the impact of skills, we also compare with low-level policy that learns these tasks using only the original action space, and the same low-level policy that incorporates the demonstrations by pre-training with behavioral cloning, We parametrize the policy for all approaches with ducling double deep Q-networks [56, 86, 83] with e-greedy exploration. We report the returns of evaluation episodes with = O, which are run every 10 episodes.
We report returns averaged over 5 seeds. See Appendix F.4 for full details.
Results. Overall, LOVE learns new tasks across all 4 settings comparably to or faster than both skill methods based on maximizing likelihood and incorporating demonstrations via behavior cloning (Figure 5). More specifically, when Npick = 3, DDO with the min. skill length constraint performs comparably to LOVE, despite not segmenting the demonstrations into sequences of moving to and picking up objects. We observe that while a single DDO skill does not move to and pick up an object, the skills consistently move toward objects, which likely helps with exploration. Imitating the demonstrations with behavior cloning also accelerates learning when Npick = 3 with dense rewards, though not as much as LOVE or DDO, and yields insignificant benefit in all other settings. Skill learning with VTA yields no benefit over no skill learning at all.
Recall that N pick in all of the demonstrations. LOVE learns new tasks in the generalization setting of Npick = 5 much faster than the other methods. With dense rewards, DDO (min. skill length) also eventually solves the task, but requires over 8x more timesteps. The other approaches learn to pick up some objects. but never achieve optimal returns of 5. With sparse rewards, LOVE is the only approach that achieves high returns, while all other approaches achieve 0 returns. This likely occurs because this setting creates a challenging exploration problem, so exploring with low-level actions or skills learned with VTA or DDO rarely achieves reward. By contrast, exploring with skills learned with LOVE achieves rewards much more frequently, which enables LOVE to ultimately solve the task.
7.3 Scaling to High- High-diiensionnl Observations
Prior works in the offline multi-task setting have learned skills from lowddimensional states and then transferred them to high-dim pixel observations [68]. or have learned hierarchical models from pixel observations [39, 21]. However, to the best of our knowledge, prior works in this setting have not directly learned skills from high-dim pixel observations, only from low-dimensional states. Hence, in this section, we test if LOVE can scale up to high-dim pixel observations by considering a challenging 3D image-based variant of the multi-tass domain above, illustrated in Figure 6. The goal is still to pick up Npic objects in the correct order, where objects are blocks of different colors. Each task includes 3 objects randomly sampled from set of Nov = 6 different colors. The observations are 400 x 60 x 3 egocentric panoramic RGB arrays. The actions are to move forward / backward, turn
left/ / right, and pick up.

---

Returns
250
750
1000
1250
1000
Timestops (1c3)
Optimal
VTA (min. skill length)
skils
LovE
DDO (min. skill length)
No skils (RC)
Figure 6: 3D visual multi-task domain.
Figure 7: Returns on the 3D visual multi-task domain with I-stddev error hars (5 seeds). Only LOVE achieves near-optimal returns.
We consider only the harder sparse reward and generalization setting, where the agent must pick up Npcck = objects in the correct order, after receiving 2,000 pre-collected demonstrations of the approximate shortest path of picking up Npick hyperparameters as in multi-task domain and only change the observation encoder and action decoder (full details in Appendix D.3). Figure 7 shows the results. Again, LOVE learns skills that each navigate to and pick up an object, which quickly achieves optimal returns, indicating its ability to scale high-dim observations. In contrast, the other approaches perform much worse on pixel observations than the previous low-dim observations. DDO also makes progress on the task, but requires far more samples, move toward the objects, All other approaches fail to learn at all, including the variants of VTA and DDO without the min, skill length, which are not plotted.
objects in the correct order. We the same
again learns skills that only operate for few timesteps, though they
8 Conclusion
We started by highlighting the underspecification problem: maximizing likelihood does not nec- essarily yield skills that are useful for learning new tasks. To overcome this problem, we drew a connection between skill learning and compression and proposed via compression and a differentiable model for optimizing our objective. Empirically, we found that the underspecification problem occurs even on simple tasks and learning skills with objective allows learning new tasks much faster than learning skills by maximizing likelihood.
objective for skill learning
Still, important future work remains. LOVE applies when there are useful and consistent structures that can be extracted from multiple trajectories. This is often present in multi-task demonstrations, which solve related tasks in similar ways. However, open challenge for adapting to general offline data like D4RL [22], to ensure that the learned skills do not overfit to potentially noisy or unhelpful behaviors often present in offline data. In addition, we showed connection between skill learning and compression by minimizing the description length of a crude two-part code. However, we only proposed way optimize the message length term and not the model length. Completing this connection by accounting for model length may therefore be an promising direction for future work.
Reproducibility. Our code is publicly available htttp::/ggicubb.com//ddaia
Acknowledgements. We would like to thank Karol Hausman, Abhishek Gupta, Archit Sharma, Sergey Levine, Annie Xie, Zhe Dong, Samuel Sokota for valuable discussions during the course of this work, and Victor Akinwande, Christina Baek, Swaminathan Gurumurthy for comments on the draft. We would also like to thank the anonymous reviewers for their valuable feedback. Last, but certainly not least, YJ and EZL thank the beautiful beach of Kona for initiating this project.
YJ is supported by funding from the Bosch Center for Artificial Intelligence. EZL is supported by National Science Foundation Graduate Research Fellowship under Grant No. DGE- 165651.. CF is a Fellow in the CIFAR Learning in Machines and Brains Program. This work was also supported in part by the ONR grant 100014-21-1-220 Google, and Intel. Icons in this work were made by ThoseIcons, FreePik. VerkorsMarket, from Flatlcon.
10

---

References
[1] Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. arXiv preprint 196.449606.07419, 2016.
[2] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Of- fline primitive discovery for accelerating offline reinforcement learning. arXiv preprint 21X1v:2010.16611, 2020.
[3] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pages 166-175. PMLR, 2017.
[4] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[5] Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, pages 521-531, PMLR, 2021.
[6] Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1):41-77,20 2003.
[7] Ritwik Bera, Vinicius G Goecks, Gregory M Gremillion, John Valasek, and Nicholas R Waytowich. Podnet: neural network for discovery of plannable options. arXiv preprint arXiv:1911.0017/ 2019.
[8] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1912.06800, 2019.
[9] David M Blei and Pedro Moreno. Topic segmentation with an aspect hidden markov model. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages -43-348, 2001.
[10] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
[11] Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 1 2015.
[12] Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
[13] Hanjun Dai. Bo Dai, Yan-Ming Zhang, Shuang Li, and Le Song. Recurrent hidden marimyakov model. 2016.
[14] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. arXiv preprint
[15] Zhe Dong, Bryan Seybold, Kevin Murphy, and Hung Bui. Collapsed amortized variational inference for switching nonlinear dynamical systems. In International Conference on Machine Learning, pages 2638-2647. PMLR, 2020.
[16] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 11110:1611.02779, 2016.
[17] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations, 2019. URL

---

[18] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interac- tion through video prediction. Advances in neural information processing systems, 29:64-72. 2016.
[19] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning, arXiv preprint arXiv:/704.03012, 2017.
[20] Emily B Fox, Erik B Sudderth, Michael I Jordan, and Alan S Willsky. A sticky hdp-hmm with application to speaker diarization. The Annals of Applied Statistics, pages 1020-1056, 2011.
[21] Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXivv1703.088 2017.
[22] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[23] Francisco M Garcia, Bruno C da Silva, and Philip S Thomas. A compression-insppred framework for macro discovery. arXiv preprint arXiv:1711.0900 2017.
[24] Zoubin Ghahramani and Geoffrey E Hinton. Variational learning for switching state-space models. Neural computation, 12(4):331-86) 2000.
[25] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: 1.07007, 2016.
[26] Peter Grunwald. tutorial introduction to the minimum description length principle. arXiv preprint math/0406077, 2004.
[27] Jean Harb, Pierre-Lue Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option:
Learning options with a deliberation cost. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[28] James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. arXiv preprint arXiv: 08866. 2019.
[29] Karol Hausman, Jost Tobias Springenherg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018. htRps: http:////ppparreeeeeeeeee/ttrt/t net:/Eorum?.e--r2077Z2ZR
[30] Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint 1707.02270 2017.
[31] Geoffrey E Hinton and Richard S Zemel. Autoencoders, minimum description length, and helmholtz free energy. Advances in neural information processing systems, 6:3-10, 1994.
[32] Antti Honkela and Harri Valpola. Variational learning and bits-back coding: an information- theoretic view to bayesian learning. IEEE transactions on Neural Networks, 15(4))800-81) 2004.
[33] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[34] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax. arXiv preprint arXiv: 1.01 2016.
[35] Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language abstraction for hicrarchical deep reinforcement learning. arXiv preprint arXiv:1906.00 071990600733,, 2019.
[36] Yiding Jiang Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2020. URL tttpss//openrere// net wttp://wwwt..eetetttsssssss.
12

---

[37] Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. Advances in neural information processing systems, 29:2946-2954, 2016.
[38] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.0599 2020.
[39] Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Neural Information Processing Systems, 32:1 15:11171-1, 2019.
[40] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 12.6000, 2014.
[41] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13X1v:1312.61144 2013.
[42] Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Gonzalz, Edward Grefenstette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 2218-3428. PMLR, 2019.
[43] Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: options for robot learning from demonstrations. In Conference on robot learning, pages 418-437, PMLR, 2017.
Discovery of deep continuous
[44] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29:3675-3666 3683, 2016.
[45] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NIPS, volume 1, page 2, 2010.
[46] Sang-Hyun Lee and Seung- Woo Seo. Learning compound tasks without task-specific knowledge via imitation and self- self-supervised learning. In International Conference Machine Learning. pages 5747-5756. PMLR, 2020.
[47] Youngwoon Lee, Shao- Hua Sun. Sriram Somasundaram, Edward S Hu, and Joseph J Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2018.
[48] Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial Intelligence and Statistics, pages 914-922. PMLR, 2017.
[49] Scott W Linderman, Andrew C Miller, Ryan P Adams, David M Blei, Liam Paninski, and Matthew J Johnson. Recurrent switching linear dynamical systems. arXiv preprint 1610v.6110.8800 2016.
[50] Evan Z. Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta reneoorriifor lcarning without sacrifices. In International conference on machine learning, pages 6925-6935 PMLR, 2021.
[51] Evan Zheran Liu, Ramtin Keramati, Sudarshan Seshadri, Kelvin Guu, Panupong Pasupat, Emma Brunskill, and Percy Liang. Learning abstract models for strategic exploration and fast reward transfer. arXiv preprint 2tt.::20000008966 2020.
[52] Siqi Liu, Guy Lever, Zhe Wang. Josh Merel, SM Eslami, Daniel Hennes, Wojeiech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. From motor control to team play in simulated humanoid football. arXiv preprint arXiv:2105. 12 196, 2021.
[53] Yuchen Lu, Yikang Shen, Siyuan Zhou, Aaron Courville, Joshua B Tenenbaum, and Chuang Gan. Learning task decomposition with ordered memory policy network. arXiv preprint 10977 2021.
13

---

[54] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 2016.
[55] Amy McGovern and Richard S Sutton. Macro-actions in reinforcement learning: An empirical analysis. Computer Science Department Faculty Publication Series, page 15, 1998.
[56] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-53,, 2015.
[57] Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
[58] Ofir Nachum, Shixiang Gu. Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. arXiv preprint arXiv: 08296, 2018.
[59] Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J Lim. Skill-based meta-reinforrement learning. In Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems, 2021. URL jsV1-AQVEFY.
[60] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 2014.
[61] Behnam Neyshabur, Srinadh Bhojanapalli, David McAAlleste, and Nathan Srebro. Exploring generalization in deep learning. arXiv preprint arXiv: 1706 089477 2017.
[62] Scott Niekum, Sachin Chitta, Andrew Barto, Bhaskara Marthi, and Sarah Osentoski. Incre- mental semantically grounded learning from demonstration. In Robotics: Science and Systems, volume 9, pages 10-15607. Berlin, Germany, 2013.
[63] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference Machine Learning. pages 2661-2670. PMLR, 2017.
[64] Aaron van den Oord. Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 03499, 2016.
[65] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint arXiv:1711.0093 2017.
[66] Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.1194: 2020.
[67] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on muchine learning, pages 5331-5340. PMLR, 2019.
[68] Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, al. Learning trans- ferable motor skills with hierarchical latent mixture policies. arXiv preprint arXiv: 12.05062, 2021.
[69] Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465 471, 1978.
[70] Melrose Roderick, Christopher Grimm, and Stefanic Tellex. Deep abstract n-networks. arXiv preprint arXiv: 1rXiv:1710.00459 2017.
[71] Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference.
In International Conference on Machine Learning, pages 8624-8633. PMLR, 2020.
[72] Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. International Conference on Learning Represen- tations, 2020. URL https: : http///ppnneeeeeeeeeeeee wttp://ew......ooooooooooootooo.
httts://opperreiieee
14

---

[73] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representa- lions; 2020. URL. htw.::ttp////ett //tp:://////eeeeeee/ee/ee net/EEorumiddd-JJg... ZRRKK.
[74] Arjun Sharma, Mohit Sharma, Nicholas Rhinehart, and Kris M Kitani. Directed-infe gail: Learning hierarchical policies from unsegmented demonstrations using directed information. arXiv preprint arXiv: 1810.0126, 2018.
[75] Kyriacos Shiarlis, Markus Wulfmeeirr, Sasha Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. In International Conference on Machine Learning, pages 4654 4663. PMLR, 2018.
[76] Yuri M Shtarkov. Univresal sequential coding of single messages. Prob. Pered. Inf., 23:175-186, 1987.
[77] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learn- ing Representations, 2021. URL htttp:://operreeeeetee wwt.eewwweeeeeeeeeeeeeeeeeeeeeee
[78] Sumedh A Sontakke, Sumegh Roychowdhury, Mausoom Sarkar, Nikaash Puri, Balaji Krishna- murthy, and Laurent Itti. Video2skill: Adapting events in demonstration videos to skills in an environment using cyclic mdp homomorphisms, arXiv preprint
2021.
[79] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[80] Daniel Tamneberg, Kai Ploeger, Elmar Rueckert, and Jan Peters. Skid raw: Skill discovery from raw trajectories. IEEE Robotics and Automation Letters, 6(3):4696-4703, 2021.
[81] Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. Advances in neural information processing systems, 7, 1994.
[82] Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. arXiv preprint arXiv:1805.08.0 2018.
[83] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q gelaamnig. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[84] Alexander Sasha Vezhnevets. Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg. David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549. PMLR, 2017.
[85] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: 161 1611005663, 2016.
[86] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanetot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003, PMLR, 2016.
[87] Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiiv22122.056 2021.
[88] Mengjiao Yang, Sergey Levine, and Ofir Nachum. Trail: Near-optimal imitation learning with suboptimal data. arXiv preprint 10 2021.
[89] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
[90] Jesse Zhang, Karl Pertsch, Jiefan Yang, and Joseph J Lim. Minimum description length skills for accelerated reinforcement learning. In Self-Supervision for Reinforcement Learning Workshop- ICLR 2021, 2021. URL https: http:::://oppaeeeioeeeeeeeeeeeee wttp://www.......oo..ootoetotxxt
15

---

[91] Zelin Zhao, Chuang Gan, Jiajun Wu, Xiaoxiao Guo, and Joshua B Tenenbaum. Augmenting policy learning with routines discovered from demonstration. arXiv preprint arXiv:2012.12469 2020.
[92] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. arXiv preprint arXiv:2011.072111 2020.
[93] Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demon- strations for long-horizon robot manipulation. arXiv preprint arXhv:2109.13841, 2021.
[94] Luisa Zintgraf. Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning, arXiv preprint arXiv:1910.08.0 2019.
16


Reviews:
[{"review_id": "r0O_f9NHtAj", "reviewer": "Reviewer_CrHt", "summary": "This submission introduces a general-purpose prior for option discovery from demonstrations: minimum description length, i.e., a compression objective. The authors extend the VTA model from Kim et al., 2019, to include a compression objective and a policy decoder (which can constitutes the option policies). The experiments demonstrate that the algorithm is effective in discrete action settings with both 2D and 3D observations.", "questions": "See above", "limitations": "The conclusion in the main body mentions possible future work but fails to raise awareness on limitations, or at least on open (and as for now) unaddressed open questions. For example, would the proposed method potentially work on the maze navigation tasks in D4RL (https://sites.google.com/view/d4rl/home)? I think that corresponding evaluations can be regarded as outside the scope of this work, but if the authors see possible issues in applying the method to other popular offline RL tasks, this would be an interest addition to the paper's conclusion.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, and the motivation and presentation is good, and I think that finding good, general priors for option discovery is of high interest. The experiments are described clearly and, although they clearly expose a very suitable structure for learning discrete options, showcase the efficacy of the method sufficiently well.\n\nAs the authors note, MDL for option discovery was recently proposed by Zhang et al., 2021 -- it would have been interesting to include this as another baseline for comparison.\n\nIn terms of ablations, I'd encourage the authors to further analyze the effective numbers of skills that are used (explaining the effect of filtering unused skills as described in section 5). In a similar vein, I was wondering how robust the proposed algorithm is to the number of options that should be learned. If I understand correctly, in the experiments the number of objects to navigate to is known beforehand and the number of options is specified accordingly. Suppose that this information is hard to obtain from a given set of demonstrations -- how much would it hurt to specify too many or too few options? To add to that, in Fig. 8 (Appendix), 7 out of 10 skills seem to result in the same actions. Why is that?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lJ3dLTad0E", "reviewer": "Reviewer_cwAV", "summary": "This paper presents a method from offline demonstration learning to HRL. The paper builds upon the generative model of VTA and add an information cost to enforce a better segmentation, which ends up aligning with the ground truth boundaries, The recovered options can be used in the HRL with augmented action space which shows good performance,", "questions": "Can you provide the comparison between your method and compILE? \nCan you also provide some experiments on robotics task with continuous state/action space like jaco pinpad in [1] or reacher?\n\n[1] https://arxiv.org/pdf/2103.10972.pdf", "limitations": "Not that I can see.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe motivation is clear, and the paper is well written and easy to follow. I find the proposed method to be novel and clean. The experiment results also suggest the method is effective at recovering the ground truth boundaries. The connection to MDL also seems convincing. The experiment on HRL and learning new tasks is also impressive.\n\n\nWeakness:\nI would like to see discussion and comparison with CompILE [1], which I think is quite similar to this work, since they are both based on generative models.\n\n\n[1] https://arxiv.org/abs/1812.01483.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "ZdSenq9BJd1", "reviewer": "Reviewer_gYjp", "summary": "This paper presents a regularization method for learning skills from data. The authors are inspired by the minimum description length and combine the maximum likelihood objective with the compressing term to improve the effectiveness of learned skills. The resulting approach LOVE demonstrates the faster-converging speed and semantically important results compared to prior work. Furthermore, LOVE can scale to tasks with image observations. ", "questions": "1. Why don't the authors try to experiment on Atari to show the learned skills?\n2. Why can LOVE discover better boundaries in Figure 3?\n3. How to balance the maximum likelihood target and the length constraint target? If there are many skills of the same length, would the target become ineffective?\n4. Can the proposed compresion scheme generalize to other option learning frameworks?", "limitations": "The authors shortly describe the limitations in the conclusion part. However, I don't quite understand that part and I think the important limitations of this paper are not discussed (refer to the above limitation part).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## After reading other reviewers' comments which are more positive than I thought and reading other rebuttals, I decided to increase my score from five to six and then to seven.\n# Strength\n1. Learning skills from demonstrations is an important task in hierarchical reinforcement learning.\n2. The proposed compression metric should be effective to improve the learned skills in HRL.\n3. The proposed visualizations in Figure3 and Table1 demonstrate the effectiveness of LOVE in various perspectives. \n4. The code is provided.\n\n# Weakness\n1. The authors state in many places, \"prior works do not directly learn skills from high-dimensional pixel observations\". This is false. In fact, many work tries to learn skills on Atari games (real-image games), such as the option critic architecture [4] and RAPL [83].\n2. The proposed compression term is merely a trick beyond VTA. And the compression metric is already used in RAPL [83] and the unsupervised skill discovery literature. So the novelty is limited. \n3.  The title is a bit misleading. The title states \"learning options via compression\". However, the compression is merely a regression term in the learning scheme, and the major body of this approach is still VTA.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Tg6IDEDw229", "reviewer": "Reviewer_94Fc", "summary": "This paper proposes a framework to discover options from the offline data. The framework is build upon the Variational Temporal Abstraction(VTA) method. To discover semantically meaningful skills, they involve an optimization object of compression. They carry out experiments on 2D and 3D multi-task domains. The experiment results show that the proposed framework is able to discover meaningful options and reuses them for new tasks.", "questions": "1. There are a lot of methods for offline option discovery proposed rencently. Can you explain the reason of choosing DDO?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is reasonable for option discovery.\n2. The analysis and introduction of the method is very clear and make it easy to follow.\n\nWeaknesses:\n1. Although I'm not an expert in the field of option discovery from offline data, the idea of penalizing option switching or reward option continuation is common is option discovery from online data., e.g. option-critic[1] and [2]. Thus, the novelty may be not enough.\n2. The choice of comparison method. There are a lot of methods for offline option discovery proposed rencently. Can you explain the reason of choosing DDO?\n[1] The Option-Critic Architecture\n[2] A Compression-Inspired Framework for Macro Discovery", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept