Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
Authors: Xiaofei Wang
Kimin Lee
Kourosh Hakhamaneshi
Pieter Abbeel
Michael Laskin
Venue: CoRL2021 Poster (2021)
Keywords: Reinforcement Learning
Skill Extraction
Human Preferences

Abstract: A promising approach to solving challenging long-horizon tasks has been to extract behavior priors (skills) by fitting generative models to large offline datasets of demonstrations. However, such generative models inherit the biases of the underlying data and result in poor and unusable skills when trained on imperfect demonstration data. To better align skill extraction with human intent we present Skill Preferences (SkiP), an algorithm that learns a model over human preferences and uses it to extract human-aligned skills from offline data. After extracting human-preferred skills, SkiP also utilizes human feedback to solve downstream tasks with RL. We show that SkiP enables a simulated kitchen robot to solve complex multi-step manipulation tasks and substantially outperforms prior leading RL algorithms with human preferences as well as leading skill extraction algorithms without human preferences.

Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
Xiaofei Wang, Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, and Michael Laskin University of California, Berkeley
Abstract: A promising approach solving challenging long-horizon tasks has been to extract behavior priors (skills) by fitting generative models to large offline datasets of demonstrations. However, such generative models inherit the biases of the underlying data and result in poor and unusable skills when trained on im- perfect demonstration data. To better align skill extraction with human intent we present Skill Preferences (SkiP), an algorithm that learns a model over human preferences and uses it to extract human-alignen skills from offline data. After ex- tracting human-pree preferred skills, SkiP also utilizes human feedback to solve down- stream tasks with RL. We show that SkiP enables simulated kitchen robot solve complex multi-step manipulation tasks and substantially outperforms prior leading RL algorithms with human preferences as well as leading skill extraction algorithms without human preferences.
Keywords: Reinforcement Learning, Skill Extraction, Human Preferences
Introduction
Deep reinforcement learning (RL) is framework for solving temporally extended tasks that has resulted in number of breakthroughs in autonomous control including mastery of the game of Go [1, 2]. learning to play video games [3, 4, 5]. and learning basic robotic control [6. 7J. However, today's RL systems require substantial manual human effort to engineer rewards for each task which comes with two fundamental drawbacks. The human effort required to design rewards is imprac- tical to scale across numerous and diverse task categories and the engineered rewards can often be exploited by the RI. agent to produce unintended and potentially unsafe control policies [8, 9, 10]. Moreover, becomes increasingly difficult to design reward functions for the kinds of complex tasks with compositional structure often encountered real-world settings. In this work, we are interested the following research question how can we learn robotic control policies that are aligned with human intent and capable of solving complex real-world tasks?
promising approach better align RL with human intent that proposes an alternate approach to traditional RL algorithm design. Rather than manually engineering a reward function and then training the RL agent, human-in-the-lor proposes for humans to provide feedback interactively to the agent as it is training. This paradigm shift sidesteps reward exploitation by providing the RI. algorithm immediate feedback align best with human intent and, if efficient in terms of human labels required, has the potential to scale RL training across a diverse variety of tasks more reliably than reward engineering.
Human-in-the-loop RL [11, 12, 13] has emerged
RL
So far human-in-the-lon RL systems have been used to play Atari games [12], solve simulated locomotion and manipulation tasks [11, 13], and better align the output of language models [14]. While these initial results have been promising, human-in-the-loop methods are still out of reach for the kinds of long-horing horizon compositional tasks that are desired for real-world robotics. The primary reason is that current methods do not scale efficiently with respect to human labels for more chal-
Correspondence: w.xf berkeeeeleey.dd... Project page: https://sites. wwwiaittssssgggg.. google. coom/view/ssee wwwiee/skklllpee ppref.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

(a) Skill Extraction
(b) Skill Execution
Extract human-preferred skills from offline data
Skill
dec
Action
(4, (9.4. Tei (z|s)
(3,2.
PV
eni skill SPRER
reploy buffer
F Rafs.a
(od, +7)
Ra
maximize Py(t) log.lll. (1))
Our method- Skill Preferences (SkiP)- consists of two phases. During the skill extractions phase, human feedback is used to learn skills. During the skill execution phase. human feedback is used finetune the skills solve various downstream tasks. First, skills are extracted from a noisy offline dataset with human feedback. Second, skills are executed with RL in the environment with task-specific human feedhack.
Figure 1:
lenging tasks. As task complexity increases, the number of human feedback interactions required to attain suitable policy becomes impractical.
To address the ability of RL algorithms to scale to more complex long-horizon tasks, a number of recent works [15, 16, 17] have proposed data-drives extraction of behavioral priors, which we refer as skills. In these methods, behavioral prior fit an offline dataset of demonstrations and is then used to guide the RL policy to solve downstream tasks by regularizing it to stay near the behavioral distribution. Such methods have been shown to successfully solve tasks such as diverse object manipulation [16] and operating kitchen with robotic arm [15]. However, they still require engineered rewards for downstream tasks and, more importantly, assume access clean offline dataset of expert demonstrations that are specifically relevant to the downstream tasks. In real-world scenarios, such clean datasets are highly unlikely to exist. We desire skill extraction methods that robust to noisy datasets, collected by a range of policies, with highly multi-modal structure.
In this work, we introduce Skill Preferences (SkiP), an algorithm that integrates human-in -n-tthe-lopp RL with data-driven skill extraction. Our main insight is that human feedback can be incorporated not only for downstream RL, as done in prior work, but also for extracting human-aligned skills. SkiP learns a human preference function and uses it to weigh the likelihood of trajectories in the offline dataset based on their degree of alignment with human intent. By incorporating human feed- back during skill extraction, SkiP is able to extract structured human-preferred skills from noisy offline data and addresses the core limitation of prior skill extraction approaches the dependence on curated expert datasets. SkiP is both capable of efficiently extracting skills and solving different downstream tasks with respect to human labels. Similar to how prior work in human-in -nt-he--ooopp RL suggested replacing manually engineered reward functions with human feedback, work sug- gests to replace the manual effort needed to curate clean offline datasets with human feedback. We summarize main contributions below:
1. We introduce Skill Preferences (SkiP), an algorithm that incorporates human feedback to extract skills from offline data and utilize those skills solve downstream tasks.
2. We show that, unlike prior leading methods for data-driven skill extraction, SkiP is able to extract structured skills from noisy offline datasets.
3. We show that SkiP is able to solve complex multi-step manipulation tasks in robotic kitchen environment substantially more efficiently than prior leading human-in-the-loop and skill extraction baselines.
2 Background
Reinforcement Learning: As is common with RL methods, we assume that the control process is a Markov Decision Process (MDP) with discounted returns. Such MDPs are defined by the tuple
M = AS R, 7') consisting of states E S, actions A, rewards R = R(s, a), an initial

---

state distribution So Po(-), and a discount factor E [0, A control policy maps states to actions within the MDP and usually takes the form of a probability distribution a ~ (-]ss) The value function V (s) and action-valus value function Q" (s, a) describe the value with respect to future expected returns with respect to an initial state or state-action pair.
(8))): = EMM
7 R(s 0z) 100 =
("(s.a) R(s,a) + (Bg (*)()))
I=
where the first expectation EM,- denotes actions are sampled according to π and future states are sampled according to the MDP dynamics. The goal in RL is the learn the optimal policy:
π arg max J(,, M) == Eg [V*(() (s)).
In addition to the standard MDP setting, our method will also learn skills E Z which consist of an encoder that maps state-action sequences to skill g(0) (2) a,,..., S1+ 11- 9---1-) and decoder that maps state-skill pairs to atomic actions g (a1, 02: ... an|s,
3 Method
The two primary contributions SkiP are (i) introducing human feedback during the skill extraction process to learn structured skills from noisy data and (ii) utilizing human preferences over skills for downstream RL training. Our approach shown schematically
Fig. and detailed in full in Algo. 1. Due to utilizing human feedback to learn the behavioral prior, unlike prior approaches of skill extraction from offline data [17, 16, 15], our method is robust to suboptimal or noisy data.
We first summarize the algorithm and then proceed with its derivation. Shown in Algo. 1, SkiP consists of two phases (i) skill extraction and (ii) skill execution. A human teacher provides feedback during both phases. During skill extraction, a human teacher labels whether a trajectory preferred or not (for details see Sec. 4) to train preference classifier. A behavioral prior is then fit to the offline data with weighted human preference function. During skill execution, the learned skills are rolled out by an RL agent- - a Soft Actor Critic (SAC) [18] that trained with task- specific human preferences. As such, human feedback is used during both phases of the algorithm. We proceed define notation and provide derivation.
The SkiP Algorithm:
Preliminaries and Notation:
Our method is composed of two phases (i) the skill extraction phase and (ii) the skill execution phase. During the skill extraction phase, we are given an of- fline dataset D which consists of tank-sgnostic. multi-modal, and potentially noisy demonstra- tions. We denote trajectory sequences as T. (st, at..... 8 4++H-1). action sequences as a, = 44+ 11), and skills which decode into action sequences as 2 E Z.
Learning Behavioral Priors with Human Feedback (Skill Extraction):
Our main insight is to human preferences in order to fit a weighted behavioral prior over an offline dataset of (potentially noisy) demonstrations. Our method builds prior work for behavioral extraction from offline data via expected maximum likelihood latent variable models [17, 16, 15].
Specifically, prior work [17, 16, 15] considers a parameterized generative model Po (ax (ar|s:) over action sequences where a, = (at: ..., (1+H-1) that represents a behavioral prior and is trained to replicate the transition statistics in the offline dataset:
Σ log (pa (pa(aalss))
Pa E arg max ENDD a
In our approach, we consider an adaptive behavioral prior that is biased towards trajectories that achieve higher rewards according to the human preference function. This can be particularly useful in diverse datasets collected with suboptimal or noisy policies or multiple policies of varying exper- tise. For example, one could imagine multiple humans collecting demonstrations or multiple robots exploring their environment. Similar to Siegel et al. [19], we seek a behavioral prior that is biased towards the high reward trajectories in the dataset while also staying close to the average statisties in the dataset. However, unlike prior work on weighted behavioral priors [19, 20, 21] the weight is determined through the human preference function and we aim to maximize sequene likeli- hood as opposed to single-times timestep actions.
3

---

Algorithm SkiP: Skill Preferences
Skill Extraction Phase
INPUT: offine dataset B
Initialize prior p. skill encoder 4dz skill decoder Рéy Initialize learned preference classifier Ps
A human provides labels (yI, 32....) for 10% of the trajectories in B and stores them in new buffer D>
for each iteration do
Update u by maximizing E. ((g.r)=bly (og)))T) + 10 y) log(1 Pa (T))
for each iteration do
Update p. Aog Po; by optimizing Cerior (3)
(Update preference weighted behavioral prior]
Skill Execution Phase
Initialize parameters of actor тo.. critics Qo2 and Q and reward model R.
Initialize dataset of preference D <- 9 and dataset of transitions B <- 0
for Each iteration do
Each environment step do (T(2))))). S14 p( 2.) B BU (sL, R1, (st, 21), s1+H)
iteration % == then
for step t= 1...M do
(To 1i (+) ~ 13 query human for label y, +- DU (TO -11)) .1)
(Get preference lahels]
for each gradient step R. do
Sample (T)²), x(2), v) ~ update R. with min Creward
(Update preferences}
Relabel entire replay buffer B with R.
for each gradient step of agent
Sample (s, a, s', Ht) ~ B, update nei by optimizing CSAC (Appendix B.1)
(Update agent)
Update Q62 and Q by optimizing L (Appendix B.1)
We formulate this as:
|r| Σ ((T)) Po (arls)) such that EAeD [DE (Pa [p) S
Pa E arg max EN-D
(2)
where p denotes the empirical behavioral policy and w(st, ae) is the weighting functiion The non- parametric solution to the above optimization is given by:
Pa (ay st) x p(ar(s)) exp (w(r)/n))
where have used to avoid specification of the normalization factor, and v represents tem- perature parameter that related to the constraint level 5. The above non-parametric policy can be projected into the space of parametric neural network policies as [20, 19]:
[l exp (w(T:)/T) log (pa (Pa(aals)))
Pa E arg max BNDD
For the choice of the weighting function, we use the learned preference classifier Pe (y(T) which inputs trajectory and outputs the likelihood of this trajectory being human-preferred with y E [0, 1]- (x(y)) is learned by sampling a small subset of the offline dataset and soliciting human feedback label preferred versus preferred trajectory: (<T)) := log (TE).
In this process, we treat the temperature 7) as the dyper-paramene choice. This implicitly defines the constraint threshold 5, and makes the problem specification and optimization more straightforward. For our practical implementation, we variational autoencoder similar to [17, 15] but softly weighted to maximize the likelihood of human- preferred transitions. We introduce latent variable with a Gaussian prior such that the ELBO loss given by:
buman--referreed
log logp(a)ls) 2 E.wD, (x1r) log Ps,
2) +B (log((>)) - log aou (z|T))
Lx
Lug
This is the standard VAE loss applied to action sequence modeling where 3 is scalar controlling the regularization strength and Φ1, 62 are neural network parameters that are optimized during train- ing. Note that qo, encodes trajectories into a latent vector and Pár decodes latent vectors and the

---

starting state back into action sequences. Our training objective weighs this loss with the preference function. Thus, our overall skill extraction objective maximize:
arg max E [P (Fr + -reg))
(5)
01.02
Reward learning and human preferences over skills (Skill Execution): Unlike traditional RL. where the hand-engineered rewards are available, we consider the preference-basee RL frame- work [11, 12, 13, 22|: (human) teacher provides preferences between the agent's behaviors and the agent uses this feedback to perform the task. In order to incorporate human preferences into deep RL, Christiano et al. [11] proposed a framework that learns a reward function R. from preferences. In this work, we modify the preference framework to operate not over atomic state-action transitions but rather state-skill transitions that have substantially longer time spans.
Formally, assume access
offline dataset (the agent's replay buffer) B of state-action tran- sitions and sample state-skill sequence pairs 7(*), .22 for which a human provides a binary label E 1), where 7(x) = (st, 24, S 2t-H,.... S(e+ MYH; M(++M)))) where H is the length of actions the skill decodes and M is the total number of state-skill transitions. Note how such trajectories are H times longer than we were to sample state-action trajectories of length M.
The reward function R therefore fits Bernuulli distribution across sequences. In this work, we learn a parameterized reward function RG as in [13] utilizing a Bradley-Tery model [23] in the following manner:
cxp E, (s
Pull( > [3)
(6)
Eie(o.1) exp E, (s..
Here, the operator A > B means that A is preferred to B. R. can therefore be interpreted as binary preference classifier where labels are provided through human feedback. The parameters n) of the neural network are updated by optimizing a binary cross-entropy loss:
C Reward
(7 .1 wDD (0) log Pr( > -|+)| + + P,( To [0
Experimental Setup
Environments: For our experiments, we use the robot kitchen environment and offline dataset from the D4RL suite [24]. This environment requires a 7-DOF (6-DOF arm and 1-DOF gripper) robotic arm to solve complex multi-step tasks in kitchen. Due to the 7-DOF control and compositional long-horizon nature of the tasks, this environment cannot solved by standard methods such as SAC or behavior cloning [15].
We desire our method work on suboptimal offline data and, unlike prior skill extraction approaches [15, 16, 17] do not assume that the offline dataset consists solely of expert demonstrations. We simulate a noisy offline dataset by combining 601 expert trajectories and 601 noisy trajectories generated by random policy. The expert trajectories involve various structured kitchen interactions such as opening the microwave and operating the stove. We solicit human feedback 10% of the total trajectories or equivalently 120 human labels.
Offline dataset:
Downstream tasks: We 6 different downstream tasks shown in Fig. 2 that vary in difficulty to evaluate our approach. The task suite consists of tasks that require one, two, or three subtasks to be completed in row in order achieve the overall goal. We note that even the tasks with one subtask is challenging for RL methods that operate over atomic actions and do not leverage skills, as is shown in the experimental results.
Simulated human: Similar to prior work [11, 13), we obtain feedback from simulated human teachers instead of real humans. During skill extraction, human provides labels whether trajec- tory is noisy or structured.' During skill execution, the simulated human assigns positive labels to trajectory segments that have made more progress toward completing the desired task. Progress is calculated by computing ||sM - a||2 - ||s] 8 8/22, where ş is the state when the target task is completed.
Here, we remark that limited number of human labels (10% of the total trajectories) is utilized in our experiments for skill extraction.
5

---

one task
two tasks
tasks
microwave
kettle
microwave
microwave
kettle
kettle
burner
kettle
burner
burner
cabinet
We evaluate in the robot kitchen environment from D4RL [24], which requires 7-DOF robotic arm to operate kitchen. Within this environment, we consider a variety of manipulation tasks of varying difficulty. The simplest tasks involve one subtask opening a microwave moving the kettle- while more challenging tasks require the agent to compose multiple subtasks, Overall, we consider evaluation tasks that require chaining one, two, or three subtasks.
Figure 2:
kötchen robot
offline noisy dataset
Learn Skills
expert
Human
diverse skills
Feedback
Figure 3: An illustration of the skill extraction procedure within the robot kitchen environment. Starting with a noisy offline dataset, which consists of both expert and random actions, our method fits behavioral prior to the offline data using human feedback to identify human-pret preferred motions which results a set of diverse skills that can then be finetuned to downstream tasks.
buman-preferred
Baselines: In addition to our method, we compare to Atomic Preferences which based on PEB- BLE [13]: a state-of of-tte-o-thheartt human preference RL method. it pretrains the SAC agent with behavior cloning over the optimal offline dataset and trains the online SAC agent with human preferences atomic transitions instead of high-level skill transitions. We also compare to Flat Prior which learns single-step action prior the atomic action space the optimal dataset and trains online SAC agent regularized with the action prior over ground-truth reward. The Oracle we compare to is SPiRL, leading skill extraction with access to the ground truth (expert demonstrations and ground truth reward) in Fig. 4.
5 Experimental Results
For the experimental evaluation of our approach, investigate the following questions: (a) Can SkiP solve challenging long-horizon tasks and how does our method compare to prior leading ap- proaches? (b) How do SkiP compare to oracle baseline that extracts skills from perfect expert demonstrations and has access to the ground truth reward? (c) Is it necessary to provide human feed- back during skill extraction is it sufficient to fit unweighted behavioral prior the offline data? (d) How should we incorporate human feedback during the skill execution phase?
We evaluate SkiP and related baselines the 6 tasks shown in Fig. 2 and display the learning curves in Fig. 4. We observe that SkiP the only method (except for the Oracle) that is capable of solving the majority of tasks in the robot kitchen task suite and outperforms the baselines on all environments. On 5 out of tasks, SkiP able to match the oracle baseline asymptotically which means that it arrives at the optimal solution.
Main Results:
SkiP is also human-label efficient. During skill extraction, only 120 labels are required to train the preference classifier. During skill execution, 300-1K labels are required to solve most tasks depending on the task's complexity. We hypothesize that human label efficiency better during the skill extraction phase because classifying structured and noisy skills from static offline dataset is

---

Microwave
Kettle
Microwave-
1.00
1.00
2.0
M
0.75
0.75
1.0
0.50
0.50
1.0
0.25
0.25
0.5
0.00
0.00
0.0
Kettle-Bumer
Microwave-Ketti-Beee Burner
Nettee-Bumer-Cabne
3-
2
2
averge
1.0
0.5
0.0
0-
0.25
1.00
0.25
SKIP 3x
SKIP
PEBBLE
Flat Prior
Oracle
0TV stops (1M)
Figure 4: SkiP and baselines (Sec. evaluated over six tasks in the robot kitchen environment shown in Fig. 2. SkiP outperforms both baselines across the majority of the tasks and is the only method that capable of matching the Oracle on most tasks. We also compare SkiP to SkiP with 3x more human labels and find comparable performance between the two versions. SkiP solves most tasks given 300-100 human labels depending on the complexity of the task.
Microwave
Kettle
Microwave-Kattle
1.00
2.0
N
W
0.75
average
0.00
0.00
0.0
Kettle-Bumerr
MicrowsveKKetle-Buree
Ketle-Bumer-Cabine
2.0
3
1.5
1-
0.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.30
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
feedback)
(no feedback)
Figure 5: SkiP with human feedback vs SkiP without human feedback during skill extraction. learn- ing curve with shaded region representing standard error across three seeds. Both algorithms learns prior from the suboptimal dataset and were evaluated with online RL. SkiP with human feedback outperforms SkiP without human feedback on all environments
casier than classifying task-specific preferences from an evolving replay buffer. Further human label efficiency improvements pose interesting research directions for future work.
Ablation Studies: To further understand the properties of the SkiP algorithm, we investigate whether human feedback is necessary during skill extraction well how the human preference reward function compares to alternate approaches to human feedback during skill execution.
Is it necessary to provide human feedbuck during skill extraction is it sufficient to fit un- weighted behavioral prior over the offline data? The offline dataset used throughout this paper consists of suboptimal data that is a mixture of expert and random actions. We compare fitting Gemaan-eedbbck weighted behavioral prior as opposed to an unweighted behavioral prior that max- imizes the likelihood of all action sequences equally. For the skill execution phase, both methods have access to the same human preference reward function. The results shown in Fig. 5 indicate that the method, which extracts skills without human feedback, is unable to solve any of the tasks suggesting that human feedback is essential for skill extraction from suboptimal offline data.
How should we incorporate human feedbuck during the skill execution phase? ? Instead of prefer- ences, a simpler approach to learning from human feedback provide binary feedback if task (or subtask) has been solved and learning reward classifier to guide the RI. agent. We implement

---

Microwave
Kettle
Microwave- Kettle
1.0
M
0.8
0.75
0.8
1.0
0.50
0.25
0.4
0.5
0.2
0.0
Kettle-Bumer
MerrowsveKKattie-ee Burner
Matlo-Burmer-aablee
average reward
2-
1.5-
2
0.5
0.50
env steps (1M)
SKIP (with proference)
SkiP (with learned sparse reward)
Figure 6: SkiP with preferences vs SkiP with learned sparse reward. Learning curve with shaded region representing standard error across three seeds. Both algorithms use the same prior. SkiP with preferences outperforms SkiP with learned sparse reward on out of environments.
this by providing positive reward of for high-level transition (St; 2, 8t+ when subtask has been completed and 0 otherwise. Using the same number of human queries for both approaches, we compare learning with preferences opposed to learning from sparse rewards. For both approaches, we use human feedback for skill extraction. As shown in Fig. 6, RL with a reward classifier for sub- task completion able solve some tasks but generally performs much worse than RL with human preferences.
6 Related Work
fluman-in-the-loge Reinforcement Learning: Several works have successfully utilized feedback
from real humans train RI. agents [25, 11, 12, 26, 13, 27, 28]. One of major directions directly utilizing the human feedback as learning signal [29, 27, 25] but assumed unlimited access to human labels which limited their practicality for more challenging tasks. To address this limitation, number of works proposed learning reward model from human feedback [26, 28, 30, 31, 32, 33]. Recently, several works have successfully combined human preferences with deep RL algorithms to learn basic locomotion skills well playing video games from pixels using human [11, 12, 34, 13]. However, these methods are limited to short-horizon or cyclic tasks and do not scale to more challenging compositional multi-step tasks. In this work, we investigate how to scale human preferences to such challenging tasks by specifying preferences over skills.
Data-driven Extraction of Behavioral Priors: Behavioral prior or skill extraction refers to fitting a distribution over an offline dataset of demonstrations and biasing the agent's policy towards the most likely actions from that distribution. Commonly used for offline RL [21. 19, 20], behavioral priors learned through maximum likelihood latent variable models can also been used as skills for structured exploration in RL [16], solve complex long-horizon tasks from sparse rewards [15, 17), and regularize offline RL policies [21, 20, 35]. limitation of these skill extraction methods that the quality of the behavioral prior is highly dependent the demonstrations in the offline dataset. Since a behavioral prior models maximum likelihood transitions in the offline dataset, suboptimal, noisy, irrelevant transitions can degrade downstream policy learning. In this work, introduce human feedback into the skill extractions phase to learn a human preferred behavioral prior which enables skill extraction methods to be robust to suboptimal offline data.
Conclusion
We presented Skill Preferences (SkiP) an algorithm that uses human feedback for both skill extrac- tion well as execution, and showed that SkiP enables robotic agents to solve long-horizon com- positional manipulation tasks. We hope that this work excites other researchers about the potential of learning with skills and human feedback.

---

8 Acknowledgements
We would like to thank Berkeley DeepDrive, Tencent, ONR Pecase N000141612723, and NSF NRI 2024675 for supporting this research.
References
[1] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1 140-141000 1140,144, 2018.
[2] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-399 2017.
[3] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7550):529-5333 2015.
[4] OpenAl. Openai five. https://blog openai. com/oppnaii-iveee//eeeeao 2018.
[5] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-33 2019.
[6] I. Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 1910.0711,. 2019.
[7] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. In Conference on Robot Learning, 2018.
[8] D. Hadfield-Menell S. Milli, P. Abbeel, S. Russell, and A. Dragan. Inverse reward design. In Advances in Neural Information Processing Systems, 2017.
[9] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete prob- lems ai safety, arXiv preprint arXiv: 160.:6666.6665 2016.
[10] A. M. Turner, N. Ratzlaff, and Tadepalli. Avoiding side effects in complex environments. arXiv preprini arXiv:2006.06547, 2020.
[11] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in neural information processing systems, 2017.
[12] B. Ibarz, J. Leike, Т. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 2018.
[13] K. Lee, L. Smith, and P. Abbeel. Pebble: Feedback-efficiee interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint 10X0092106.0509,, 2021.
[14] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. F. Christiano, and G. Irving. Fine-tuning language models from human preferences. CoRR, abs/1903 1et/11999999,.... 2019. URL httt:://arx/a /ttp///aaviv///////av///////aaav org/ abs/1909 08593.
[15] K. Pertsch, Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.
[16] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2021. URL https: //ppeneevie http://eeeeeeeeeeeeeeeeeeeeeeeee aet/foorum2xd=Vvvv--00=

---

[17] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. {OPAL): Offline primitive discov- ery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021. URL htw.::tps/// htttss://ppenreiieee
weet/forunnddd-VGLL....
[18] T. Haarojja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning. 2018.
[19] N. Siegel, J. T. Springenberg. F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, and M. A. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. ArXiv, (bs/2000..08366. 2020.
[20] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable ooflpy reinforcement learning. CoRR, abs/1910.001777, 2019. URL http: // //arxiv. org/ aaaaabss1991 https//11...00..111 00177.
[21] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. CoRR, 0bs/1911.113611 2019. URL http://arxiv org/ httb////9111 11361.
[22] J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018.
[23] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[24] Fu, A. Kumar, O. Nachum, Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint PrXiv:204.002//1 2020.
[25] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint arXiv:1902.04257. 2019.
[26] W. B. Knox and P. Stone. Interactively shaping agents via human reinforcement: The tamer framework. In International Conference Knowledge Capture, 2009.
[27] MacGlashan, M. K. Ho, Loftin, B. Peng, D. Roberts, M. E. Taylor, and M. L. Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning, 2017.
[28] G. Warnell, N. Waytowich, V. Lawhern, and P. Stone. Deep tamer: Interactive agent shaping in high- state spaces. In Conference on Artificial Intelligence, 2018.
[29] P. M. Pilarski, M. R. Dawson, T. Degris, F. Fahimi, J. P. Carey, and R. S. Sutton. Online human training myoelectric prosthesis controller via actor-critic reinforcement learning. In International Conference on Rehabilitation Robotics, 2011.
[30] L. Pinto and A. Gupta. Supersizing seff-supervision: Learning grasp from 50k tries and 700 robot hours. International Conference on Robotics and Automation, 2016.
[31] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):411-366 2018.
[32] J. Fu. A. Singh, D. Ghosh, L. Yang, and S. Levine. Variational inverse control with events: general framework for data-driven reward definition. Advances Neural Information Proce Processing Systems, 2018
[33] A. Xie, A. Singh, S. Levine, and C. Finn. Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning. 2018.
[34] Z. Cao, K. Wong, and C.-T. Lin. Human preference scaling with demonstrations for deep reinforcement learning. arXiv preprint xrXiv22007.12004 2020.
[35] A. Nair, A. Gupta, M. Dalal, and S. Levine, Awac: Accelerating online reinforcement learning with offline datasets, 2020.
10


Reviews:
[{"review_id": "wL2R2ylfHBu", "reviewer": "Reviewer_noDs", "summary": "This paper presents a clean and elegant approach for 1) extracting skills from offline datasets of suboptimal/noisy human demonstrations and 2) using the extracted skills to solve downstream, multi-step complex robot manipulation tasks, *by intelligently leveraging a human-in-the-loop*. Critically, the proposed approach — SkiP — uses preferences from humans to learn good priors over skills that are worth extracting from offline datasets (e.g., skills that lead to higher rewards/succeed at the goal more often), and in addition, use human preferences during task learning to identify the “right” skill transitions (which skills to use) to best solve complicated tasks.\n\nThis approach is well-motivated, and the evaluation is complete, and clear. Looking at a series of D4RL tasks with a 7-DoF robot manipulator (6-DoF Arm + 1-DoF Gripper) the proposed approach SkiP not only outperforms strong approaches that 1) just extract skills without preference information and 2) are trained with human preference information, but is also label efficient from a human perspective, with clean ablations that show how SkiP performs when human-labels are removed at each of the two stages (extraction and learning).\n\nCoupled with nice oracle experiments and visualizations, this is a nice step forward in learning and using skills from offline data with arbitrary optimality guarantees, presenting a nice step forward for long-horizon robotics.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is thorough and extremely clear. The strengths of this approach involve the simplicity of the two stage approach — first extract skills by using human preferences to “weight” behaviors that are more relevant/meaningful, then use human preferences again during learning to figure out the right “skill transitions” to quickly learn how to compose skills to solve complex tasks. The experiments are complete, the baselines strong, and the ablations/oracle numbers paint a clear picture of the benefits of the proposed approach.\n\nIn terms of weaknesses, I am worried about the actual performance of SkiP models in the presence of offline data that is more muddled (all demonstrations are suboptimal in some way). As I currently understand the data used to train the SkiP models, a dataset of off-line demonstrations was constructed from 50% optimal demonstrations of a task, and 50% data drawn from a random policy. I think this is a nice first step, but I don’t know how this approach would perform with noise directly injected into the optimal demonstrations (sub-optimal behaviors at times). \n\nSeparately, while I understand that this is keeping with prior work, using simulated humans for labeling is still a limitation in my opinion; I don’t think that this is specific to this work, but I would hope that future work looks to how actual humans preferences correlate with those of the simulated humans.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a straightforward solution, performs a robust and thorough evaluation, insightful analyses, and situates itself well related to prior work. While there are some weaknesses around the offline dataset used to validate the method, I believe this is strong work that could be extremely useful for the field, and propel our ability to scale manipulation to more complex, long-horizon tasks.\n\nWould be nice to eventually get this on a robot though ;) ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Questions\n---\n\n- [Curiosity more than anything else] Do you have a sense of how robust SkiP is to noise in the human-preference labeling process? Is there more of an impact when there’s preference noise during skill extraction (my guess) or learning?\n\nTypos\n---\nFigure 1 Caption - “to denoise behavioral prior”? \n131 - Guassian —> Gaussian\nFigure 2 Caption - “mire challenging” —> “more challenging” "}}, {"review_id": "iEQNmV-OpVO", "reviewer": "Reviewer_hYAX", "summary": "This work introduces an algorithm, SkiP, that uses preferences to learn skills from noisy demonstrations, and uses these skills along with more preference data to learn tasks using RL. This work also explores how much preference feedback is needed, as well as determining whether preferences or sparse rewards produce better learning results during skill execution. SkiP is demonstrated outperform baseline algorithms, while improving over prior methods by utilizing noisy demonstration data. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "SkiP’s main strength is its ability to extract skills from noisy demonstrations without having to have an expert define a reward function, and the results seem convincing of this claim. The experiments are well-rounded as well. This paper shows SkiP’s performance with more human data, to show that the training does not improve much with more feedback than the amount used (120 preferences/~10% of trajectories during skill extraction). SkiP is also tested to see if sparse rewards could be used rather than preferences during skill execution. It seems evident from the testing that SkiP’s formulation provides good learning over other formulation options. \n\nSkiP does still seem to require correct preference feedback to learn well. If a person was confused about how certain skills worked, or mislabeled preferences (particularly given the 300-1K required labels during skill execution), it’s unclear how SkiP would perform. However, SkiP is still able to learn quite well from noisy demonstrations, which is very useful for skill learning.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending a weak accept because while SkiP is able to learn very well from noisy demonstrations, it still requires a decent amount of correct information from humans in the form of trajectory preferences. However, this work does seem like it is a good step forward, and preferences are easier to provide than demonstrations.\n\nUpdate post-rebuttal: The new experiments on more realistic noisy demonstrations and the analysis of noisy preference labels have caused me to change my recommendation to strong accept (contingent on these results being added to the paper or appendix).", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Issues to address in the author response/revision period:\n-\tIs there any existing data that suggests how SkiP might behave if some preferences were mislabeled?\n-\tIs the state-skill sequence on line 147 supposed to end in s_(t+(M)H), z_(t_(M)H) instead of s_((t+M)H), z_((t+M)H), as there are M state-skill transitions of length H starting from s_t? Or perhaps I misunderstood the notation.\n-\tUnderstandably, this wasn’t tested with human teachers, but instead with simulated people. How long might it take for a human teacher to provide 120 + 300/1K preference labels to full state-skill sequence pairs?\n-\tJust a small note that there are a few little typos throughout the paper (vecotr instead of vector in line 134, for example)"}}, {"review_id": "VsjU9tnqab6", "reviewer": "Reviewer_6ucT", "summary": "The paper proposes a deep RL algorithm that jointly leverages human preference feedback and offline datasets of agent experience (via the extraction of learned skills) to enable the learning of long-horizon tasks. As a result, it can learn longer and more complex tasks than prior human-in-the-loop RL papers, and better cope with mixed offline data quality than prior skill-based RL approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- **The intuition of the method is convincing**: human feedback is used in two separate ways by the approach: (1) to guide skill extraction towards human skill preferences, (2) to provide feedback during downstream learning. Both cases are well motivated in my opinion:\n    (1) Generally, we cannot assume that the offline experience dataset used for skill pre-training contains only optimal sequences. At the same time we cannot assume that it features reward annotations for the target task. Thus, we lack a signal to determine which of the sequences are high quality and should be used for learning skills. Using human feedback for this choice, as proposed by the paper, is a particularly intuitive choice since humans are able to provide an estimate of the desirability of a motion sequence irrespective of the particular downstream task.\n    (2) Using preferences for downstream task learning is not a new idea, there is a long line of work trying to learn rewards from preferences (see related work section). One critical issue is that early in training, when mostly random trajectories get compared to one another, it is hard to formulate effective preferences and training is slow. Thus prior work has used unsupervised pre-training too jump-start this process (PEBBLE, Lee et al. 2021). Here, the unsupervised pre-training is replaced with pre-training from demonstration-like offline data which enables preference-based learning on substantially more complex and longer-horizon tasks -- an extension which makes sense to me.\n  \n- The approach is technically simple: given the problem setting the chosen technical approach is rather straightforward but seems to achieve good empirical performance. Thus, I would see its simplicity as a strength.\n\n- The experimental evaluation shows the benefits of the approach, both in using human feedback for skill extraction (Fig 5) and using learned skills for preference-based RL (Fig 4)\n\n- The performed experiment on preference-based reward vs (learned) sparse reward (Fig 6) is interesting -- it essentially shows that human preferences can provide a denser reward signal than the rather sparse environment subtask rewards. This might be a known result in the literature on RL from human preferences, but I am less familiar with this literature and thus found it interesting.\n\n\n## Weaknesses\nIn my opinion, the main weakness is in the experimental evaluation of the proposed approach. In particular the dataset used for skill pre-training does not reflect a realistic mixed-quality dataset:\n\n(A) The offline data used for skill pre-training consists to equal parts of (1) expert-level human-teleoperated sequences and (2) completely random-action trajectories. Thus it is trivial to distinguish between \"good\" and \"bad\" trajectories and, more importantly, none of the behaviors seen in the \"bad\" trajectories is actually relevant for solving the downstream task. In a more realistic evaluation scenario the data would instead consist of \"better\" trajectories that feature _mostly_ useful behavior and \"worse\" trajectories that contain lower quality interactions but _also_ have some reasonable behavior. It seems important to investigate whether the proposed simple approach of preference-based weighting during skill extraction would still work in such a more ambiguous scenario instead of the black-and-white scenario that was tested now, particularly since the preference-based skill extraction is the major novelty of the proposed approach.\n\n(B) The approach is only evaluated in a single environment with a single pre-training dataset (although on different downstream tasks). Thus it lacks evaluation on different robotic agents with different observation and action spaces and different dataset characteristics. That being said: while this does make the evaluation less comprehensive, I would see this point as less severe than the point about the used dataset above, since the kitchen environment used for evaluation is rather challenging.\n\nMinor Comments:\n\n(C) The current submission seems a bit unpolished which sometimes impacts its clarity. While this can surely be addressed with a bit of minor rewriting, it seems important to clarify some details (see questions below) and proof-read the submission again to fix typos (eg caption to Fig 2, L134, caption to Fig 6)\n\n\n## Questions\n(a) Is the human feedback labeling during skill execution done per sequence or per skill segment? What are the states that get compared in the preference function in L177?\n\n(b) Why is the \"Flat Prior\" in Fig 4 a flat line and not a learning curve? Since it finetunes the policy with SAC I would expect this to be a learning curve.\n\n(c) Does the proposed approach use regularization with the skill prior during downstream RL as is done in SPiRL [15]? The update equation Algorithm 1 points to (in appendix A) is the vanilla SAC update equation that uses the non-learned uniform prior over actions (ie regularizes with a maximum entropy objective) instead of minimizing divergence to the learned skill prior. If indeed the vanilla SAC objective is used can you please elaborate on why the learned skill prior is not leveraged here?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I am convinced by the relevance of the problem and the intuition of the proposed approach. The technical implementation is simple and solid. I am slightly on edge due to the somewhat toyish \"mixed-quality dataset\" setup, but I lean towards accepting the submission, in particular if the authors can address some of the points suggested above -- focussing on a more realistic mixed-quality dataset composition.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "## Suggestions to improve the paper\n- In order to address (A) I would suggest collecting a more realistic mixed-quality dataset, e.g. by training a BC policy on the expert data and using it to collect lower-quality trajectories for training (they will be lower quality due to accumulating errors) --> this technique was used in SPiRL [15] to investigate robustness to lower-quality data. Alternatively one could also add action-noise to the demonstration actions or to rollouts from a trained oracle SPiRL policy to generate lower-quality sequences that are **not** completely random. By combining these sequences with the higher-quality expert sequences the proposed approach could be evaluated in a more realistic mixed-quality setting.\n\n- With lower priority: to address (B) one could add evaluation of the idea on another environment, e.g. any of the other environments used in the SPiRL paper, to show that the proposed approach is applicable in flexible environments.\n\n- It would be interesting to add a baseline to the comparison in Fig 6 that uses the SkiP skills with **true** sparse subtask reward (ie does not learn to estimate the subtask reward but uses the GT environment reward) --> this could show how much performance we loose from the reward estimation (also in comparison to the results in Fig 4)"}}, {"review_id": "Qr7WqmIaJi", "reviewer": "Reviewer_XBne", "summary": "This paper studies the problem of reinforcement learning with access to large offline (potentially diverse and noisy) demonstration datasets. Under this setting, the traditional approach is to learn behavioral priors in the form of skills (i.e. a latent var z that maps to a sequence of actions) in the first phase, and then running standard reinforcement learning algorithms on top of the learned skills in the second phase. Doing RL on top of skills, as opposed to raw actions, makes it easier to solve long-horizon tasks.\n\nThe paper operates in the above setting, and examines the scenario in which we have additional access to humans in the loop -- i.e. how can we maximally improve existing approaches if we're allowed a reasonable amount of human labels. The proposed method (SkiP), proposes to take advantage of human labels in both the first phase (skill extraction where we learn to map latent z to action sequences) and the second phase (skill execution where we run RL over skills). For skill extraction, the humans label trajectories with a likelihood score from 0 to 1, and a generative model of skills is learned with a beta-VAE objective that weighs trajectories based on the labels. For skill execution, the humans are given pairs of trajectories and choose the more preferred trajectory, which is used to recover the reward function that maximizes the probability (under the Bradley-Terry model) of the human preferences.\n\nThe paper experiments on a robot kitchen environment that involves long horizon tasks. The SkiP algorithm is tested using 100 to 1000 (simulated) human labels, showing better results than PEBBLE (which also incorporates human feedback but over individual actions instead of skills).\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the problem addressed is important, and the proposed method is simple and clear\n- learning preferences over action sequences instead of atomic actions is more efficient and natural for humans\n\nWeaknesses:\n- the initial conditions and termination conditions of the skills are not learned, instead a horizon H is picked and the total trajectory is decomposed into action sequences of length H. This seems to essentially be offloading all the work to the humans, e.g. a sequence of H actions will be rated highly if the actions make sense and also aligns with reasonable initial/terminal conditions. There seems to be a missing piece of segmenting the data into skills before presenting to the human (or even incorporating human feedback to segment the data in a smart way).\n- the motivation leading up to Eq 4 and 5 is not very well presented. Eq 3 and the equation before it are presented without much intuition (looks like it's pulled from [19,20], but should probably be presented more formally e.g. as a lemma). Also, hyperparameters change from delta to T (and possibly to beta?), but the exact correspondence is not clear. It's not clear to me where T ends up affecting the final objective in equation 5. More importantly, the beta-VAE objective is also pulled without a direct connection to the derivation in Eq 3. For example, it's not obvious that putting the preference P(\\tau) as a multiplier is the principled thing to do / aligns with Eq 3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed method is simple and tackles an interesting problem. But, as it stands, the contribution is a bit thin -- the modeling techniques are chosen without new insights and without comparison to any other choices. The paper can be improved by considering more sophisticated approaches to curating action sequences to present to the human labeler, comparing with other choices of generative models (or having a more convincing derivation of the current modeling approach e.g. why beta-VAE). ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the problem addressed is important and the proposed method is simple, clear, and able to learn very well from the demonstrations. The two-phase approach is intuitive, and the experimental evaluation are convincing and show interesting results.

There are a number of concerns brought up by the reviewers that the authors should try to address during the response period, including: 
1) The offline dataset is not representative of a realistic mixture of mixed-quality demonstrations, and collecting and learning from a more realistic dataset (as suggested by one of the reviewers) can demonstrate the effectiveness of the approach a bit better.
2) The definition of skills: the authors use a fixed time horizon for skills and the initial and termination conditions are not learned. It would be interesting to have an automatic way of segmenting the skills or incorporating human feedback to segment the skills. This also makes me wonder if the skills are just expert action reconstruction sequences (which makes me question the generalizability of them), and if you could simply remove the preference predictor weighting and train a standard VAE over the expert data and basically learn the same action sequences (skills).
3) There are some concerns about the contributions of the paper. Specifically the paper does not provide new insights or comparison to other modeling choices (e.g. the specific choice of generative models).
4) The approach is only evaluated in a single simulated environment, and it would be interesting to assess the effectiveness of the approach beyond this environment or on real robotic systems.

**post rebuttal**
The authors have addressed the majority of reviewers' concerns, and provided new experiments. Overall based on the reviewer's suggestions, I recommend this paper for publication.
Decision: Accept (Poster)