LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
Authors: Albert Wilcox
Ashwin Balakrishna
Brijen Thananjeyan
Joseph E. Gonzalez
Ken Goldberg
Venue: CoRL2021 Poster (2021)
Keywords: Reinforcement Learning
Imitation Learning
Safety

Abstract: Reinforcement learning (RL) has shown impressive success in exploring high-dimensional environments to learn complex tasks, but can often exhibit unsafe behaviors and require extensive environment interaction when exploration is unconstrained. A promising strategy for learning in dynamically uncertain environments is requiring that the agent can robustly return to learned Safe Sets, where task success (and therefore safety) can be guaranteed. While this approach has been successful in low-dimensions, enforcing this constraint in environments with visual observation spaces is exceedingly challenging. We present a novel continuous representation for Safe Sets framed as a binary classification problem in a learned latent space, which flexibly scales to high-dimensional image observations. We then present a new algorithm, Latent Space Safe Sets (LS3), which uses this representation for long-horizon control. We evaluate LS3 on 4 domains, including a challenging sequential pushing task in simulation and a physical cable routing task. We find that LS3 can use prior task successes to restrict exploration and learn more efficiently than prior algorithms while satisfying constraints. See https://tinyurl.com/latent-safe-sets for supplementary material.

LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
Albert Wilcox", Ashwin Balakrishna", Brijen Thananjeyan,
Joseph E. Gonzalez, Ken Goldberg
cqual contribution
(albertwilcox, ashwin bawaakaishaaa oberkeley edu
Abstract: Reinforcement learning (RL) has shown impressive success in explor- ing high-dimensional environments learn complex tasks, but can often exhibit unsafe behaviors and require extensive environment interaction when exploration is unconstrained. A promising strategy for learning in dynamically uncertain envi- ronments is requiring that the agent can robustly return to learned safe sets, where task success (and therefore safety) can be guaranteed. While this approach has been successful in low-dimensions, enforcing this constraint in environments with visual observations exceedingly challenging. We present novel continuous representation for safe sets by framing it as binary classification problem in a learned latent space, which flexibly scales image observations. We then present a new algorithm, Latent Space Safe Sets (LS²), which uses this representation for long-horize tasks with sparse rewards. We evaluate LS3 on 4 domains, in- cluding challenging sequential pushing task in simulation and physical cable routing task. We find that LS can use prior task successes to restrict exploration and learn more efficiently than prior algorithms while satisfying constraints. See https:/ /tt:::///ii/llee cwwccommmccaaoc tomm//atent-eet as for code and supplementary material.
Keywords: Reinforcement Learning, Imitation Learning, Safety
Introduction
Visual planning over learned forward dynamics models is a popular area of research in robotic control from images [1, 2, 3, 4, 5, 6, 7), as it enables closed-loop, model-based control for tasks where the state of the system not directly observable or difficult to analytically model, such as the configuration of a sheet of fabric or segment of cable. These methods learn predictive models over either images or a learned latent space, which can then be used to plan actions which maximize some task reward. While these approaches have significant promise, there are several open challenges in learning policies from visual observations. First, reward specification particularly challenging for visuomotor control tasks, because high-dimensional observations often do not expose the necessary features required to design dense, informative reward functions [8]. especially for Long-horize tasks. Second, while many prior reinforcement learning methods have been successfully applied to image-based control tasks [9, 10, 11, 12, 13], learning policies from image observations often requires extensive exploration due the high dimensionality of the observation space and the difficulties in reward specification, making safe and efficient learning exceedingly challenging.
One promising strategy for efficiently learning safe control policies is to leam safe set [14, 15], which captures the set of states from which the agent is known to behave safely, which often reformulated as the set of states where it has previously completed the task. When used to restrict exploration, this safe set can be used to enable highly efficient and safe learning [14, 16, 17), as exploration is restricted to states in which the agent is confident in task success. However, while these safe sets can give rise to algorithms with a number of appealing theoretical properties such as convergence to goal set, constraint satisfaction, and iterative improvement [14, 16, 18), using them for controller design for practical problems requires developing continuous approximations at the expense of maintaining theoretical guarantees [17]. This choice of continuous approximation is key element in determining the applications to which these safe sets can be used for control.
University of California, Berkeley.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Sumple Trajectories, Chock Constraints, Sort by Value
1. Observe Image
Encode Image fane (2
Candidate Accion
8t
Place
Spuce Set Bla)
Sado Plans
Unaade Plun
Labert Spece Olastarl
Figure I: Latent Space Safe Sets (LS"): At time 1, LS" observes an image of the environment. The image is first encoded to latent vector ~ (2)))))))). Then, LS uses sampling based optimization procedure to optimize H H-length action sequences by sampling H-length latent trajectories over the learned latent dynamics model fayn. For each sampled trajectory, LS checks whether latent space obstacles are avoided and if the terminal state in the trajectory falls in the latent space safe set. The terminal state constraint encourages the algorithm to maintain plans back to regions of safety and task confidence, but still enables exploration. For feasible trajectories, the sum of rewards and value of the terminal executes the first action the optimized plan and then performs this procedure again the next timestep.
computed and used for sorting. I.
Prior works have presented approaches which collect discrete safe set of states from previously successful trajectories and represent a continuous relaxation of this set by constructing a convex hull of these states [14] or via kernel density estimation with tophat kernel function [17]. While these approaches have been successful for control tasks with low-dimensional states, extending them to nigh-diimessionl observations presents two key challenges: (1) scalability: these prior methods cannot be efficiently applied when the number of observations in prior successful trajectories is large, as querying safe set inclusion scales linearly with number of samples it contains and (2) representation capacity: both of these prior approaches do not scale well to high dimensional observations and are limited in the space of continuous sets that they can efficiently represent. Applying these ideas to visuomotor control is even more challenging, since images do not directly expose details about the system state or dynamies that are typically needed for formal controller analysis [14, 16, 19].
This work makes several contributions. First, we introduce scalable continuous approximation method which makes it possible to leverage safe sets for visuomotor policy learning. The key idea is to reframe the safe set approximation as binary classification problem in learned latent space, where the objective is to distinguish states from successful trajectories from those in unsuccessful trajectories. Second, we present Latent Space Safe Sets (LS*), a model-based RL algorithm which encourages the agent maintain plans back to regions in which it is confident in task completion, learning in high dimensional spaces. This constraint makes it possible to define a control strategy to (1) improve safely by encouraging consistent task completion (and therefore avoid unsafe behavior) and (2) learn efficiently since the agent only explores promising states in the immediate neighborhood of those in which it was previously successful. Third, we present simulation experiments on visuomotor control tasks which suggest that L. can learn to improve upon demonstrations more safely and efficiently than prior algorithms. Fourth, we conduct physical experiments on a vision- based cable routing task which suggest that LS can learn more efficiently than prior algorithms while consistently completing the task and satisfying constraints during learning.
when
2 Related Work
2.1 Safe, Iterative Learning Control
In iterative learning control (II.C, the agent tracks a reference trajectory and uses data from controller rollouts to refine tracking performance [20]. Rosolia et al. [21], Rosolia and Borrelli [18. 14] present a new class of algorithms, known as Learning Model Predictive Control (LMPC), which are reference- free and instead iteratively improve upon the performance of an initial feasible trajectory. To achieve this, Rosolia et al. [21], Rosolia and Borrelli [18, 14] use data from controller rollouts to learn safe set and value function, with which recursive feasibility, stability, and local optimality can be guaranteed given known, deterministic nonlinear system or stochastic linear system under certain regularity assumptions. However, a core challenge with these algorithms is that they assume known system dynamics, and cannot be applied to high-dimensional control problems. Thananjeyan al. [17] extends the LMPC framework to higher dimensional settings in which system dynamics unknown and must be learned, but the visuomotor control setting introduces number of new

---

challenges as learned system dynamics, safe sets, and value functions must flexibly scale to visual inputs. Richards al. [15] designs expressive safe sets for fixed policies using neural network classifiers with Lyapunov constraints. In contrast, LS$ constructs a safe set for an improving policy by optimizing a task cost function instead of uniformly expanding across the state space.
2.2 Model Based Reinforcement Learning
There has been significant recent progress in algorithms which combine ideas from model-based planning and control with deep learning [22, 23, 24, 25, 26, 27]. These algorithms gaining popularity in the robotics community as they enable leaning complex policies from data while maintaining some of the sample efficiency and safety benefits of classical model- control techniques. However, these algorithms typically require hand-engineered dense cost functions for task specification, which often be diffieult to provide, especially in high-dimensional spaces. This motivates leveraging demonstrations (possibly suboptimal) to provide an initial signal regarding desirable agent behavior. There has been some prior work leveraging demonstrations in model- based algorithms such as Quinlan and Khatib [28] and Ichnowski et al. [29], which use model- based control with known dynamics to refine initially suboptimal motion plans, and Fu al. [24], which uses demonstrations to seed a learned dynamics model for fast online adaptation using iLQR [24]. Thananjeyan et al. [17], Zhu et al. [30] present ILC algorithms which rapidly improve upon suboptimal demonstrations when system dynamics are unknown. However, these algorithms either require knowledge of system dynamics [28, 29] or are limited low-dimensional state spaces [24, 17, 30] and cannot be flexibly applied to visuomotor control tasks.
2.3 Reinforcement Learning from Pixels
Reinforcement learning and model-based planning from visual observations is gaining significant recent interest as RGB images provide an easily available observation space for robot learning [1, 31]. Recent work has proposed a number of model-free and model-based algorithms that have in laboratory settings in a number of robotic tasks when learning from visual observations [32, 33, 10, 34, 12, 13, 1, 35, 31]. However, two core issues that prevent application of many RL algorithms in practice, inefficient exploration and safety, are significantly exacerbated when learning from high-dimensional visual observations in which the space of possible behaviors is very large and the features required to determine whether the robot is safe are not readily exposed. There has been significant prior work on addressing inefficiencies in exploration for visuomotor control such as latent space planning [2, 31, 35] and goal-conditioned reinforcement learning [13, 10]. However, safe reinforcement learning for visuomotor tasks has received substantially less attention. Thananjeyan et al. [36] and Kahn et al. [37] present reinforcement learning algorithms which estimate the likelihood of constraint violations to avoid them [36] or reduce the robot's velocity [37]. Unlike these algorithms, which focus on presenting methods to avoid violating user-specified constraints, LS3 additionally provides consistent task completion during learning by limiting exploration to the neighborhood of prior task successes. This difference makes LS3 less susceptible to the challenges of unconstrained exploration present in standard model-free reinforcement learning algorithms.
success
3 Problem Statement
We consider an agent interacting in a finite horizon goal-conditioned Markov Decision Processes (MDP) which can be described with the tuple M (S, (S,9,44 A, P().)))) T)). S and A are the state and action spaces, P : S x A x S distribution over subsequent states, R S x A x S - R the reward function, μ the initial state distribution (su ~ μ)), and T is the time horizon. In this work, the agent is only provided with RGB image observations E W Hx3 = S, where W and H are the image width and height in pixels, respectively. We consider iterative tasks, where the agent must reach fixed goal set C S as efficiently as possible and the support of p is small. While there are number of possible choices of reward functions that would encourage fast convergence G. providing shaped reward functions can be exceedingly challenging, especially when learning from high dimensional observations. Thus, as in Thananjeyan et al. [17], we consider a sparse reward function that only indicates task completion: R(s, a, s') = 0 if s' E G and -1 otherwise. To incorporate constraints, we augment M with an extra constraint indicator function C : + {0, 1} which indicates whether a state satisfies user-spe state-space constraints, such as avoiding known obstacles. This is consistent with the modified CMDP formulation used in [36]. We assume that R and C can be evaluated on the current state of the system, but may be approximated using prior data for use during planning. We make this assumption because
+ [0, 1] maps a state and action to a probability
3

---

Observation
Enodder
St
Lakel Space
Vadeo
Pontics fe
VTA)
Sand
Agent
Latrat Bracaio
Jaso
Oburdde
Figure 2: L.S Learned Models: I.S learns low-dimensional latent representation of image-observations (Section 4.1) l and learns a dynamics model, value function, reward function, constraint classifier, and safe set for constrained planning and task cask-complotion driven exploration in this learned latent space. These models are then used for model-based planning to maximize the total value of predicted latent states (Section 4.3) while enforcing the safe (Section 4.2) and user-specified constraints (Section 4.3).
in practice we plan over predicted future states, which may not predicted sufficiently high fidelity to expose the necessary information to directly evaluate R and C during planning.
Given policy т Furthermore, we define (s)(s as the probability of future constraint violation (within time horizon ?") under policy from state s. The objective is maximize the expected return R while maintaining constraint violation probability lower than dc. This can be written formally as follows:
.A, we define its expected total return in M as Rº JR. [], R(s,, at)
arg max R Rooop (s)) S 6c}
We assume that the agent is provided with an offline dataset D of transitions in the environment of which some subset Deonstrait C D are constraint violating and some subset Ducccess D appear in successful demonstrations from suboptimal supervisor. As in [36], Deomstraitt contains examples of constraint violating behaviors (for example from prior runs of different policies or collected under human supervision) so that the agent can learn about states which violate user-specified constraints.
Latent Space Safe Sets (LS$)
We describe how LS uses demonstrations and online interaction safely learn iteratively improving policies. Section 4.1 describes how we learn low-dimensional latent representation of image obser- vations facilitate efficient model-based planning. To enable this planning, we learn a probabilistic forward dynamics model as in [26] in the learned latent space and models to estimate whether plans will likely complete the task (Section 4.2) and to estimate future rewards and constraint violations (Section 4.3) from predicted trajectories. In Section 4.4, we discuss how these components are synthesized L.$3. Dataset DD is expanded using online rollouts of LS$ and used to update all latent space models (Sections 4.2 and 4.3) after every K rollouts. See Algorithm and the supplement for further details on training procedures and data collection.
Algorithm Latent Space Safe Sets (LS
Require: offline dataset D, number of updates U
1: Train VAE encoder fene and decoder face (Section 4.1) using data from D
2: Train dynamics Sayn: safe set classifier fa(Section 4.2), and the value function V goal indicator fg. and constraint estimator fe (Section 4.3) using data from D.
3: for (1..... U do
4: for E (1....,K) K} do
5: Sample starting state so from μ..
6: for E {1,...,, T) do
7:
Choose and execute at (Section 4.4)
8:
Observe $1+1, reward rr, constraint Cy.
9:
D := (st, at St,1,7t,
10:
Update faya: V. fg. fe. and fs with data from D.
4.1
Learning a Latent Space for Planning
Learning compressed representations of images has been a popular approach vision based control to facilitate efficient algorithms for planning and control which reason about lower dimensional

---

inputs [2, 35, 6, 38, 311. To learn such representation, we train a --variational autoencoder [40] on states in D to map states to a probability distribution over d-dimensional latent space Z. The resulting encoder network [enc(z|s) is then used to sample latent vectors 2, ~ [enc(=e(ss) to train a forward dynamics model, value function, reward estimator, constraint classifier, safe set, and combine these elements to define a policy for model-basel based planning. Motivated by Laskin et al. [41], during training we augment inputs to the encoder with random cropping, which we found to be helpful in learning representations that are useful for planning. For all environments we use latent dimension of d = 32, as in [2] and found that varying d did not significantly affect performance.
4.2 Latent Safe Sets for Model-Basee Control
LS learns a binary classifier for latent states to learn a latent space safe set that represents states from which the agent has high confidence task completion based on prior experience. Because the agent can reach the goal from these states, they are safe: the agent can avoid constraint violations by simply completing the task as before. While classical algorithms use known dynamics to construct safe sets, we approximate this set using successful trajectories from prior iterations. At each iteration j. the algorithm collects K trajectories in the environment. We then define the sampled safe set at iteration j. S as the set of states from which the agent has successfully navigated to G in iterations 0 through j of training. where demonstrations trajectories are those collected at iteration O. We refer to the dataset collecting all these states as Duccess- This discrete set is diffieult to plan with continuous- vaalued state distributions so we leverage data from D success (data in the sampled safe set), data from D \ D Puccess (data outside the sampled safe set), and the learned encoder from Section 4.1 to learn a continuous relaxation of this set in latent space (the latent safe set). We train a neural network with a binary cross-entropy loss to learn a binary classifier fs(() that predicts the probability of state 8 with encoding z being in St. To mitigate the negative bias that appears when trajectories that start in safe regions fail, we utilize the intuition that if state se+1 S' then it is likely that s, is also safe. To do this, rather than just predict 1gs, we train fs with a recursive objective to predict max( 1gı, 7h (s)+ The relaxed latent safe set parameterized by the superlevel sets of fs. where the level 0g is adaptively set during execution: Sy = (=+) 2 6z}.
4.3 Reward and Constraint Estimation
In this work, we define rewards based on whether the agent has reached state E G. but we need rewards that are defined on predictions from the dynamics, which may not correspond to valid real images. To address this, we train classifier fg Z + {0. 1} to map the encoding of a state to whether the state is contained in G using terminal states in Dencesss (which are known to be in G) and other states D. However, in the temporally-extendae prediction alone is insufficient because rewards only indicate whether the agent is in the goal set, and thus provide no signal on task progress unless the agent can plan to the goal set. To address this, as in prior MPC-literature [17, 16, 14, 8]. we train secrsively-definee supplement). Similar to the reward function, we use the encoder (Section 4.1) to train a classifier fc Z + [] with data of constraint violating states from Deonstraint and the constraint satisfying states in D \ Peomstrain to map the encoding of a state to the probability of constraint violation.
sparse reward tasks we consider, reward
value function (details in the
4.4 Model-Based Planning with LS$
LS3 aims to maximize total rewards attained in the environment while limiting constraint violation probability within some threshold åc (equation 1). We optimize an approximation of this objective over an H-step receding horizon with podel-predictive control. Precisely, LS3 solves the following optimization problem to generate an action to execute at timestep I:
arg max
E Exat-
Σ fo(e)4s) V (24ta)
(2)
01.0- # 1EA
s.t.
2 fono (Zcl(8))
(3)
Sk+1 ~ dom(2k+1/22 ak) Wk E (l...., 1+ H 13
P (( E E 2 1 dz
55)
(2 E 2x) < dc Vi E so..... H - 1}
(6)
5

---

Pointmass Navigation
Reacher
Sequential Pushing
Physical Cable Routing
Start
End
Figure 3: Experimental Domains: LS evaluated on 3 oonghhorizon, image- based, simulation environments: a visual navigation domain where the goal is to navigate the blue point mass to the right goal set while avoiding the red obstacle, 2 degree of freedom reacher arm where the task is to navigate around red obstacle to reach the yellow goal set, and sequential pushing task where the robot must push each of 3 blocks forward target displacement from left to right. We also evaluate LS on a physical, cable-routing task on da Vinci Surgical Robot, where the goal is guide a red cable to green target without the cable or robot arm colliding with the blue obstacle. This requires learning visual dynamics, because the agent must model how the rest of the cable will deform during manipulation to avoid collisions with the obstacle.
In this problem, the expectations and probabilities are taken with respect to the learned, probabilistic dynamics model [2yn(2++1 12441222 dz). The optimization problem is solved approximately using the cross- entropy method (CEM) [42] which is a popular optimizer model-based RL [43, 17, 16, 44, 36].
The objective function is the expected sum of future rewards if the agent executes 0c:++-11 and then subsequently executes π (equation 2). First, the current state S encoded 2 (equation 3). Then, for a candidate sequence of actions d 1- 1,, an H-step latent trajectory { F++ is sampled from the learned dynamics Jayn (equation 4). I.S constrains exploration using two chance constraints: (1) the terminal latent state in the plan must fall in the safe set (equation 5) and (2) all latent states in the trajectory must satisfy user-specified state-space constraints (equation 6). 2c is the set of all latent states such that the corresponding observation is constraint violating. The optimizer estimates constraint satisfaction probabilities for a candidate action sequence by simulating it repeatedly over Sayn. The first chance constraint ensures the agent maintains the ability to return to safe states where it knows how to do the task within steps necessary. Because the agent replans at each timestep, the agent need not return the safe set: during training, the safe set expands, enabling further exploration. In practice, we set ōg for the safe set classifier fs adaptively as described in the supplement. The second chance constraint encourages constraint violation probability of more than dg. After solving the optimization problem, the agent executes the first action in the plan: or (2+) de where a, is the first element of (:++ observes state, and replans.
5 Experiments
We evaluate L.S on 3 robotic control tasks in simulation and physical cable routing task on the da Vinei Research Kit (dVRK) [45]. Safe RL of particular interest for surgical robots such as the dVRK due to its delicate structure, motivating safety, and relatively imprecise controls [17, 46], motivating closed-loop control. We study whether LS$ can learn more safely and efficiently than algorithms that do not structure exploration based on prior task successes.
5.1 Comparisons
We evaluate L in comparison prior algorithms that behavior clone suboptimal demonstrations before exploring online (SACID) [47] or leverage offline reinforcement learning to learn policy using all offline data before updating the policy online (AWAC) [48]. For both of these comparisons we enforce constraints via a tuned reward penalty of À for constraint violations as in [49]. We also implement a version of SACfD with a learned recovery policy (SACID+RRL) using the Recovery RI algorithm [36] to use prior constraint violating data to try to avoid constraint violating states. We then compare L to an ablated version without the safe set constraint (just binary classification (BC) in equation (LS$ (-Safe Set)) to evaluate if the safe set promotes consistent task completion and stable learning. Finally, we compare LS to an ablated version of the safe set classifier (Section 4.2) without recursive objective, where the classifier is just trained to predict 1g, (LS' (BC SS)). See the supplement for details on hyperparameters and offline data used for LS$ and prior algorithms.
5.2 Evaluation Metrics
For each algorithm on each domain, we aggregate statistics over random seeds (10 for simulation experiments, for the physical experiment), reporting the mean and standard error across the seeds.

---

Pointbot Navigation
Reacher
Sequential Pushing
-50
-50
-60
-50-
-70-
-70
-75
-00
-80
-100
-90
-125
-100
-100
-150
50
100
150
200
50
100
150
200
250
100
200
300
400
500
Training Trajectories
Training Trajectories
Training Trajectories
Demonstrationn
SACID
AWAC
SACID RRL
LS' (BC 55)
US" (-55)
LS"
Figure 4: Simulation Experiments Results: Learning curves showing mean and standard error over 10 random seeds. We see that L learns more quickly than baselines and ablations. Although SACID and SACI+-RRL.
converge to similar reward values, LS is much more sample efficient and stable across random seeds.
We present learning curves that show the total sum reward for each training trajectory to study how efficiently LS and the comparisons learn each task. Because all tasks use the sparse task completion based rewards defined in Section 3, the total reward for a trajectory is the time to reach the goal slower convergence to G. Thus, for a task with task
set, where more negative rewards correspond horizon T, a total reward greater than -T implies successful task completion. The state is frozen in place upon constraint violation until the task horizon elapses. We also report task success and constraint satisfaction rates for LS$ and comparisons during learning to study (1) the degree to which task completion influences sample efficiency and (2) how safely different algorithms explore. LS$ collects K = 10 trajectories in between training phases on simulated tasks and K 5 in between training phases for physical tasks, while the SACíD and AWAC comparisons update their parameters after each timestep. This presents metric in terms of the amount of data collected across algorithms.
5.3 Domains
In simulation, we evaluate L on vision-based continuous control domains that are illustrated in Figure 3. We evaluate LS3 and comparisons on a constrained visual navigation task (Pointmass Navigation) where the agent navigates from a fixed start state to fixed goal set while avoiding a large central obstacle. We study this domain to gain intuition and visualize the learned value function, goal/constraint indicators, and safe set Figure 2. We then study a constrained image-based reaching task (Reacher) based on [50], where the objective is to navigate the end effector of a 2- 2-Ii planar robotic arm to a yellow goal position without the end-effector entering a red stay out zone. We then study a challenging sequential image-basel robotic pushing domain (Sequential Pushing), in which the objective is to push each of the blocks forward on the table without pushing them to either side and causing them fall out of the workspace. Finally, we evaluate L.$ with physical experiment on the da Vinci Research Kit (dVRK) [51] (Figure 3), where the objective is to guide the endpoint of a cable to a goal region without letting the cable end effector collide with an obstacle. The Pointmass Navigation and Reaching domains have a task horizon of T = 100 while the Sequential Pushing domain and physical experiment have task horizons of T 150 and T 50 respectively. See the supplement for more details on all domains.
mage-baasd
5.4 Simulation Results
We find that LS is able to learn more stably and efficiently than all comparisons across all simulated domains while converging to similar performance within 250 trajectories collected online (Figure 4). LS is able to consistently complete the task during learning, while the comparisons, which do not learn a safe set to structure exploration based on prior successes, exhibit much less stable learning. Additionally, in Table and Table 2, we report the task success rate and constraint violation rate of all algorithms during training. We find that LS3 achieves a significantly higher task success rate than comparisons on all tasks. We also find that LS violates constraints less often than comparisons on the Reacher task, but violates constraints more often than SACfD and SACID+RRL on the other domains. This is because SACiD and SACID+RRL spend much less time in the neighborhood of constraint violating states during training due to their lower task success rates. Because they do not efficiently learn to perform the tasks, they not violate constraints as often. We find that the AWAC comparison achieves very low task performance. While AWAC is designed for offline reinforcement learning, to the best of our knowledge, has not been previously evaluated on as in this paper, which we hypothesize are very challenging for it.
image-based tasks
7

---

Table I: Task Success Rate over all Training Episodes: We present the mean and standard error of training- time task completion rate over random seeds. find LS outperforms comparisons across all 3 domains, with the gap increasing for the challenging sequential pushing task.
SACED
AWAC
SACFD+RRL
LS (-SS)
LS
POINTMASS NAVIGATION
0.363 + 0.068
0.312 + 0.093
0.184 = 0.053
0.818 = 0.019
0.988 = 0.004
REACHER
0,502 + 0.072
0.255 + 0.080
0.473 = 0.056
0.736 + 0.025
0.870 - 0.024
SEQUENTIAL PUSHING
0.425 + 0.064
0.006 + 0.003
0.466 0.065
0.366 + 0.030
0.648 0.049
Table 2: Constraint Violation Rate: We report mean and standard error training raining-time constraint violation rate over 10 random seeds. LS violates constraints less than comparisons on the Reacher task, but SAC and SACID+RRL achieve lower constraint violation rales on the Navigation and Pushing tasks, likely due to spending less time in the neighborhood of constraint violating regions due to their much lower task success rates.
SACFD
AWAC
SACED+RRE
LS (-SS)
LS
POINTMASS NAVIGATION
0,006 1 0.002
0.104 0.070
0.001 0.001
0.019
0.000
0.005
0.001
REACHER
0.146 + 0.039
0.398 + 0.107
0.142 + 0.031
0.247 0.027
0.102 0.027
SEQUENTIAL PUSHING
0.033 + 0.003
0.138 + 0.028
0.054 + 0.006
0.122 + 0.031
0.107 0.016
Physical Cable Routing
Task Success Rate
Constraintiloatioon Rate
0.5
0.153
-20
0.075
-40
-50
*Trintng Trjjectoiee
40
0.- S420 SACD1
0.003
54,10 S4C1D RAL
Demonstrations
SACID
SACID RRL
LS3
present learning curves, task success rates and constraint violation rates with a mean and standard error across 3 random seeds. LS learns a more efficient policy than the demonstrator while still violating constraints less than comparisons, which are unable to learn the task.
Figure 5: Physical Cable Routing Results:
We find L has a lower success rate when the safe set constraint is removed (LS* Set)) as expected. The safe set is particularly important in the sequential pushing task, and LS$ (-Safe Set) has a much lower task completion rate than LS3 LS3 without the recursive classification objective from Section 4.2 (LSS (BC SS)) has similar performance to LS3 on the navigation environment, but learns substantially more slowly on the Reacher environment and performs significantly worse than LS on the more challenging Pushing environment as the learned safe set unable to exploit temporal structure to distinguish safe states from unsafe states. See the supplement for details on experimental parameters and offline data used for LS3 and comparisons and ablations studying the effect of the planning horizon and threshold used to define the safe set.
5.5 Physical Results
In physical experiments, compare LS3 to SACID and SACID+RRL (Figure 5) on the physical cable routing task illustrated in Figure We find L.$ quickly outperforms the suboptimal demonstrations while succeeding at the task significantly more often than both comparisons, which are unable to learn the task and also violate constraints more than LS3, We hypothesize that the difficulty of reasoning about cable collisions and deformation from images makes it challenging for prior algorithms to make sufficient task progress as they do not use prior successes to structure exploration. See the supplement for details experimental parameters and offline data used for LS and comparisons.
6 Discussion and Future Work
We present LS3, a scalable algorithm for safe and efficient policy learning for visuomotor tasks. L structures exploration by learning safe set in learned latent space, which captures the set of states from which the agent confident in task completion. I.$ then ensures that the agent can plan back to states in the safe set, encouraging consistent task completion during learning. Experiments suggest that L can safely and efficiently learn 4 visuomotor control tasks, including a challenging sequential pushing task in simulation and cable routing task physical robot. In future work, we are excited to explore further physical evaluation of LS on safety critical visuomotor control tasks and applications systems with dynamic constraints velocity acceleration.
8

---

Acknowledgments
This research performed at the AUTOLAB UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab, the Real-Time Intelligent Secure Execution (RISE) Lab, and the CITRIS "People and Robots" (CPAR) Initiative. The authors were supported in part by the Scalable Collabo- rative Human-Roo Robat Learning (SCHooL) Project, an NSF National Robotics Initiative Award, and by donations from Google and Toyota Research Institute and equipment grants from PhotoNeo, Nvidia, and Intuitive Surgical. Any opinions, findings, and conclusions or recommendations expressed in this material those of the author(s) and do necessarily reflect the views of the sponsors. We thank our colleagues who provided helpful feedback, especially Suraj Nair, Zaynah Javed and Daniel Brown. Ashwin Balakrishna was supported by NSF GRFP.
References
[1] F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 1812.0051 2018.
[2] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. Proc. Int. Conf. on Machine Learning. 2019.
[3] R. Hoque, Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Visuospatial foresight for multi-step, multi-task fabric manipulation. Proc. Robotics: Science and Systems (RSS), 2020.
[4] Lenz, R. A. Knepper, and A. Saxena. Deepmpe: Learning deep latent features for model predictive control. In Robotics: Science and Systems. Rome, Italy, 2015.
[5] S. Nair and C. Finn. Hierarchical foresight: Self-supervised learning of visual subgoal generation. Proc. Int. Conf. on Learning Representations, 2019.
tasks via
[6] S. Nair, S. Savarese, and Finn. Goal-aware prediction: Learning to model what matters. In Proceedings of the 37th International Conference on Machine Learning, pages 7207-7219, 2020.
[7] K. Pertsch, O. Rybkin, Ebert, C. Finn, D. Jayaraman, and S. Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. Proc. Advances in Neural Information Processing Systems, 2020.
[8] Tian, S. Nair, F. Ebert, Dasari, B. Eysenbach, C. Finn, and Levine. Model- visual planning with self-supervised functional distances. Proc. Int. Conf. on Learning Representations, 2021.
[9] Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications.
[10] A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. Proc. Advances in Neural Information Processing Systems, 2018
[11] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 2016.
[12] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, and Levine. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. Conf. on Robot Learning (CoRL), 2018. 8.
[13] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self-supervised reinforcement learning. Proc. Int. Conf. on Machine Learning, 2020.
[14] U. Rosolia and Borrelli, Learning model predictive control for iterative tasks. a data-driven control framework. IEEE Transactions on Automatic Control, 2018
[15] S. M. Richards, F. Berkenkamp, and A. Krause. The Yyapunov neural network: Adaptive stability certification for safe learning of dynamical systems. In Conference on Robot Learning, pages 466-476. PMLR, 2018

---

[16] B. Thananjeyan, A. Balakrishna, U. Rosolia, J. E. Gonzalez, A. Ames, and K. Goldberg. Abe-Impe: Safe sample-based learning mpe for stochastic nonlinear dynamical systems with adjustable boundary conditions, 2020.
[17] B. Thananjeyan, A. Balakrishna, U. Rosolia, F. Li, R. McAllister, J. E. Gonzalez, S. Levine, Borrelli, and K. Goldberg. Safety augmented value estimation from demonstrations (saved); Safe deep model-based rl for sparse cost robotic tasks. IEEE Robotics and Automation Letters, 5(2):3612-3619, 2020.
[18] U. Rosolia and F. Borrelli. hample-based learning model predictive control for linear uncertain systems. CoRR, 1ed/1904.6432,, 2019. URL
[19] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin. Hamilton-jacobi reachability: A brief overview and recent advances. In Conference on Decision and Control (CDC), 2017.
[20] D. A. Bristow, M. Tharayil, and A. G. Alleyne. A survey of iterative learning control. IEEE control systems magazine, 2006.
[21] U. Rosolia, X. Zhang, and F. Borrelli. A Stochastic MPC Approach with Application to Iterative Learning. 2018 IEEE Conference on Decision and Control (CDC), 2018.
[22] M. Deisenroth and C. Rasmussen. PILCO: model-based and data-efficient approach to policy search. In Proc. Int. Conf. on Machine Learning, 2011.
[23] I. Lenz, R. A. Knepper, and A. Saxena. DeepMPC: Learning deep latent features for model predictive control. In Robotics: Science and Systems, 2015.
[24] Fu, Levine, and Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2016.
[25] K. Lowrey, A. Rajeswaran, Kakade, E. Todorov, and I. Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. In Proc. Int. Conf. on Machine Learning, 2019.
[26] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Proc. Advances in Neural Information Processing Systems, 2018
[27] A. Nagabandi, G. Kahn, R. S. Fearing, and Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In Proc. IEFE Int. Conf. Robotics and Automation (ICRA), 2018.
[28] Quinlan and O. Khatib. Elastic bands: connecting path planning and control. In International Conference Robotics and Automation, pages 802-807 vol.2, 1993.
[29] J. Ichnowski, Y. Avigal, V. Satish, and K. Goldberg. Deep learning can accelerate grasp- optimized motion planning. Science Robotics, 5(48), 2020.
[30] 7. Zhu, N. Pivaroa, S. Gupta, A. Gupta, and M. Canova. 2021.
[31] M. Lippi, Poklukar, M. C. Welle, A. Varava, H. Yin, A. Marino, and Kragic. Latent space roadmap for visual action planning of deformable and rigid object manipulation. In IEEE/RSI International Conference on Intelligent Robots and Systems, 2020.
[32] A. A. Rusu, M. Večerík, T. Rothörl, N. Heess, R. Pascanu, and Hadsell, Sim-to-real robot learning from pixels with progressive nets. In Conference on Robot Learning, pages 262-270. PMLR, 2017.
[33] G. Schoettler, A. Nair, J. Luo, S. Bahl, J. A. Ojea, E. Solowjow, and S. Levine. Deep rein- forcement learning for industrial insertion tasks with visual inputs and natural rewards. Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2020.
[34] A. Singh, L. Yang, K. Hartikainen, C. Finn, and Levine. End-to-end robotic reinforcement learning without reward engincering. Proc. Robotics: Science and Systems (RSS), 2019.
10

---

[35] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: Deep struc- tured representations for model-basee reinforcement learning. International Conference on Machine Learning, pages 7444-7453. PMIR, 2019.
[36] B. Thananjeyan, A. Balakrishna, S. Nair, M. Luo, K. Srinivasan, M. Hwang. J. E. Gonzalez, J. Ibarz, C. Finn, and K. Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. NeurlPS Deep Reinforcement Learning Workshop, 2020.
[37] G. Kahn, A. Villaflor, V. Pong, P. Abbeel, and S. Levine. Uncertainty-awan reinforcement learning for collision avoidance. CoRR, 2017.
[38] A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and C. Finn. Universal planning networks. Proc. Int. Conf. on Machine Learning, 04 2018.
[39] B. lchter and M. Pavone. Robot motion planning in learned latent spaces. IEEE Robotics and Automation Letters, 4(3):2407-2414, 2019. doi: w.w.L.R...0192901111
[40] L. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerch- ner. beta-vae: Learning basic visual concepts with a constrained variational framework. Proc. Int. Conf. on Learning Representations, 2017.
[41] M. Laskin, K. Lee, A. Stooke, L. Pinto, Abbeel, and A. Srinivas. Reinforcement learning with augmented data. 2020. arXiv:2004. 14990.
[42] R. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1(2):127-190, 1999.
[43] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Proc. Advances in Neural Information Processing Systems, 2018
[44] J. Zhang. B. Cheung, C. Finn, S. Levine, and D. Jayaraman. Cautious adaptation for reinforce- ment learning in safety-critical settings. In International Conference on Machine Learning, pages 11055-1 1065. PMLR, 2020.
[45] P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio. An open- source research kit for the da Vinci surgical system. In Proc. IEFE Int. Conf. Robotics and Automation (ICRA), 2014.
[46] D. Seita, S. Krishnan, R. Fox, S. McKinley, J. Canny, and K. Goldberg. Fast and reliable autonomous surgical debridement with cable-driven robots using a two-phase calibration proce- dure. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2018.
[47] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proc. Int. Conf. on Machine Learning, 2018.
[48] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2021.
[49] C. Tessler, D. J. Mankowitz, and S. Mannor. Reward constrained policy optimization. In Proc. Int. Conf. on Learning Representattonss 2019.
[50] Y. Tassa, S. Tunyasuvunakool, Muldal, Y. Doron, Liu, S. Bohez, J. Merel, Erez,
T. Lillicrap, and N. Heess. dm-control: Software and tasks for continuous control, 2020.
[51] P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio. An open- source research kit for the da vinci@ surgical system. In 2014 IEEE international conference on robotics and automation (ICRA), pages 6434-6439, IEEE, 2014.
11


Reviews:
[{"review_id": "WxDB2SFeD0h", "reviewer": "Reviewer_5joi", "summary": "The paper presents an approach for exploration in image-based tasks in the presence of unsafe regions, i.e. when exploration is constrained. The algorithm, LS3, learns a latent space which enforces reconstruction, dynamics, a value function, and most novelty and crucially a safe set. This safe set is constructed from demonstration states which previously resulted in success and is used to guide its exploration towards safe regions. LS3 is demonstrated on several robotic tasks to demonstrate it learns faster than other state of the art methods and at times to higher performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is clear and interesting\n- The choices for network architecture are clear and reasonable\n- The approach learns quickly and safely over a range of tasks and on robot experiments\n\nWeaknesses:\n- It is not clear how much exploration is limited compared to other approaches\n- The constraint violation rate of other algorithms is surprising given it is a focus of the approach and should be investigated more\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The algorithm is well motivated and clearly described in this work. The results are promising and demonstrated on a number of tasks, including an on robot experiment. A few results require deeper investigation, particularly in how this approach limits exploration in the limit and surprising results on safety of exploration during learning.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "There are two issues that I believe are important to investigate to fully understand the performance of LS3:\n- One potential drawback of this method is the converged performance due to limitations in exploration to safe regions. Figure 4 cuts off training at 250 trajectories. It should be shown on a log scale to a larger number of trajectories. \n- Table 2 shows that LS3 actually violates constraints more often than other methods in some domains. As LS3 is particularly created to allow safe exploration, this is a surprising result. However it is only briefly discussed and conjectured about. Though the hypothesis seems reasonable, it should be verified and thoroughly understood since it is somewhat antithetical to the purpose of this work.\n\nMinor notes:\n- An interesting avenue that is not discussed is how the approach performs with more complex dynamics, such as systems with velocity and dynamical constraints. These are common systems that require safety and staying in safe regions. This however complicates the state space construction and may make the learning much more difficult. It would be interesting to understand the potential of this method in those settings.\n- Figure 2 should say where each of the different learned sections are discussed.\n- The on robot task is appreciated, but I believe the result would be much stronger with a pushing task.\n"}}, {"review_id": "UwzVAISsvpF", "reviewer": "Reviewer_aHEL", "summary": "The paper proposes the Latent space Safe Sets LS3 method aiming solve the vision-based control tasks safely and more efficiently. The approach is based on modeling latent probabilistic model of the world, classifying future constraint violation, which enables trajectory optimization with estimated probability of constraint violation over this latent probabilistic dynamical model.\n\nA key assumption is that a dataset containing both colliding and free transitions in the environment is provided. Learning proceeds by first training the world dynamical model, variational decoder, encoder, safe set classified, value function goal indicator, and constraint estimator, and then collecting data during execution and updating the model with new data. \n\nSimulation experiments on 3 vision-based control tasks and real-world experiment with a cable-routing task on a daVinci robot are presented.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ relies on powerful latent-space world-state representation with VAE to encode uncertainty\n+ a unified framework for predicting the world state and incorporating constraints in an efficient manner (through reduced latent space)\n+ use of trajectory selection/optimization with constraints can give more confidence than e.g. a learned policy\n+ the simulation studies are interesting and a nice illustration of the method\n\nWeaknesses:\n- details of the CEM optimizer are missing (even in the references these details are hard to find), i.e. would be interesting to say how these probabilities are estimated, if it is from samples how many samples are needed, is there a way to come up with a measure of trust in this estimation since this is what the original objective of the paper is...  \n- the sample-based optimization relies on stochastic rollouts and it is not super clear what the role of uncertainty is vs samples to perturb the actions in CEM\n- shouldn't the recurrent dynamics block also encode uncertainty?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a principled way to incorporate safety, and while some key details about how probability distributions are approximated is missing, it is an important and promising contribution. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please address the list of the weaknesses above, as much as possible. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "Uh4IJnF4DzL", "reviewer": "Reviewer_phDr", "summary": "The paper proposes a framework for safe learning with visual inputs for iterative tasks. The paper combines existing work on learning latent dynamics from visual input and learning-based model predictive control. In particular, the proposed method learns a safe set in the latent space as a terminal constraint for the MPC algorithm. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper considers an important problem of safe learning with visual inputs, which has great potential impact. \n- The figures in the paper are informative and helpful for readers to understand the paper. \n- I appreciate the level of details of the algorithm implementation provided in the appendices.\n- The authors conduct thorough sensitivity study on the safety parameters for the proposed algorithm as well as the baselines. \n\nWeaknesses:\n- In terms of novelty, this work also seems to mostly a combination of learning latent visual dynamics [2] and SAVED [18].\n- Although the primary claims of the paper seems to be \"ensure safety\" (see line 53-56), this is unfortunately not sufficiently supported by the rest of the paper, both theoretically and experimentally. Personally I don't think the authors are required to address safety *both* theoretically and experimentally. However, it would be good if the authors can justify either one of them during the rebuttal period or in the revision. \n  - In the theoretical point of view, learning-based MPC [19] only provides safety and feasibility guarantees when 1) the ground truth dynamics is used within the MPC and 2) the dynamics is deterministic. Both of them don't hold in this paper, as learned stochastic dynamics are used. Even if we assume that the learned dynamics is accurate, the fact that it is stochastic already breaks a lot of the premises of learning-based MPC. As an example, the latent space safe set actually does not \"ensures that the agent can plan back to regions in which it is conﬁdent in task completion\" (as the authors claimed in line 53), because having one success example for a stochastic MDP doesn't ensure that we can return to the safe region even if we apply the same actions. \n  - Algorithmically, I see that the proposed method makes some compromises because stochastic learned dynamics breaks the assumptions in learning-based MPC. These compromises can provide us a practical algorithm (in the sense that we can run them), but could be even more harmful to safety. For example, the authors solve the MPC problem, which is a constrained optimization problem, using CEM. However, it is possible that all particles violate the constraints of terminating in the safe set, which is noted by the authors in the appendices on choosing $\\delta_S$ (another cause of this is the broken of recursive feasibility due to stochastic dynamics). To mitigate this, the authors proposed to online adjust $\\delta_S$ such that there exists a constraint satisfying particle. The experimental results seem to show that the performance with and without the safe set are quite similar except that the policy starts with a lower reward without the safe set constraint. I hope the authors can further clarify why this performance gap at the beginning of training is important.\n  - In terms of the experimental results, the proposed method does not show significant improvement over baselines in terms of constraint violations. Much to my surprise, SACfD, which is not even a safe RL approach, does almost as well as the proposed method in the point mass environment, only 1 std worse than the proposed method in the reacher task, and significantly better in the sequential pushing task. Although the authors explained that this is partially due to that SACfD also has worse performance. I still find it a little unsatisfying as SACfD is performing almost as well as the proposed algorithm at the end of learning in terms of the reward. \n- I find it a little hard to interpret the results from the cable routing task. I hope the authors can further explain why this task is challenging. The robot is controlled by $\\Delta x$ and $\\Delta y$ in the end-effector space and the robot is moving quite slowly according to the videos. It seems to me that one can almost achieve the task by replaying successful demonstrations, or just following an end-effector trajectory given by some simple heuristics, e.g. moving straight enough then take a turn. This makes me wonder why RL is a good approach to solve this problem. I do like to know the authors' thoughts on this. \n- I also hope that the authors can clarify how function $f_S$ is learned. In line 186-187, the authors seem to define $f_S$ as a solution to a fixed point problem. However, in the appendices, the authors seem to just treat learning $f_S$ as a classification problem. It would be great if the authors can provide more information on this. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper seeks to solve the problem of safe RL with visual input, which is an important research question. However, the paper seems to be more of a combination of existing work in learning visual dynamics and learning-based MPC. The combination also breaks some of the sound theoretical properties of learning-based MPC, as I mentioned in the \"Strengths and Weaknesses\" part. The experimental results at this point were not sufficient to justify the contribution of safe learning as standard RL algorithm (SAC) seems to do as well as the proposed method in terms of safety and reward. I hope that the authors can provide more justifications on the theoretical soundness of the paper and/or the experimental results. I am happy to update my reviews during later stages. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See the \"Strengths and Weaknesses\" part. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "BiC4szDsVE4", "reviewer": "Reviewer_pzwD", "summary": "This paper presents a visuomotor reinforcement learning algorithm (LS3) that attempts to guarantee safe learning and minimize constraint violations during the exploration process. It achieves this by leveraging an offline dataset containing some examples of task successes and some examples of constraint violations. A latent representation is first pre-trained (using a VAE) on the offline demonstrations, and then the algorithm uses model-predictive control (MPC) to learn several classifiers, a value function, and a latent dynamics model, to try and guarantee safe model-based exploration rollouts during the training process. The algorithm is shown to be effective across 3 simulated tasks and a physical cable routing task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is mostly well-written (apart from a few clarity issues), and the problem it addresses is well-motivated. The margin by which the proposed algorithm (LS3) outperforms baselines in the physical robot experiments is also very impressive. The video overview is also very useful.\n\nWeaknesses\n\nOne of the biggest issues is that this paper appears to be extremely similar to another algorithm (SAVED, reference 18 in the paper). The methodology of using safe sets during MPC for safe exploration was introduced in that paper. The main novelty in this paper appears to be extending that algorithm to visuomotor tasks. This is not necessarily a bad thing, but from a technical standpoint, the main difference appears to be pre-training a VAE on an offline dataset, and using the encoder to embed all image states into latent states -- all other components are then trained on latent states instead of image states. \n\nIf there are more important technical ingredients that are different from SAVED, they should be stated more explicitly, and ablations should be included to show their importance. Adding SAVED as an additional baseline (and providing it with access to to ground-truth low-dimensional observations) could be useful to provide an upper bound on performance.\n\nThere are also some clarity issues. In the title, the paper mentions two keywords - \"long-horizon\" and \"iterative\", but what they refer to is not very clear in the paper text. In section 5.3, it is stated that the horizon for these tasks are 50-150 timesteps - this does not appear to be long-horizon, as compared to prior work. Some point of comparison for why these tasks are long-horizon is necessary, especially when considering prior work in safe reinforcement learning. Similarly, the paper mentions \"iterative learning control\" where an agent usually tracks a reference trajectory, but what is an \"iterative task\"?\n\nMore detailed comments and questions follow:\n- Is it necessary to update the VAE using online data? If not, why does it suffice to learn it purely on the offline dataset - is it because the amount of variation in start and goal states is small?\n- lines 133-136 were confusing - make sure to state that learning approximations to R and C is okay, you just assume that you can't evaluate the R and C functions directly on future predicted states (hence the need for learned approximations)\n- The definitions in lines 176-179 are pretty convoluted - it seems like an equivalent way to say the same thing is that the set $S^j$ contains the states visited by all successful trajectories in iteration j. The next lines futher complicate things - what's the difference between $D_{success}$ and $S^j$? Is $D_{success}$ just the union over all $S^j$?\n- SACfD + RRL appears to be a very strong baseline, matching asymptotic performance on the two harder simulated tasks (reacher and sequential pushing). This is why, at first glance, there appeared to be a large discrepancy between Table 1 and Figure 4. The task success rate was much larger for LS3 than SACfd+RRL, but as previously mentioned, SACfD+RRL appears to match or exceed the return of LS3 on 2 of the 3 tasks. Lines 275-276 suggest that the table is reported over the entire training process, as opposed to the final trained policy - this is understandable for the constraint violation metric, but unconventional for task success rate. The table caption should be made more clear - perhaps change the bolded portion to read \"Task Success Rate over all Training Episodes\".\n- Are lines 187-188 defining the relaxed Latent Safe Set via the normalized classifier logits (e.g. the level $\\delta_S$ = 0.5 would correspond to normal classification)? \n- Should equation 2 have a discount factor in front of $f_G$ (since the value function appears to correspond to discounted returns)? Furthermore, the preliminaries section suggests that negative rewards are used ({-1, 0}), but the classifier defines sparse positive rewards ({0, 1}) - which one is used?\n- Equation 4 and 5 both use j, but in seemingly different contexts\n- line 218 - typo (\"how to the task\")", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While this paper presents interesting empirical results, it is hard to assess the value of the contribution without understanding more about how it differs to its nearest neighbor (SAVED, reference 18), and whether the proposed algorithms amounts to running SAVED with a pre-trained latent representation.\n\n(Post-Discussion). The authors have addressed most of my comments, and I have changed my score accordingly", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Comment on the differences between SAVED and the proposed algorithm, especially if there are changes in this algorithm that are non-trivial, or that are critical for success in visuomotor settings compared to settings with low-dimensional ground-truth observations. If possible, include a baseline comparison against SAVED (as an upper bound on performance with image observations), and additional ablations of the method that bring it closer to SAVED -- these comparisons would be helpful in understanding the novelty of the work."}}]

Author rebuttal:

Meta-review: The paper addresses an important research challenge. Safety when learning from visual input is an important unsolved problem which is highly relevant for the robotics and machine learning communities. The paper is well written and technically sound. The paper mostly relies on combining existing techniques to achieve the desired end result. 

Pros:
- Paper addresses an important hard problem
- Real robot experiments

Cons:
- To correspond to the claims, safety should be explicitly shown in theory or/and experimentally. Please, see comments from reviewers phDr, pzwD
- Experiments do not show an advantage over previous (non-safety targeted) methods. The paper should discuss this in more detail and potentially provide more experimental results to yield more insight into the benefits of the approach.

Update: The safety aspect has been clarified during the discussion period and the paper has been updated. The proposed approach does not provide theoretical safety guarantees but more a practical approach that uses a safety set to make exploration less susceptible to failures. There is experimental evidence of improved sample efficiency compared to comparison methods.
Decision: Accept (Poster)