Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes the Latent space Safe Sets LS3 method aiming solve the vision-based control tasks safely and more efficiently. The approach is based on modeling latent probabilistic model of the world, classifying future constraint violation, which enables trajectory optimization with estimated probability of constraint violation over this latent probabilistic dynamical model.

A key assumption is that a dataset containing both colliding and free transitions in the environment is provided. Learning proceeds by first training the world dynamical model, variational decoder, encoder, safe set classified, value function goal indicator, and constraint estimator, and then collecting data during execution and updating the model with new data. 

Simulation experiments on 3 vision-based control tasks and real-world experiment with a cable-routing task on a daVinci robot are presented.


strengths_and_weaknesses: Strengths:
+ relies on powerful latent-space world-state representation with VAE to encode uncertainty
+ a unified framework for predicting the world state and incorporating constraints in an efficient manner (through reduced latent space)
+ use of trajectory selection/optimization with constraints can give more confidence than e.g. a learned policy
+ the simulation studies are interesting and a nice illustration of the method

Weaknesses:
- details of the CEM optimizer are missing (even in the references these details are hard to find), i.e. would be interesting to say how these probabilities are estimated, if it is from samples how many samples are needed, is there a way to come up with a measure of trust in this estimation since this is what the original objective of the paper is...  
- the sample-based optimization relies on stochastic rollouts and it is not super clear what the role of uncertainty is vs samples to perturb the actions in CEM
- shouldn't the recurrent dynamics block also encode uncertainty?
summary_of_recommendation: The paper presents a principled way to incorporate safety, and while some key details about how probability distributions are approximated is missing, it is an important and promising contribution. 
issues: Please address the list of the weaknesses above, as much as possible.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a framework for safe learning with visual inputs for iterative tasks. The paper combines existing work on learning latent dynamics from visual input and learning-based model predictive control. In particular, the proposed method learns a safe set in the latent space as a terminal constraint for the MPC algorithm. 
strengths_and_weaknesses: Strengths:
- The paper considers an important problem of safe learning with visual inputs, which has great potential impact. 
- The figures in the paper are informative and helpful for readers to understand the paper. 
- I appreciate the level of details of the algorithm implementation provided in the appendices.
- The authors conduct thorough sensitivity study on the safety parameters for the proposed algorithm as well as the baselines. 

Weaknesses:
- In terms of novelty, this work also seems to mostly a combination of learning latent visual dynamics [2] and SAVED [18].
- Although the primary claims of the paper seems to be "ensure safety" (see line 53-56), this is unfortunately not sufficiently supported by the rest of the paper, both theoretically and experimentally. Personally I don't think the authors are required to address safety *both* theoretically and experimentally. However, it would be good if the authors can justify either one of them during the rebuttal period or in the revision. 
  - In the theoretical point of view, learning-based MPC [19] only provides safety and feasibility guarantees when 1) the ground truth dynamics is used within the MPC and 2) the dynamics is deterministic. Both of them don't hold in this paper, as learned stochastic dynamics are used. Even if we assume that the learned dynamics is accurate, the fact that it is stochastic already breaks a lot of the premises of learning-based MPC. As an example, the latent space safe set actually does not "ensures that the agent can plan back to regions in which it is conﬁdent in task completion" (as the authors claimed in line 53), because having one success example for a stochastic MDP doesn't ensure that we can return to the safe region even if we apply the same actions. 
  - Algorithmically, I see that the proposed method makes some compromises because stochastic learned dynamics breaks the assumptions in learning-based MPC. These compromises can provide us a practical algorithm (in the sense that we can run them), but could be even more harmful to safety. For example, the authors solve the MPC problem, which is a constrained optimization problem, using CEM. However, it is possible that all particles violate the constraints of terminating in the safe set, which is noted by the authors in the appendices on choosing $\delta_S$ (another cause of this is the broken of recursive feasibility due to stochastic dynamics). To mitigate this, the authors proposed to online adjust $\delta_S$ such that there exists a constraint satisfying particle. The experimental results seem to show that the performance with and without the safe set are quite similar except that the policy starts with a lower reward without the safe set constraint. I hope the authors can further clarify why this performance gap at the beginning of training is important.
  - In terms of the experimental results, the proposed method does not show significant improvement over baselines in terms of constraint violations. Much to my surprise, SACfD, which is not even a safe RL approach, does almost as well as the proposed method in the point mass environment, only 1 std worse than the proposed method in the reacher task, and significantly better in the sequential pushing task. Although the authors explained that this is partially due to that SACfD also has worse performance. I still find it a little unsatisfying as SACfD is performing almost as well as the proposed algorithm at the end of learning in terms of the reward. 
- I find it a little hard to interpret the results from the cable routing task. I hope the authors can further explain why this task is challenging. The robot is controlled by $\Delta x$ and $\Delta y$ in the end-effector space and the robot is moving quite slowly according to the videos. It seems to me that one can almost achieve the task by replaying successful demonstrations, or just following an end-effector trajectory given by some simple heuristics, e.g. moving straight enough then take a turn. This makes me wonder why RL is a good approach to solve this problem. I do like to know the authors' thoughts on this. 
- I also hope that the authors can clarify how function $f_S$ is learned. In line 186-187, the authors seem to define $f_S$ as a solution to a fixed point problem. However, in the appendices, the authors seem to just treat learning $f_S$ as a classification problem. It would be great if the authors can provide more information on this. 
summary_of_recommendation: The paper seeks to solve the problem of safe RL with visual input, which is an important research question. However, the paper seems to be more of a combination of existing work in learning visual dynamics and learning-based MPC. The combination also breaks some of the sound theoretical properties of learning-based MPC, as I mentioned in the "Strengths and Weaknesses" part. The experimental results at this point were not sufficient to justify the contribution of safe learning as standard RL algorithm (SAC) seems to do as well as the proposed method in terms of safety and reward. I hope that the authors can provide more justifications on the theoretical soundness of the paper and/or the experimental results. I am happy to update my reviews during later stages. 
issues: See the "Strengths and Weaknesses" part.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an approach for exploration in image-based tasks in the presence of unsafe regions, i.e. when exploration is constrained. The algorithm, LS3, learns a latent space which enforces reconstruction, dynamics, a value function, and most novelty and crucially a safe set. This safe set is constructed from demonstration states which previously resulted in success and is used to guide its exploration towards safe regions. LS3 is demonstrated on several robotic tasks to demonstrate it learns faster than other state of the art methods and at times to higher performance.
strengths_and_weaknesses: Strengths:
- The paper is clear and interesting
- The choices for network architecture are clear and reasonable
- The approach learns quickly and safely over a range of tasks and on robot experiments

Weaknesses:
- It is not clear how much exploration is limited compared to other approaches
- The constraint violation rate of other algorithms is surprising given it is a focus of the approach and should be investigated more

summary_of_recommendation: The algorithm is well motivated and clearly described in this work. The results are promising and demonstrated on a number of tasks, including an on robot experiment. A few results require deeper investigation, particularly in how this approach limits exploration in the limit and surprising results on safety of exploration during learning.
issues: There are two issues that I believe are important to investigate to fully understand the performance of LS3:
- One potential drawback of this method is the converged performance due to limitations in exploration to safe regions. Figure 4 cuts off training at 250 trajectories. It should be shown on a log scale to a larger number of trajectories. 
- Table 2 shows that LS3 actually violates constraints more often than other methods in some domains. As LS3 is particularly created to allow safe exploration, this is a surprising result. However it is only briefly discussed and conjectured about. Though the hypothesis seems reasonable, it should be verified and thoroughly understood since it is somewhat antithetical to the purpose of this work.

Minor notes:
- An interesting avenue that is not discussed is how the approach performs with more complex dynamics, such as systems with velocity and dynamical constraints. These are common systems that require safety and staying in safe regions. This however complicates the state space construction and may make the learning much more difficult. It would be interesting to understand the potential of this method in those settings.
- Figure 2 should say where each of the different learned sections are discussed.
- The on robot task is appreciated, but I believe the result would be much stronger with a pushing task.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a visuomotor reinforcement learning algorithm (LS3) that attempts to guarantee safe learning and minimize constraint violations during the exploration process. It achieves this by leveraging an offline dataset containing some examples of task successes and some examples of constraint violations. A latent representation is first pre-trained (using a VAE) on the offline demonstrations, and then the algorithm uses model-predictive control (MPC) to learn several classifiers, a value function, and a latent dynamics model, to try and guarantee safe model-based exploration rollouts during the training process. The algorithm is shown to be effective across 3 simulated tasks and a physical cable routing task.
strengths_and_weaknesses: Strengths

The paper is mostly well-written (apart from a few clarity issues), and the problem it addresses is well-motivated. The margin by which the proposed algorithm (LS3) outperforms baselines in the physical robot experiments is also very impressive. The video overview is also very useful.

Weaknesses

One of the biggest issues is that this paper appears to be extremely similar to another algorithm (SAVED, reference 18 in the paper). The methodology of using safe sets during MPC for safe exploration was introduced in that paper. The main novelty in this paper appears to be extending that algorithm to visuomotor tasks. This is not necessarily a bad thing, but from a technical standpoint, the main difference appears to be pre-training a VAE on an offline dataset, and using the encoder to embed all image states into latent states -- all other components are then trained on latent states instead of image states. 

If there are more important technical ingredients that are different from SAVED, they should be stated more explicitly, and ablations should be included to show their importance. Adding SAVED as an additional baseline (and providing it with access to to ground-truth low-dimensional observations) could be useful to provide an upper bound on performance.

There are also some clarity issues. In the title, the paper mentions two keywords - "long-horizon" and "iterative", but what they refer to is not very clear in the paper text. In section 5.3, it is stated that the horizon for these tasks are 50-150 timesteps - this does not appear to be long-horizon, as compared to prior work. Some point of comparison for why these tasks are long-horizon is necessary, especially when considering prior work in safe reinforcement learning. Similarly, the paper mentions "iterative learning control" where an agent usually tracks a reference trajectory, but what is an "iterative task"?

More detailed comments and questions follow:
- Is it necessary to update the VAE using online data? If not, why does it suffice to learn it purely on the offline dataset - is it because the amount of variation in start and goal states is small?
- lines 133-136 were confusing - make sure to state that learning approximations to R and C is okay, you just assume that you can't evaluate the R and C functions directly on future predicted states (hence the need for learned approximations)
- The definitions in lines 176-179 are pretty convoluted - it seems like an equivalent way to say the same thing is that the set $S^j$ contains the states visited by all successful trajectories in iteration j. The next lines futher complicate things - what's the difference between $D_{success}$ and $S^j$? Is $D_{success}$ just the union over all $S^j$?
- SACfD + RRL appears to be a very strong baseline, matching asymptotic performance on the two harder simulated tasks (reacher and sequential pushing). This is why, at first glance, there appeared to be a large discrepancy between Table 1 and Figure 4. The task success rate was much larger for LS3 than SACfd+RRL, but as previously mentioned, SACfD+RRL appears to match or exceed the return of LS3 on 2 of the 3 tasks. Lines 275-276 suggest that the table is reported over the entire training process, as opposed to the final trained policy - this is understandable for the constraint violation metric, but unconventional for task success rate. The table caption should be made more clear - perhaps change the bolded portion to read "Task Success Rate over all Training Episodes".
- Are lines 187-188 defining the relaxed Latent Safe Set via the normalized classifier logits (e.g. the level $\delta_S$ = 0.5 would correspond to normal classification)? 
- Should equation 2 have a discount factor in front of $f_G$ (since the value function appears to correspond to discounted returns)? Furthermore, the preliminaries section suggests that negative rewards are used ({-1, 0}), but the classifier defines sparse positive rewards ({0, 1}) - which one is used?
- Equation 4 and 5 both use j, but in seemingly different contexts
- line 218 - typo ("how to the task")
summary_of_recommendation: While this paper presents interesting empirical results, it is hard to assess the value of the contribution without understanding more about how it differs to its nearest neighbor (SAVED, reference 18), and whether the proposed algorithms amounts to running SAVED with a pre-trained latent representation.

(Post-Discussion). The authors have addressed most of my comments, and I have changed my score accordingly
issues: Comment on the differences between SAVED and the proposed algorithm, especially if there are changes in this algorithm that are non-trivial, or that are critical for success in visuomotor settings compared to settings with low-dimensional ground-truth observations. If possible, include a baseline comparison against SAVED (as an upper bound on performance with image observations), and additional ablations of the method that bring it closer to SAVED -- these comparisons would be helpful in understanding the novelty of the work.
review_title: null

Meta-review: The paper addresses an important research challenge. Safety when learning from visual input is an important unsolved problem which is highly relevant for the robotics and machine learning communities. The paper is well written and technically sound. The paper mostly relies on combining existing techniques to achieve the desired end result. 

Pros:
- Paper addresses an important hard problem
- Real robot experiments

Cons:
- To correspond to the claims, safety should be explicitly shown in theory or/and experimentally. Please, see comments from reviewers phDr, pzwD
- Experiments do not show an advantage over previous (non-safety targeted) methods. The paper should discuss this in more detail and potentially provide more experimental results to yield more insight into the benefits of the approach.

Update: The safety aspect has been clarified during the discussion period and the paper has been updated. The proposed approach does not provide theoretical safety guarantees but more a practical approach that uses a safety set to make exploration less susceptible to failures. There is experimental evidence of improved sample efficiency compared to comparison methods.
Decision: Accept (Poster)