Safe Policy Learning for Continuous Control

Abstract: We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that keep the agent in desirable situations, both during training and at convergence. We formulate these problems as {\em constrained} Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them. Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the selected action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints. Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Moreover, our action-projection algorithm often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with the state-of-the-art baselines on several simulated (MuJoCo) tasks, as well as a real-world robot obstacle-avoidance problem, demonstrating their effectiveness in terms of balancing performance and constraint satisfaction.

Reviews and author responses:

Yes, editing section 3.1 is in your best interest as it was the hardest section for me to follow in an otherwise well argued paper. 

Line Search:

Okay, that sounds fair. Thank you for the clarification. Maybe make this clear somewhere? Appendix is also fine as these are finer details and you are already running out of page limit and I feel this subtlety useful information for the community as well. 

HalfCheetah:

I think there is some miscommunication, from your explanation it sounds like you are talking about joint velocities and not the velocity of the Cheetah as a whole. I might have misread this then. 
Yes, joint velocity as the reference you cited proposes is perfectly reasonable constraint.  From  your description in the paper it sounds more like the restriction is on the velocity of the cheetah as a whole, which I find  counterintuitive to the task (run as fast as possible). If it joint velocities, can you add this ? In which case it is perfectly reasonable to expect torque and velocity constraint to achieve similar performance in the final result. 
If it is not joint velocities , I am afraid we have to agree to disagree on this one and I wouldn't hold this against you. 

Thank you for your additional comments/feedback. We have incorporated the changes you suggested in your initial review, except those regarding rewriting Sections 2 and 3.1, in the new version of the paper (and we will fix the page limit once everything is updated). We will improve the readability of these two sections and make them more self-contained in the final version of the paper. Below is our response to your two new comments. 


CPO and Linesearch

We agree with the reviewer that (backtracking) linesearch can help with enforcing the constraints and improving the balance between constraint satisfaction and performance. Yet, unless we have access to the true gradient in SGD-based policy gradient (PG) methods, such as TRPO, CPO, and our Lyapunov-based algorithms, linesearch still cannot guarantee constraint satisfaction. This is why CPO, a TRPO-based algorithm, in addition to linesearch, uses other techniques, such as safe-guard policy update and cost shaping (see Section 6.2 and 6.3 of the CPO paper https://arxiv.org/abs/1705.10528), to enforce constraint satisfaction. It is important to note that both linesearch and these additional techniques are agnostic to CPO and can be used together with our Lyapunov PG algorithms. Therefore, for the sake of clarity and fairness in comparing different algorithms, we decided to remove all the techniques used to improve the performance/constraint satisfaction balance from the algorithms, and conduct our experiments with their vanilla version. In order for all methods to be consistent in terms performance-feasibility tradeoff, for fixed learning rates, we run grid-search with the same intervals to choose the best learning rate for each method. 

We would like to emphasize that the vanilla version of CPO is equivalent to our Lyapunov-based PG algorithm with \theta-projection (as shown in the paper). We would leave further improving our Lyapunov-based algorithms by adding techniques such as line search, safe-guard policy update, cost shaping, etc., and experimenting with them in larger scale problems for future work. 


HalfCheetah constraint:

We agree with the reviewer that joint torque limit is an intuitive constraint. In fact, both joint torque limit and velocity constraints are well-motivated and have been shown to be useful in different applications (please find more details in Section 3 of https://arxiv.org/pdf/1904.12901.pdf). While the joint torque limit is an action-based constraint, which can alternatively be enforced by specific policy parameterizations (e.g., see Appendix C of the SAC paper https://arxiv.org/abs/1801.01290), the velocity constraint is a kinematic state-based constraint that depends on the state transitions of the MDP. Aside from tackling specific robot control problems, the objective of the benchmark MuJoCo (including HalfCheetah) experiments is to demonstrate/compare the effectiveness of different safe RL algorithms in solving CMDPs. Therefore, we decide to test our algorithms with the latter choice because it cannot be easily formulated without the CMDP framework.


Assuming that you will make the changes promised.  I still have the following concerns

I am still sceptical about your comments on CPO comparison. Your comment is circular argument which I can't follow clearly. 
I am still not sure how constraint satisfaction works without line-search ? Without this a fixed step size may be picked which will always underperform/ violate constraints. In contrast you could pick very small fixed step size ensuring constraint satisfaction but algorithm would be suboptimal.  Neither is  good comparison metric. 
I also disagree that comparison with line-search is future work, as constraint satisfaction is guaranteed for a given fixed step size. 

Cheetah Constraints:
If empirically results are same I suggest  to use joint constraints. Using velocity constraints can be achieved by massive jerks in place, which by no means is safe planning. In contrast if you can run faster whilst keeping joint movement smooth, this is better performance.  

We thank the reviewer for providing useful feedbacks. Please find the itemized feedback to your questions/comments below: 

Value-based algorithm in intro: 
We originally refer to the Lyapunov approach in Chow’18, which primarily relies on the discrete action space assumption (that is key for defining the LP formulation of the feasibility set F_L). It is true that value-based methods can also be used in continuous action RL, although it is less common than policy-based method because solving the max-Q problem can be computationally complex. To avoid these misnomers, we will add more descriptions in the introduction section regarding value-based RL for continuous control in the final paper.

Section 2: 
We mostly adopt the descriptions of Lyapunov functions from Chow’18. We made it clear in the introduction section that using Lyapunov functions to guarantee safety in RL is not a novel contribution of this paper. To improve the readability of this section, we will rewrite the descriptions in this section and add the missing references in the final paper. 

pi_0:
Existence of a feasible \pi_0 can be checked/ensured by minimizing the cumulative constraint cost function. If the corresponding optimal policy satisfies the constraint, one can simply use that as \pi_0. Otherwise, the problem has an empty feasibility set. This argument can be found in Chow’18, and we will add these descriptions about pi_0 in the final paper.

Section 3.1:
We will modify the flow of presentation and streamline the mathematical expressions to improve the readability of this section in the final paper.

Figure 6: 
Figure 6a shows the average success rate over 100 tasks (randomly sampled start and goal robot positions). A task is considered successful if the robot reaches the goal, regardless of the constraint. This is difficult because the robot navigates without the map, relying only on its noisy sensors. Figure 6b shows the average cumulative cost over those 100 tasks. Specifically, for each task we report the constraint experienced over the entire trajectory, and average that over the 100 trials. We will clarify that in the caption of the Figure.

Experiments:
HalfCheetah Safe: In this experiment our constraint is on the speed of the Cheetah. Restricting the joint torque is also an alternative constraint that we tried (but we omit their numerical results for the sake of brevity). We only apply one of the constraints in each experiment. Empirically, policies trained in these two types of constraints have similar performance, while intuitively these two cases have different meanings (bounding the total speed versus bounding the total torque).
 
Comparison with CPO without linesearch: 
Here we choose to compare the Lyapunov-based approaches with CPO without linesearch is mainly because linesearch in TRPO (and CPO) is generally a technique (that is agnostic to the choice of RL algorithms) to ensure constraint satisfaction as well as performance improvement. It is not only limited to improving the performance of TRPO/CPO. While linesearch can also be used in the Lyapunov-based policy gradient algorithm, it is not a part of the original algorithm. Therefore, for the sake of fair comparisons we remove linesearch in CPO during evaluation. Comparisons with linesearch will be left as future work.

Unclear statements:
DDPG v.s. PPO: Here the term covariate shift means that the training data is generated by a policy that is different from the current policy that is being trained, i.e., the RL training is done in an off-policy fashion. 

Sec 5, second paragraph:
The robot’s actions are two-dimensional vectors. First dimension is the robot's desired linear velocity (speed at which the robot should go straight). The second dimension is the robot's angular velocity - speed at which the robot should turn. Both velocity vectors are applied on the center of the mass of the robot. This is commonly known in robot kinematics literature as twist (https://en.wikipedia.org/wiki/Robot_kinematics). We will clarify that in the final paper.

Minor points: 
Thank you for catching that. We will correct the typos in the final paper.


We thank the reviewer for appreciating our work of deriving a novel, Lyapunov-based approach to enforce safety in reinforcement learning (RL) algorithms, and the effort of making these algorithms work in practice (in both the MuJoCo benchmark experiments and the indoor robot navigation example).  


We thank the reviewer for appreciating our work in terms of novelty, theory, and experiments. 


Summary:

Authors propose ideas to perform safe RL in continuous actions domain with modifications to Policy Gradient (PG)  algorithms via either constraining the policy parameters or constraining the actions selected by PG with a surrogate/augmented state dependent objective. The paper is well motivated and the experiments (although I have some reservations about the setup) demonstrate efficacy of the proposed method.  

Review:
--> Introduction
I do not agree with the statement that value function based algorithms are restricted to discrete action domains, especially when you rely on  “ignoring function approximation errors” for some of your claims. 

Again in switching from value function to PG is true for traditional RL/Control theory but this is not valid here. ( your methods rely on Q(s, a) which is action-value function, or the constraint in equation 3 is integral of Q over all actions which would be value-function in traditional definition )
Note: this is explained very well towards the end in the Appendix B, but this is a review of the paper and not Appendix B or C. 


--> Section 2
section 2.3 I would strongly advise the authors to rewrite this, this section reads like it was copied as is from the reference [Chow et al 2018].  especially the way Lyapunov function is defined. And the language and arguments are almost same. Some sentences cite the reference but conclusions drawn on these are not cited, are you claiming that these conclusions are original from this paper ?

It is not clear to me how the feasibility of initial pi_0 is ensured ? Did I miss this somewhere ?


→ Section 3

Section 3 is pleasant to read and very easy to understand, however, same cannot be said of the
section 3.1. I had to spend significant time reading 3.1 and I am still not sure I have understood it very well. 

Experiments:

I don’t think halfCheetah-Safe is actually actually an useful experiment, Limiting the joint torques is perfectly understandable, just limiting speed and getting smooth motion could just be an artifact of the simulation environment. Are both constraints applied simultaneously (torque and speed) ? It is unclear from the text. 

I am not sure CPO without linesearch is actually a fair comparison. Line search may actually deem of the actions unsafe and therefore I would presume original CPO do be less prone to constraint violation than the proposed modification in your experiments. Again PPO is more heuristic than TRPO which makes it hard to compare like for like. PPO might give higher rewards but constraint violations may increase as well. An important point for Safe-RL I feel.

Figure 6 
Can you be more specific as what the figure 6 is showing ? Constraint ? is this constraint violation count? or cumulative sum of constraint slack over the whole trajectory ?
 

Not part of assessment :


Unclear Statements:

Page 7, DDPG Vs PPO: explain clearly what you mean by  “covariate shift” or remove the statement altogether. 

Page 7, section 5 second paragraph, “The actions are the linear …. center of mass” I couldn’t understand this ? What do you mean by actions are velocity ?



Minor points (Language, Typos):

page 3, last paragraph,  Chow et al. is repeated, I can see why this happens there but suggest editing to avoid this. [This is also in intro paragraph, there it is just a typo and should be rectified]

Figure 6: Captions labels are incorrect.







The paper presents a technique for extending existing reinforcement learning algorithms to ensure safety of generated trajectories in terms of not violating given constraints.

I have very little knowledge of this area and as a result was not able to evaluate the paper thoroughly. However, the problem addressed is certainly a very important one and based on my high-level understanding of the concepts involved the approach seems sensible. The experiments are clear and well designed, showing the trade-off between performance and safety.

This is a very complete submission. There is a novel analysis,
simulations, as well as some results on real data. The authors propose
Lyapunov-based safe RL algorithms that can handle
problems with large or infinite action spaces, and return safe
policies both during training and at
convergence. As far as I can tell the approach is novel, makes sense,
and requires a lot of technical innovations. I was impressed with the
method and the analysis behind the method. The incorporation of the
Lyapunov idea from control theory makes a great deal of sense in this
application. However, it is not trivial to get from using this tool to
a working method.

Meta-review: The paper is about learning policies in RL while ensuring safety (avoid constraint violations) during training and testing. 

For this meta review, I ignore Reviewer #3 because that review is useless. The discussion between the authors and Reviewer #1 was useful.

Overall, the paper introduces an interesting idea, and the wider context (safe learning) is very relevant. However, I also have some concerns.
One of my biggest concerns is that the method proposed here relies heavily on linearizations to deal with nonlinearities. However, the fact that this leads to approximation errors is not being acknowledged much. There are also small things, such as the (average) KL divergence between parameters, which makes no sense to me because the parameters don't have distributions (section 3.1). 

In terms of experiments, I appreciate that the authors tested the proposed method on multiple environments. The results, however, show that safety cannot be guaranteed. For example, in Figure 1(c), SDDPG clearly violates the constraints. The figures are also misleading because they show the summary statistics of the trajectories (mean and standard deviation). If we were to look at individual trajectories, we would find trajectories that violate the constraints. This fact is brushed under the carpet in the evaluation, and the paper even claims that "our algorithms quickly stabilize the constraint cost below the threshold". This may be true on average, but not for all trajectories. A more careful analysis and a more honest discussion would have been useful. In the robotics experiment, I would like to understand why we allow for any collisions. Why can't we set $d_0=0$, thereby disallowing for collisions. The threshold in the paper looks pretty arbitrary.  Again, the paper states that  "Figure 4a and Figure 4b show that the Lyapunov-based PG algorithms have higher success rates". This is a pretty optimistic interpretation of the figure given the size of the error bars. 

There are some points in the conclusion, I also disagree with:
1) "achieve safe learning": Given that some trajectories violate the constraints, "safe" is maybe a bit of an overstatement
2) "better data efficiency": compared to what?
3) "scalable to tackle real-world problems": I disagree with this one as well because for all experiments you will need to run an excessive number of trials, which will not be feasible on a real-world system (assuming we are talking about robots).

Overall, I think the paper has some potential, but it needs some more careful theoretical analysis (e.g., effect of linearization errors) and some better empirical analysis. 

Additionally, given that the paper is at around 9 pages (including the figures in the appendix, which the main paper cites), we are supposed to have higher standards on acceptance than an 8-pages paper.

Therefore, I recommend to reject this paper.

Decision: reject