Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors introduce an algorithm to learn a stable controller using deep NN actor-critic method. They define the stability in the mean cost criteria,  which is used to constrain the critic network as a Lyapunov function. In addition, the semi-positive definiteness of the Lyapunov function is enforced by constructing the critic.
The problem is important to control with deep RL. The paper is written clearly. The reviewer has the following questions regarding the stability of the learned policy.

- How is the stability in the mean cost related to the stability of stochastic systems? See, for example, the Lyapunov stability of stochastic systems (survey in [1])?
- The authors enforce semi-positive definiteness using the construction of the value function approximation as the quadratic function bases. Then the semi-negative definiteness is enforced using penalty on the Lyapunov stability of the critic. Then the target network is trained to minimize the difference between the target and the critic. The question is, how is the stability of the target ensured by minimizing the difference with a Lyapunov critic? Is it possible to have an unstable target function that happens to minimize the distance? 

[1] H. J. Kushner, “A partial history of the early development of continuous-time nonlinear stochastic systems theory,” Automatica, vol. 50, no. 2, pp. 303–334, 2014.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this work the authors studied the model-free RL approach for learning a policy with stability guarantees. Leveraging the Lyapunov stochastic stability criterion, instead if minimizing the cumulative cost (plus a soft entropy), they propose optimizing an objective function with a specific Lyapunov critic, which is a specific critic function that satisfies the Lyapunov criterion to guarantee stability. They also show in several Cartpole, Mujoco, and Repressilator experiments that this approach is more robust to perturbations (such as sinusoids), where the agent are more robust to dynamic uncertainties and disturbances. 

In general, the topic of guaranteeing stability is a topic in safe RL, and I find this work of enforcing stability in model-free RL interesting. Through the specific parameterization of quadratic Lyapunov function (in the latent space), the authors proposed learning a new critic function that is a value function but at the same time (almost) satisfies the Lyapunov constraints. While this is an interesting idea, and the experimental results look promising, I do have several questions.  

First, regarding the learning problem of Lyapunov function, how does the proposed way of learning L differ from the one in Richard'18: The lyapunov neural network: Adaptive stability certification for safe learning of dynamic systems, where the problem is formulated as a classification (while in here it is a regression problem)? 
Second, while this approach is intuitive, since the approach is penalty-based (Lagrangian based), I do not see how the Lyapunov criteria in Theorem 1 is guaranteed, in this case is stability guaranteed by the policy learning  algorithm? If not, what do the authors do to enforce that? 
Third, if one formulates the immediate constraint cost of the CMDP to be the distance of the state to the equilibrium point,  then the (undiscounted, shortest-path type)  CMDP total cost constraint should guarantee stability (because the total distance cumulative cost is bounded, meaning that the distance cost converges to zero). Then, one can use the Lyapunov approach by Chow'19 (in modulo to their setting in discounted MDPs) to enforce stability (which is a specific notion of safety in this case).  How does the proposed method compare with this approach? Can the authors provide numerical comparisons with the method proposed by Chow'19 as well?
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: 

######## Rebuttal Response:

Thanks for the thorough response.

Q2: The title still hasn’t changed on the current draft
Q4: To be more precise: 
‘a novel data-based approach for analyzing the stability of the closed-loop system is proposed by constructing a Lyapunov function parameterized by deep neural network’ - this alone is not novel, you would need to specify how your method of doing this is new 
‘a practical learning algorithm is designed to search the stability guaranteed controller’ - this is a natural consequence of contribution 1, some further justification is needed as to why this could be viewed as an interesting contribution (i.e. the Lagrangian approach, if this is novel)
‘ the learned controller is able to stabilize the system when interfered by uncertainties such as unseen disturbance and system parameters variations of certain extent’ - this is not a contribution, but an experimental result.

Q5: The review believes that model-free control and stability-guarantees are fundamentally orthogonal ideas, rather than just under-studied work as the authors have been suggesting in the script and rebuttal. Given that discrete-time Lyapunov stability is defined through expressions along the lines of L(f(x)) - L(x) < 0, for Lyapunov function L and closed-loop dynamics f, claiming that stability is being ‘analyzed’ without f is disingenuous. Instead, by making the value function a Lyapunov function, the goal is that the *converged* value function should produce a stable policy, and still, this is surely only assured within the space of samples. Moreover, the use of the discount factor \gamma, popular in MFRL, essentially acts as a time horizon, so I’m not convinced a Lyapunov function learned with a \gamma < 1 can be called stable in the pure infinite-horizon sense.
With this in mind, I think the work would benefit from a revised central claim: that the use of Lyapunov value functions (as an inductive bias) provides more *robust* model-free controllers. I believe this message highlights the value of this work for MFRL, without making false assertions. This, in particular, would highlight the fact that many MFRL algorithms are benchmarked on deterministic environments, and therefore incredible brittle as the experimental results suggest.

Q9: This remark was aimed at earlier in the paper, either the introduction or main section, rather than the experimental section. The fact that a value function can be viewed as a Lyapunov function makes sense but I’m not sure it is a well-known fact in the wider community. Basically, an introduction to the intersection of Lyapunov stability and optimal control would improve the paper.

Q:10 The fact that the clipping of the multiplier corresponds to unstable policies during learning demonstrates that this pitfall needs to be expressed explicitly. Whether the stability guarantees apply to the converged policy or also intermediate policies is not clear on the initial reading of the paper.
For me, this highlights another weakness in the paper. This initial theorems talk of L(s), which relates to the critic L_c by L(s) = E_{u\sim\pi(s)} [L_c(s,u)], however in the subsequent objectives (eg Eq 2), this marginalization never occurs, therefore I don’t feel like you can say Theorem 2 applies to your resultant algorithm.  Moreover, with the \alpha_3 c term in Equation 3, c should be c(s, a) with a marginalized, which it doesn’t appear to be, and the hyperparameter alpha_3 is never discussed nor tuning explained. Assuming this not done in the code, the experiments need to be re-evaluated with Theorem 2 properly enforced through a sample approximation of the marginalization.

Q11/Q12: Thank you for the Markov jump experiments. I’m not sure I understand why LAC is able to learn the task while SAC cannot. To me, this suggests perhaps a lack of hyperparameter tuning for SAC or further investigation. Moreover, there are typos in captions Fig 1, e and f.
A note of figures: Please ensure all axes should be labeled and should be of sufficient size. Many are too small and unreadable. Figure 2 looks like it could be 1 plot (though perhaps requires normalization).
Given that the strength of this method is the added robustness upon convergence, I think it would be valuable to focus less on time-domain results (Figure 3) (these can be added to the appendix for clarity), but instead show how each parameter/noise variation affects the mean and variance of the episodic return. I would expect that, while LAC provides significant robustness, it is still limited. The results don’t demonstrate this. It would also be interesting to know which hyperparameter controls this limit. I imagine there is a robustness/performance tradeoff.  

In conclusion, while I appreciate the efforts the authors put into the rebuttal, the extended discussions made me rethink my rating and I have decreased my rating to reject. I believe fixing the issues highlighted above and redrafting the central message must be done before this paper is ready for publication. 


######## Review:
This paper investigates the use of Lyanpunov theory as an inductive bias for improving the stability / robustness of policies in a model-free actor-critic reinforcement learning setting. Through viewing the Critic as a Lyapunov function, optimizing the policy with a Lyapunov-based constraint is meant to ensure the stability of the policy through a ‘cost stability’ metric.. Experimental results show that Lyapunov-based Soft-Actor Critic (LAC) is more robust than SAC on some linear and nonlinear environments.

The reviewer believes that the study of intersections between Control Theory and RL to be immensely valuable and the authors outline a principled formulation. However,  the implementation, experiments and general manuscript suggest that paper requires further work before it is conference-ready. 

As the author understands it, the current state of the literature of Lyapunov methods for Deep Reinforcement Learning can be summarized as:
Richards et al, 2018, Classify stable region and learn neural Lyapunov function for a safe exploration strategy
Berkenkamp et al, 2018: Classify the stable region via GP, move there for exploration
Chow et al 2018 Constrained MDPs for discrete gridworld environments
Chow et al 2019 Constrained MDPs for continuous environments through a projection on the policy
This work: Actor-Critic constrained policy optimization with a lyapunov-based value function critic

In the introduction and the related work, too much emphasis is put on explaining stability and discussing methods like Model Predictive Control (MPC) which do not benefit the rest of the paper. Additionally, the three contributions listed do not seem particularly novel given the past literature. 

The premise of the formulation also presents several unquestioned assumptions and design decisions:
Why model-free RL, as the authors also state that many samples are required to validate stability?
How does the requirement of stability inform the search strategy in this work? Especially as SAC uses a maximum entropy stochastic policy to aid exploration. 
Do you really get ‘guarantees’ with sample-based methods? I would expect bounds based on the number of samples
The cost-based measure of stability seems open to abuse - i.e. for the half-cheetah environment only the centre-of-mass horizontal velocity in covered in the cost function, the stability of the embodiment (joint angles and velocities) are ignored. For the Fetch Reacher, a cost function in cartesian space ignores instabilities from kinematic singularities in joint space. I would image the cost function needs to be a measure on the entire dynamic state. 

The notion of a Value function as a Lyapunov function is very interesting, and since it was the basis of the work, would have benefitted from more discussion, i.e. for which cost/reward function families the equivalence is valid for, and how it compares to other Lyapunov candidate functions.

With the RL formulation, the requirement of clipping with the lagrangians is suspicious, as it suggests the objective and/or its numerics are not well posed.

With the choice of experiments, they do not seem to question the central problem outlined by the paper. Rather than show environments SAC returns unstable trajectories during learning, the experiments aim to demonstrate instead a general robustness. The reviewer appreciates that stability is difficult to assess; however, while stability is heavily linked to robustness, a paper title promising stability guarantees should demonstrate some strong empirical evidence stability.
Additionally, the choice of environments do not seem to be ideal test beds for stability - i.e, the half-cheetah is stabilized via interactions with the ground. The reviewer would prefer to see simpler nonlinear environments, such as Markov Jump Processes / Switching Linear Dynamics, where SAC clearly demonstrates instability during learning which LAC is sufficiently regularized against. Additionally, while the `repressilator’ is an interesting application to the domain of bioengineering, its addition does not seem to be especially motivated by the central goal of the paper, so just adds to confuse the reader with unnecessary theoretical content.

Moreover, a brief literature review uncovered some relevant earlier work which was not cited: 
Construction of neural network based Lyapunov functions, Petridis et al, 2006
Generation of Lyapunov functions by neural networks, Noroozi et al, 2008
Lyapunov Design for Safe Reinforcement Learning, Perkins et al, 2002
Some of the references also appear incorrectly formatted or incorrect, i.e. the reference for Spencer et al, 2018 should be the CoRL 2018 version rather than arxiv. 

Also, the general use of grammar in the manuscript would benefit from another draft. In particular, the title could be improved, i.e.
    Model-free Control of Nonlinear Stochastic Systems with Stability Guarantees

review_title: Official Blind Review #1

Meta-review: The authors propose a method to guarantee the stability of a learnt continuous controller by optimizing the objective through a Lyapunov critic. The method is demonstrated on low dimensional continuous control problems such as cart pole. 

The reviewers were mixed in their opinion of the paper, especially after the authors' rebuttal. The concerns center around some of the authors' claims regarding theoretical results, in particular that stability guarantees can be asserted for a model-free controller. This claim seems to be incorrect especially on novel data where stability cannot be guaranteed, thus indicating that 'robust controller' might be a better description. There are also concerns about the novelty and the contributions of the paper. Overall, the method is promising but the claims need to be carefully written. The recommendation is to reject the paper at this time.
Decision: Reject