Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper combines robust control theory with NNs to obtain robustness guarantees. The paper shows stable behavior on adversarial dynamic models on many different simulated tasks. The paper's main idea revolves around projecting the output of a NN policy to be within the set of a Lyapunov function (stability condition) (e.g., exponential stability). I have some concerns which are listed below,

1. What are the limitations of the approach for larger dimensional control problems, such as a 7-dof arm? It's unclear if the method can scale to large dimensional control problems. 

2. Fig.1 methods are hard to separate, plotting with qualitatively different colors would be helpful. Additionally, explaining the horizontal lines that start before 0 would be useful.

3. While the results show stable behavior, the loss in the adversarial setting  doesn't improve during training. Is there some reasoning for this? I am suspecting the adversarial dynamics is randomizing after every epoch? Showing the method improving with a fixed adversarial dynamics over epochs would be good to show improvement in addition to stability.
review_title: Interesting formulation of robust control to learn robust control with NN

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors proposed a new robust controller design approach, in which the controller is parameterised by DNN. They show that by integrating custom convex-optimization-based projection layers into a nonlinear policy, they can construct a provably robust neural network policy class.


1.This is paper is heavily in control theory. The proof of Theorem 1 and corollary 1 is standard in the sense of robust control and LMI. The key contribution I can see is Section 4.3 on deriving differential projections. 

2.In control community, adaptive dynamic programming is exactly to deal with the problem proposed in the paper. Frank Lewis and many others had done a lot of work in this field. I saw the authors cited one of his papers in 2006 in the section of RL. Unfortunately, in comparison with the classic and recent progress in this field is missing.  

3.The definition of stability and exponential stability is not given, which learning community may not be aware of.

4.A key reference is missing “H∞ Model-free Reinforcement Learning with Robust Stability Guarantee, arXiv:1911.02875”. In this paper, the authors propose a model-free approach to learning the Lyapunov function and policy simultaneously. I think this paper is more general than this ICLR submission. A remark and comparison are needed.

5.Regarding the experiment, I didn’t see how the nonlinear systems are converted to (1). If I assume this is possible, how such approximation gap can be quantified and how will this gap affect the theoretical results.

6.One important detail in the experiment is missing: how is the initial condition selected. Let’s take the cart pole as an example: if the initial condition is (very) close to the equilibrium point, the nonlinearities will be minimal. I don’t think robust LQR will be worse. While RL is outstanding in the nonlinear region, the authors should make a fair comparison/add more scenarios.   

7.Comparison with PPO and is MBP is not proper and unfair. PPO and MBP is a data-based method, i.e., a concrete system model is NOT needed. While the proposed method must need a model and the theoretical result depends on the model parameters


review_title: The rebuttal responses most of my concerns, I think it is a good combination of RL and robust control

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, a neural control method is proposed with stability guarantees. The control is assumed to be from a neural network that takes in the state. Stability is guaranteed by projecting the control to the set that satisfies the Lyapunov stability condition for the LQR problem. In particular, minimizing the cost of LQR cost subject to stability constraints can be cast as an SDP for norm-bouned linear differential inclusions. Through making use of the convex optimization layers proposed in Agrawal et al. (2019), the SDP can be added as a layer after the neural policy and efficient projections can be derived such that implicit function theorem can be utilized to differentiate through the fixed point (the optimal conditions of the SDP), such that end to end learning is possible. The proposed approach is compared with the unconstrained method on various tasks.  Both model-based and model-free RL algorithms are used as the neural policy for comparison. The stability-guaranteed approach is able to remain stable even under bounded adversarial dynamics. In comparison, the non-robust methods fail to maintain stability.
 
In general, I like the idea of enforcing stability by introducing the convex optimization layer. Although the dynamics model used is still relatively basic, but the nice convex formulation provides an opportunity to incorporate stability certificates to neural policy that enable end-to-end training. The stability is roughly maintained as illustrated in the experiments. However, it would be better if trajectories can be visualized in some way to show that the proposed method can stabilize the system. One potential concern of the method is that it would be more computationally expensive than the unconstrained method, therefore it is of interest to compare the running time. 

Some minor points regarding the experiments: the description of the methods used are not properly defined or referenced (such as PPO, RARL, MBP). Also, the experiments for PLDIs and H-infinity control settings seem to be missing (only description of the setup is found). 

----------------------After author's response--------------------

The response addressed most of my concerns and included experiments results of trajectory visualization and run-time comparison. I think the paper would be an interesting contribution to the conference.
review_title: Interesting stability-guaranteed neural control method through introducing convex optimization layers

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:
  [1] The problem of RL policies with both robustness guarantee and good average performance is interesting and useful for many practical applications.
  [2] Paper is well-written and clear.
  [3] The proposed method/approach is inspiring and novel, an the results look promising for this proposed new method.


Main concern:
  The results presented in the experiments section is not very comprehensive and not convincing enough to justify the claimed benefits of the proposed approach v.s. traditional robust LQR approach. In particular, as we could see from Table 1, although in the original dynamics scenario, the proposed robust MBP and robust PPO approach has better performance than the robust LQR, in the adversarial disturbance scenarios,  the proposed robust MBP and robust PPO might perform significantly worse (e.g., about 8 times larger loss for Microgrid case) than the traditional robust LQR, which implies that, for many scenarios that are not worst case, the proposed method could perform worse than the robust LQR (i.e., from 8 times larger loss (e.g., 7.12 v.s. 0.86) in the adversarial case, to  slightly smaller loss (0.61 v.s. 0.73) in the original case, there is a large gap there). And there is not a clear and well justified criteria in the paper to clarify, for most real world applications, what disturbance is defined as normal(/average/original) case, and what scenario is for adversarial case, and also how about the disturbance between these two scenarios? Due to above concerns, and also note that the examples provided in this paper is very limited rather than comprehensive, it is not convincing enough to claim the overall performance benefits of the proposed approach over the traditional robust control techniques. More comprehensive experimental studies and evidence are needed to well justify the claimed performance benefits.

Edit: upgrade the rating to 6 with the clarifications from the authors, with which the submission is clearer and more convincing now.
 
review_title: This paper proposed a class of nonlinear control policies that combines the expressiveness of neural networks with the provable stability guarantees of traditional robust control, under the area of safe RL. It was claimed that such new policies has the main advantages to good performance in average case than traditional robust control technique, and has the advantage of provably robustness guarantee over the  traditional non-robust RL methods.

Meta-review: While the reviewers seem to like the main idea of the work, they had several concerns, particularly regarding the experiments (both their setup and description) and the overall language of the paper that they found it more suitable for the control community than the ML and representation learning community. The authors provided very long response and tried to address the issues raised by the reviewers during the rebuttals. Fortunately, the response addressed some of the issues they raised and now they all see the paper marginally above the line. However, reading the reviews and response shows that the paper can highly benefit from better writing and describing the experiments. So, I would strongly recommend that the authors include all the information they provided for the reviewers during the rebuttal phase in the paper and improve its quality. 
Decision: Accept (Poster)