Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a model-free algorithm for stabilizing linear and nonlinear dynamical systems. The algorithm solves a series of discounted LQR problems using policy gradient, and provably converges to a stabilizing controller for LTI systems; for nonlinear systems, the algorithm recovers a stabilizing controller in a small ball around the origin (assuming origin is the equilibrium point). This work overcomes a significant limitation of prior work on policy gradient methods for control, which requires a stabilizing controller a priori. Experiments on a simulated nonlinear system suggest that the algorithm can produce a controller in a few iterations that is stabilizing in a ball of reasonable radius around the origin.

main_review: The paper considers an important problem, since many control methods require a stabilizing controller even when the system is unknown. Moreover, the paper leverages existing techniques but uses new insights to overcome the limitations of said techniques. The theoretical results appear to be correct but I wasn’t able to check all the details. 

However, the significance of the result can be strengthened by comparing with other stabilization methods for linear systems with similar access models. For example, “Finite-Time Adaptive Stabilization of Linear Systems” and “Randomized algorithms for data-driven stabilization of stochastic linear systems”  give algorithms for stabilization. How does this paper compare with the techniques used there? 

I’m also not completely convinced of model-free methods’ advantages over mode-based methods for linear systems, since linear systems don’t have arbitrary dynamics. 

Some limitations of the work:
The setting for linear systems assumes there is no noise, which is restrictive. Can the results be extended to accommodate stochastic noise?
The experimental section could be improved by including other, and possibly more difficult, tasks.

Post rebuttal: I have read the authors' response, and will increase my score. 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies how to stabilize dynamic systems using policy gradient methods. In the linear setup, a global stabilization problem is addressed. In the nonlinear setup, a local stabilization problem is solved. The basic idea is that a zero control can be used when the discount  factor is sufficiently small, and one can iteratively apply a discounted LQR formulation to increase the discount factor. The core algorithm is "Discount Annealing." Both theory and simulations are provided to justify the proposed method. Overall, this paper provides the first sample complexity result for solving stabilization via policy gradient methods.
main_review: Originality: In my opinion, this paper is quite original and considers an important question. Stabilization is a fundamental question for control, and how to solve it via policy gradient is unclear. This paper makes progress in answering such an important question, and I can see there is unique contribution in this paper. 

References: This paper missed a large body of literature on policy optimization of linear control problems. The following papers are relevant, and the authors should discuss them.

1. T. Rautert and E. W. Sachs, “Computational design of optimal output feedback controllers,” SIAM Journal on Optimization, 1997.

2. K.M°artensson and A. Rantzer, “Gradient methods for iterative distributed control synthesis,” CDC, 2009.

3. D. Malik, A. Pananjady, K. Bhatia, K. Khamaru, P. Bartlett, and M. Wainwright. "Derivative-free methods for policy optimization: Guarantees for linear quadratic systems," AIstats, 2019.

4. K. Zhang, B. Hu, and T. Basar, “Policy optimization for H2 linear control with H-infinity robustness guarantee: Implicit regularization and global convergence,” L4DC, 2020.

5. L. Furieri, Y. Zheng, and M. Kamgarpour, “Learning the globally optimal distributed LQ regulator,” L4DC, 2020.

6. J. P. Jansch-Porto, B. Hu, and G. Dullerud, “Convergence guarantees of policy optimization methods for Markovian jump linear systems,” ACC, 2020.

7. B. Gravell, P. M. Esfahani, and T. Summers, “Learning optimal controllers for linear systems with multiplicative noise via policy gradient,” IEEE TAC, 2020.

8. K. Zhang, B. Hu, and T. Basar, “On the stability and convergence of robust adversarial reinforcement learning: A case study on linear quadratic systems,” NeurIPS, 2020.

9. I. Fatkhullin and B. Polyak, “Optimizing static linear feedback: Gradient method,” arXiv preprint arXiv:2004.09875, 2020.

10. H. Mohammadi, M. Soltanolkotabi, and M. R. Jovanovic, “On the linear convergence of random search for discrete-time LQR,” IEEE Control Systems Letters, 2020.

11. H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanovic. "Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem." IEEE TAC, 2021.

12. Zheng, Y., Tang, Y. and Li, N. "Analysis of the optimization landscape of linear quadratic Gaussian (LQG) control," L4DC, 2021

13. Zhang, K., Zhang, X., Hu, B. and Başar, T. "Derivative-free policy optimization for risk-sensitive and robust control design: Implicit regularization and sample complexity."  arXiv, 2021.


Quality: This is a solid paper, although it is unclear how the authors draw the conclusion that their method improves the H-infinity control method. I have a few more questions here.
1) How can the authors prove that gamma keeps increasing to 1 without approaching some limit point in the middle? I guess the proof will require some lower bound on how much progress on gamma can be made at each iteration. Is that true? Any uniform lower bound here?
2) In the nonlinear setting, it seems that based on Lyapunov's indirect method, one can just linearize the dynamics and then stabilize the linear systems. Right? What is the point of developing new theory for "local" stabilization of nonlinear systems?
3) Any evidence supporting the claim that the proposed approach is competitive with established robust control procedures? I mean, I can't find how the authors set up robust control baselines in their numerical study.
4) For simulations, how does the proposed approach work on systems which have modes blowing up to infinity very quickly? In this case, although the theory states that a very small discount factor can be used such that the zero control gain lead to a finite cost, practically the system states in the simulations can blow up quickly to cause numerical issues, right? Can the authors provide an example on this?

Clarity: The paper is well written. The idea is quite clear.

Significance: I think the paper is significant in that it tries to answer a fundamental question with relatively simple ideas. In my opinion, such a solution for stabilization is novel and interesting. 

================
Post-rebuttal: I am convinced to increase my score to 7.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work gives a method to compute stabilizing controllers given access to noisy gradient evaluations with respect to the policy. The challenge is that if a controller does not stabilize the system, then the cost is infinite and the gradient is undefined. They key insight is to use the discount factor as  a free parameter. Namely, for sufficiently small discount factors, the cost is finite and the gradient can be evaluated. So, the algorithm alternates between taking gradient steps and increasing the discount factor. When a discount factor of 1 is reached, stabilization has been obtained and the problem reduces to a policy gradient method. Bounds on convergence and applications to nonlinear systems are given. 


main_review: This paper is well-written and insightful. The proofs appear to be rigorous. However, as discussed below, the basic idea of using the discount factor for model-free stabilization has appeared independently in the literature. The proofs appear to be very different from the existing work, and the similarities appear to be coincidental. So, overall, this is nice work, the contribution feels a bit smaller as a result.

A very similar model-free methodology has already been devised for computing stabilizing controllers and solving LQR in:

Lamperski, Andrew. "Computing Stabilizing Linear Controllers via Policy Iteration." 2020 59th IEEE Conference on Decision and Control (CDC). IEEE, 2020.

Both are model free, and both interleave policy improvement steps with increases in the discount factor. Both give stabilizing controllers and converge to the LQR solution. 

The prior work uses policy-iteration style updates in an off-policy manner. In particular, it can be used on a single trajectory or an offline data set. The current submission uses gradient evaluations, which effectively require restating the system for each new gradient. The current submission does give more explicit convergence bounds and gives applications to nonlinear systems.

Edit based on Response: Based on the response, I can see greater merit and distinction in the work and I will raise my score appropriately.
limitations_and_societal_impact: The authors have discussed the limitations. I do not forsee any negative societal impact.
review_title: null

Meta-review: All reviewers find an interesting and important idea in this paper. Although some reviewers point out that the similar idea is found in some existing works, the authors clarify the significant difference or their work from those. As a total, this is a nice paper that could appear in NeurIPS. Therefore, my recommendation is acceptance (poster) for this paper.
Decision: Accept (Poster)