Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors developed a method for learning policies for stochastic control systems with formal guarantees about the systems’ a.s. asymptotic stability over the infinite time horizon. This work is most relevant to the previous recent work on ranking supermartingales 
(RSMs) by Lechner et al. (2022), and it mainly extended that result by relaxing the restriction that "the stabilizing set is closed under system dynamics and cannot be left once entered" with the help of a novel notion of stabilizing ranking supermartingales (sRSMs). Experimental results demonstrated the effectiveness of the proposed method.
strength_and_weaknesses: Strength:

[1] The studied problem about stochastic control systems with formal asymptotic stability guarantees is interesting. The proposed method is novel and the empirical studies is helpful to demonstrate the effectiveness of the proposed method. 

[2] The paper is written reasonably clear and well organized.

Weakness:

[1] Since the main contribution of this paper compared the previous recent work by Lechner et al. (2022) is that, this work relaxed the restriction "the stabilizing set is closed under system dynamics", it would be helpful for the authors to elaborate more about why relaxing this limitation is important. Is such restriction a major limitation in real world applications? How common or uncommon to see such restriction in real world applications? Will relaxing of this restriction significantly enlarge the family of systems in real world applications (if yes, any application examples?) that the asymptotic stabilization results could be applied to, or such relaxing of the restriction is mainly important from theoretical perspective, and so far no any obvious impacts on any example real world applications? Some more discussions and explanations would on this direction would be helpful to better understand the motivations and significance of the results.

[2] I'm not fully convinced about the claimed rigorous theoretical a.s. asymptotic stability guarantee for the closed-loop system using the proposed method with neural networks approximation, and I think this claim is inaccurate and somewhat misleading. In particular, such asymptotic stability guarantee is based on the assumption that the approximating NN can learn a sRSM that satisfies all the 3 conditions in Definition 3, however, there is no any rigorous mathematical guarantee that the trained NN (which is an empirical/heuristic process) could always find such an appropriate sRSM. The description in page 5, i.e., using a NN training objective in the form of a
differentiable approximation of the sRSM conditions 2 and 3 in Definition 3, plus "if at least one sRSM condition is violated, the
verifier module enlarges the training set of the learner module by system states that violate the condition in order to guide the learner towards fixing the policy and the sRSM in the next learner iteration", are just some heuristic tips to help find a good sRSM, but there is not rigorous mathematical guarantee that an appropriate sRSM can always be found following these tips. Therefore, it might be unfair to claim that it always achieves rigorous theoretical a.s. asymptotic stability guarantee for the closed-loop system using the proposed method. I'd like to see some more clarifications/explanations from the authors on this.





clarity,_quality,_novelty_and_reproducibility: Paper is written reasonably clear and organized well, and it contains novel valuable results. 
summary_of_the_review: Generally speaking, I think this paper studies an interesting problem and the proposed method is novel  with its effectiveness appropriately demonstrated by the examples. In my opinion, the paper contains some novel valuables results that might be publishable on ICLR.

But I'd like to see more more elaborations and explanation on the motivation and the significance of relaxing the restriction that "the stabilizing set is closed under system dynamics" to better judge the contribution significance, and I also have some concern about the unfair/inaccurate claim that "it always achieves rigorous theoretical a.s. asymptotic stability guarantee for the closed-loop system using the proposed method", which I'd like to see some more clarifications/explanations from the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors consider the problem of asymptotic stability for discrete-time stochastic systems. They introduce a class of supermartingales, which they call stabilizing ranking martingales (sRSMs), which can be used to prove asymptotic stability and they show how these can be parametrised as neural networks and learned together with a controller. The method is evaluated on two 2-D systems.
strength_and_weaknesses: Strength

The paper is well written and clear and addresses an important problem. The theoretical results are non-trivial and proofs are clear and sound.

Weaknesses

My main two doubts about the paper are the following:

- Results are quite incremental compared to Lechner et al. (2022). In particular, the techniques employed to learn and verify a neural network as a sRSM follow from those of Lechner et al. (2022)., which was also considering stability of stochastic systems. The main difference in that in this paper the authors managed to remove the constraint that the system dynamics must be closed with the stabilizing set. However, while interesting,  also considering the weaknesses identified below, I am not sure this is enough contribution for a full paper at ICLR.

- Experiments are limited to 2-D systems. Would it be possible to consider also a 3-D system? If not, what is the main challenge? Also, looking at the contour lines in in Fugure 4 the expected stabilization time obtained by Theorem 2 looks quite conservative. Can you compare the value that you get with an empirical estimate obtained by running n trajectories of the systems?

- While the above are my main concerns on the paper, another weakness is that there are some claims that are not accurate compared to the literature. For instance, the authors claim that "there are only few works that consider control with formal stochastic stability guarantees" and that "existing works assume that the stabilizing set is closed under system dynamics". While this may be true wrt learning based approaches, formal results for asymptotic stochastic stability also without the assumption that the stabilizing set is closed under system dynamics have been widely studied, see e.g. [Kushner, Harold J. Stochastic stability and control. Brown Univ Providence RI, 1967] or [Khasminskii, Rafail. Stochastic stability of differential equations. Vol. 66. Springer Science & Business Media, 2011] for a more recent reference. Also, in the intro the authors  seem to claim that only recently the notion of Lyapunov functions have been extended to stochastic systems for certifying a.s. asymptotic stability, but as shown in the books above this was done way before. I suggest the authors to adjust these claims making them more precise.


clarity,_quality,_novelty_and_reproducibility: Paper is clear and well written (apart from my concerns in Weaknesses Section). I have some doubts on the originality compared to Lechner et al. (2022)
summary_of_the_review: Well written paper with interesting theoretical results. However, I feel that the results are a bit too incremental compared to the literature. Also, experiments are not very convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on learning control policies in stochastic systems that guarantee that the system stabilizes within some specified stabilization region with probability 1. The authors create an approach based on  stabilizing ranking supermartingales (sRSMs).


strength_and_weaknesses: Strengths:
The technical basis is clearly defined, and supported by proofs.

Weaknesses:

1. A major issue with this article is the lack of comparison of the approach to known methods.
 a) I want to see a clear comparison to control synthesis for stochastic systems, , e.g., stochastic hybrid systems using barrier certificates [Huang et al. 2017]--see reference below.

b). Can you clearly show a comparison to traditional stochastic control using state space methods, e.g., stochastic MPC? You assume knowledge of the state space model, so this is a natural comparison to understand.

c) There is a big overlap with [Lechner et al. 2022]. Please identify what is novel in this submission. In particular, the empirical study seems almost the same.


Related work

The claim "there are only a few works that consider control with formal stability guarantees" is not entirely correct. The authors fail to mention a significant body of work in this area, e.g, [Huang et al. 2017], as well as numerous other papers. See survey by [Lavei et al. 2022].


Empirical analysis

Why in the experiments do you focus only on a disturbance vector sampled from a zero-mean triangular distribution? This is not realistic for real-world systems.
Are the contour lines of the expected stabilization time (Figure 4) a function of the triangular-shaped random noise?

This method has been studied empirically only for toy systems. Can you comment on scale-up issues? is this approach meant to operate in real-time, or to be trained offline?

Verification

You note that verification of neural networks is inherently a computationally difficult problem, but what of the semantics? Is this even well understood? In comparison to other formal-methods approaches to the same task, e.g., stochastic control verification, there is no well-defined reason why this proposed approach is suitable in terms of semantics. So a key question: why use verification via neural network control synthesis? What not use standard control-theoretic methods?





Huang, C., Chen, X., Lin, W., Yang, Z., & Li, X. (2017). Probabilistic safety verification of stochastic hybrid systems using barrier certificates. ACM Transactions on Embedded Computing Systems (TECS), 16(5s), 1-19.

Jagtap, P., Soudjani, S., & Zamani, M. (2020). Formal synthesis of stochastic systems via control barrier certificates. IEEE Transactions on Automatic Control, 66(7), 3097-3110.

Lavaei, A., Soudjani, S., Abate, A., & Zamani, M. (2022). Automated verification and synthesis of stochastic hybrid systems: A survey. Automatica, 146, 110617.
clarity,_quality,_novelty_and_reproducibility: The novelty of the paper is hard to evaluate given the overlap with  [Lechner et al. 2022].

The paper is dominated by the theory; the empirical analysis is quite weak in comparison. It is unclear why deep learning was used; clearly, having state-space models means that one can use standard control theory methods for model estimation. There is no basis for comparing deep learning with standard control theory methods, since this is never addressed. Without such a comparison one cannot assess novelty.
summary_of_the_review: This is a technically sound paper that unfortunately provides limited basis for evaluation of the results in comparison to state of the art.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes theoretical results for the problem of stabilization in nonlinear stochastic control systems. The main contribution is that the authors introduce the notion of stabilizing ranking super-martingales (sRSMs) for certain stabilization regions, that can be used for ensuring almost sure stability.
strength_and_weaknesses: A non-inclusive list of things to be addressed is as follows. 

-  I am wondering that the suggested policy satisfies the asymptotically stability defined in Definition 1. For each step, the suggested policy controls an output to be included in the stabilizing region with probability at least 1-p satisfying the asymptotically stability. But, the asymptotically stability condition seems to require that p converges to 0 or other additional conditions. For example, suppose that the stabilizing region is X_S = [0,0,5] and the output x has the Bernoulli distribution with p. Then, x can be in the stabilizing region with 1-p, however, it does not satisfy the asymptotically stability. Theorem 2 shows that the probability p decreases over time, but it does not match the statement about the suggested policy. ------- The authors clarified this example in their rebuttal.

-------------------------------------
post rebuttal: Overall, the work does not seem to have significant novelties or important useful results. The setting is clearly hypothetical, and the theory seems like providing some conditions that their usefulness is not clear. To be a full paper, I believe that further contributions are needed, and also they must be contextualized better. Further, comparisons with the existing literature, e.g. on learning to stabilize linear systems and how this work generalize from linear to nonlinear systems, are required. 

clarity,_quality,_novelty_and_reproducibility: Some presentation issues:
-	For the pseudocode of Algorithm 1, line-by-line explanations would help readability.
-	For Cells_{\chi \ \chi_s} on page 7, more details are needed. 
-	There is no x_2 on y-axis on the left plot of Figure 4.
-	eq. equation i can be simply written as (i).
-	Explanations for Table 1 should be added.

summary_of_the_review: Overall, the paper is well written, and the results are well organized. I will try to finalize my decision after the rebuttal period to one of the accept or reject options.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers are mainly concerned that the work is incremental compared to Lechner et al. 2022. They are not sure about the importance of relaxing the restriction (the stabilizing set is closed under system dynamics) in Lechner et al. 2022. There are certain claims in the paper that the reviewers did not find convincing, which require either better explanation or a revision. There are also concerns about the limitation of the experiments and the lack of comparison with existing results. Overall, although the paper is well-written and contains novel ideas, it needs to be significantly improved and certain important concerns to be addressed. I would recommend the authors to take the reviewers' comments into account, improve their work, and prepare it for future venues.  
justification_for_why_not_higher_score: There is a big overlap with an existing paper, certain claims that have not been properly supported, limited experiments, and lack of proper comparison with existing results.
Decision: Reject