Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Update: I've checked out authors' response and other reviewers' comment, and my rating stays unchanged.

In this paper, the authors proposed two frameworks of neural stochastic control, i.e., the ES (exponential stabilizer) and the AS (asymptotic stabilizer), and presenting their advantages in the stochastic control. The convergence time and the energy cost of particular stochastic neural control were computed, and the efficacy of the proposed stochastic neural control were demonstrated in control problems arising from multiple physical systems. Compared with traditional control methods in the existing literature, one major novelty the authors claimed is that, the authors treat noise as a beneficial part and design stochastic control u_g to achieve the stabilization.
strengths_and_weaknesses: 1. The problem under study is interesting and important, and the obtained results on neural stochastic control seem to be novel and valuable, which would be good contribution to the research community in this area.

2. The paper presentation is good and clear, and I found the manuscript easy to follow. The concepts and main results are described/summarized clearly with clean proofs provided, and both the advantages and limitations of the proposed methodology are well explained.

3. The numerical experiments on multiple representative physical systems are helpful, and the advantages of the proposed method is reasonably convincingly demonstrated in these numerical examples.

4. The discussed limitations seem to be reasonable & helpful, and suggested future research directions look promising.

5. Overall speaking, in my opinion, this seems to be a solid paper that is acceptable to NIPS and could benefit the research community in this research area.
questions: 1. Some of the numerical performance comparison with baseline methods could be improved.  For example,  in page 7, when numerically comparing the performance of the proposed method with traditional linear controller, it was claimed that "Without loss of generality, we fix k = 6.0,...", and only describe the numerical performance for k=6.0. I'd expect the value of k to have quite some impacts on the numerical performance (convergence speed and energy cost here) of the traditional linear controller, and I'm not quite convinced by the argument of "fixing k=6.0" is "without loss of generality", in terms of "numerical performance comparison" (instead of stability). Did you conduct comprehensive experiments for many different values of k and confirm that their numerical performance stay similar to each other, and always much worse than the proposed new neural stochastic control method here? To be more convincing for such numerical comparison and advantage claiming, it would be helpful for the authors to conduct such more comprehensive performance investigations and at least share a brief summary about the results in the paper (and share configs to make sure the summarized results/(claimed benefits) could be reproduced using the shared codes).
limitations: The authors have adequately discussed the limitations of the proposed framework in the manuscript and suggested a couple of promising future research directions for further improvements.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper deals with stochastic problems and is in alignment with previous neural network studies inspired by the Lyapunov stability theory. In this paper, the stochastic controller is defined as a neural network that stabilizes the "zero" solution and is applied $dB_t$ in the SDE. To efficiently obtain these solutions, the authors propose two types of controllers for neural networks in stochastic control systems: the (1) exponential stabilizer and the (2) asymptotic stabilizer. The first one actually requires a neral Lyapunov function, so it can be quite accurate with nonlinear systems but computing the loss function takes a relatively long time. The second one does not require actual function but guarantees almost-sure convergence, and it updates fast in reality. This paper shows that the proposed model outperforms the classical stochastic linear control in both time and cost for systems.
strengths_and_weaknesses: This is overall a solid paper that retains theoretical and empirical novelties for challenging control problems. The technical framework can be considered novel and general since it is dealing with stochastic problems. This paper provides some convergence analyses and experimental results on several physical systems. I think this work contributes to stochastic control studies that conventionally have used linear controllers to solve such problems. Meanwhile, I also would like to point out I have a few concerns regarding the assumptions on stabilizers and experimental results. I hope to see responses to these concerns from the authors.


### Strengths

* Major
  * This work presents trainable nonlinear controllers for stochastic control problems.
  * The paper offers two distinct stabilizers which can operate with different characteristics of convergence rates and computational costs.
  * The manuscript is clearly written; I think overall analyses are technically sound.

* Minor
  * The experiments cover various physical environments. General readers can grasp how each stabilizer works through the experiments.
  * I think the techniques shown in the exponential stabilizer show a promising direction for designing physics-informed neural networks.

### Weaknesses

* Major
  * Theoretical analyses might work in ideal situations, but they seem to not fully cover addressed claims.
  * If I understand correctly, I think controlling mechanism $\mathrm{d} B_t$ is somewhat limited. For example, instead of $\mathrm{d}x = f(x)\mathrm{d}t + [g(x) + u(x)]\mathrm{d}B_t$ why not use
 $\mathrm{d}x = [f(x) + u_1(x)]\mathrm{d}t + [g(x) + u_2(x)] \mathrm{d}B_t$ for efficient stabilization?
  * I could not find meaningful comparison studies with other approaches in the experiment section. The framework proposed in this work is expected to replace classical linear controllers. Therefore, it is very important to provide comparison studies with these controllers (e.g., the classical LQR techniques). The toy experiments in Figure 4 have such results for a simple system, but it seems the performance of $u(x) = kx$ is missing for other experiments. I think this work needs a revision since comparison experiments with previous methods should be included.
* Minor
  * I think the title "Neural Stochastic Control" does not clearly explain the methodology in this paper. Maybe this has to be more specific for representing two stabilizers (ES and AS). I believe the title has to be changed to indicate that this work is for the stochastic Lyapunov theory and the stochastic asymptotic stability theory.
questions: 1. (about Theorem 4.1) In the theorem, only the result of a stochastic linear controller ($u(x) = kx$) is presented. Are there any theoretical results on the convergence of the nonlinear ES/AS controllers?
2. (about the AS loss) Could you explain more about the theoretical background of the AS loss function? Is the AS stabilizer "safe" to use for complex physical systems?
3. (about the ES stabilier) How much model complexity can the training of ES(+ICNN) model hold in a reasonable time?
limitations: From the authors' description, I can imagine a situation that AS loss is later discovered by the authors and applied in this work mainly for its low computational costs. Therefore, it is worth questioning whether the AS loss (or the notion of asymptotic attractiveness) is appropriately presented in this paper titled "Neural Stochastic Control." First, I think overall dedicated technical descriptions for this loss (Section 2.3) are too short. As far as I understand, the only reasonable description for AS controller is Theorem 2.3. I have a concern about whether Theorem 2.3 is tightly coupled with other theoretical results such as Theorem 2.2 or whether it is a way of approximation that can simplify the aforementioned framework. Therefore, the lack of justifications for AS loss is one limitation of this work.

Another limitation is that the ES stabilizer seems to be performant, but it might take a considerable amount of time to be trained. Therefore, I have another concern that this two-fold approach is actually applicable, or only AS is the reasonable algorithm that can successfully train a controller in a limited time.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper gives an iterative procedure for computing model-based controllers that stabilize the system with noise. In particular, the control law enters the equations in the form $u(x_t)dB_t$, where $B_t$ denotes Brownian motion. The controllers are designed based on two sufficient conditions for stochastic stability, one requiring a Lyapunov function and the other working directly on the terms of the SDE. These sufficient conditions are used to derive loss functions for optimization-based search algorithms. For the Lyapunov method, both the Lyapunov function candidate and the control law are parameterized by neural  networks. The controller is computed by optimizing the resulting loss with respect to the network parameters over a randomly chosen set of states. In the other approach, no Lyapunov function is needed, and so only the control parameters are optimized. The method is tested in a variety of examples. 
strengths_and_weaknesses: Strengths:
This paper shows how controllers can be designed to solve some interesting and non-standard control design problems. Namely, controlling the system via noise is theoretically established, but substantially less studied than noise-free control. The results show promising performance for a variety of nonlinear stabilization tasks. 

Edit: The newer results based on learning the dynamics and partial control show promise. I would still like better comparisons, but this merits a score increase in my view.

Weaknesses:

It is unclear if the methods in this paper have any advantages over well-known stabilization methods. So, while the methods outperform simple methods that inject noise for control, it is unclear if the method has any benefit over more standard methods that do not inject noise. 

More explanation on the comments above: The methodologies are model based. Furthermore, in all examples where the dynamics, (5), are written explicitly, they are fully actuated. In other words, the inputs directly affect all states. In all cases, very simple feedback-linearization schemes can stabilize these systems. Presumably, other schemes can stabilize them as well. 

First we consider the case that the only noise is actually due to the controller. This is  the case in the inverted pendulum and the Stuart-Landau systems. This is presumably also the case for the coupled Stuart-Landau oscillators, the echo state network, and the Lorenz system. However, (5) does appear to be written explicitly for these systems, so this is just a guess. Anyways, when the system is fully actuated and the model is fully known, and the system has no intrinsic noise (i.e. no term $g(x)dB$), we could alternatively write the system as 
$$
\dot x = f(x) + u(x).
$$
a simple controller of form 
$$
u(x) = -f(x) - x
$$
Sure, we could make the problem harder by injecting noise, but in the fully actuated case, this will do.

In the simple harmonic oscillator problem, the system has the form:
$$
dx = Ax + (g(x)+u)dB,
$$
but again it is assumed that the system is fully actuated. Furthermore, $A$ is Hurwitz.  So, inn this case, $g(x)$ could be cancelled by the controller leaving a stable system. 

A more minor weakness is that the theoretical results are completely restricted to model-based stochastic control, and so the learning component is not very strong.


questions: From what I can tell, the notation $\mathbf{u}$ is overloaded to denote simultaneously the control strategy $u(x)$ and the neural network parameters of the control strategy. Is this correct? If it is, it is a bit confusing and an alternative notation to denote the function and the parameters is advisable.
limitations: They listed this as N/A, and I agree with the assessment.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Prior work on approximating a Lyapunov-like function as a neural network focused on deterministic stabilization where the controls affect the system deterministically. For a system, for which the stochastic noise process can have a non-linear impact on the system, the gains on the noise in itself can be controlled to stabilize or de-stabilize the system. Generating such gain controls requires developing a Lyapunov-like function that satisfies certain criteria. This paper extends previous work that models such functions as a neural network. The hard constraints are converted into a soft loss function and minimized using gradient descent on the neural network. The novelty is in extending deep Lyapunov functions to stochastic stabilization where the noise process itself can be fed back with suitable gains to stabilize the system.
strengths_and_weaknesses: The experiments on Stuart-Landau equations, Lorenz oscillator are new contributions and provide strong evidence for the paper’s hypotheses. Instead of convergence to an equilibrium point, the controller is optimized to ensure convergence to a manifold of constant phase for synchronization. This can have real-world applications.


Lyapunov stability theorems have been studied in non-linear dynamical systems for a long period of time. This paper uses existing theory to bring new problems to the fore. Original theoretical contributions are interesting but not as strong a contribution as they are applicable to the linear controller only. 

questions: For the harmonic linear oscillator, how would an optimal deterministic controller (u=K_{lqr}x) perform compared to the stochastic counterparts. Maybe, previous work showed that neural lyapunov controllers perform better than LQR.

It would be better if the authors can present the preliminaries in section 2 more elaborately to enable deeper understanding. 

Is the derivative operator an extension of the Lie-derivative operator? The derivative operator here is related to Ito’s lemma from stochastic DDE literature.

Similar to SOS (sum of squares) methods, are there methods that don’t use deep learning to generate V. Is it possible to compare to these? Also, [A] uses deep RL for stochastic control. How does your work qualitatively compare to that?

Please try to make the figures more readable.

[A] Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications - Bachouch  et al

limitations: I don't see any potential negative social impact.

The general technique of learning lyapunov-like functions is limited to lower-dimensional systems as the methods require sampling to enforce the constraints.
review_title: null

Meta-review: metareview: The paper proposes two new frameworks for stochastic neural control. The methods have both theoretical and experimental proofs.
The extensive replies and additional material/experiments managed to address most concerns of the reviewers (I am also satisfied by the replies to 7JKN). As the authors pointed out, ideally some of the new material should be incorporated in the main paper rather than in the appendix.
Feedback linearization (a1wg) is indeed a powerful technique, but as the authors point out cannot be applied (completely) in all situations. Having an additional method available is valuable either way.
confidence: Certain
Decision: Accept