Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a machine learning framework, named Markov neural operator (MNO), to learn the Markov operator of dissipative chaotic systems. Different from the conventional neural networks, they use Sobolev norms in operator learning and add dissipativity losses to ensure dissipativity. They experimentally show that the MNO can accurately approximate the global attractor and estimate various statistics of the invariant measure for dissipative chaotic systems. Besides, they provide a theoretical guarantee that this model is rich enough to approximate many chaotic dynamical systems for an arbitrarily long period.
strengths_and_weaknesses: This is a novel method to approximate the global attractor and estimate the invariant statistics of dissipative chaotic systems. They pioneered the combination of the Sobolev losses and the dissipativity losses, which improves the performance for preserving the invariant statistics. The dissipativity losses they proposed can enforce the dynamics close to the attractor, which improves the robustness against large perturbations. They also investigate the effect of time steps and different Sobolev losses in operator learning, which makes this work more systematic. However, there are still some confusing details. The choice of the hypermeter lambda and the inner and outer radii of the shell in the dissipativity losses remains to be discussed. There is no description of the proportion of the Sobolev and the dissipativity losses in the loss function. In addition, sometimes we do not know whether the target system is Markovian in a model-free task. As a result, this paper lacks a method to judge whether the MNO can be directly applied to a certain target system.
questions: 1. How the choice of the inner and outer radii of the shell of the dissipativity losses influences the performance of the MNO? Whether the choice significantly affects the MNO’s step-wise error and the estimation of the statistical properties of the attractor? How to choose the radii based on the training data?
2. How to balance the influences of the Sobolev and the dissipativity losses? Is there a hyperparameter to adjust their proportion in the loss function?
3. Does the choice of lambda significantly influence the performance of the MNO? How to choose the optimal lambda according to the training data?
4. Is it possible to provide a method to judge whether the MNO can be directly applied to a certain target system?

limitations: The MNO can be used only when the dynamical system is Markovian, but actually we do not know whether that assumption holds in a model-free task. This paper lacks a method to judge whether the MNO can be directly applied to the certain target system.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This article discusses robust estimation of chaotic dynamical systems, in particular their invariant statistics, using Markov neural operators (MNOs). In case of simple ODE systems, feedforward NN  instantiate the MNO, while Fourier operators are used for spatially extended (PDE) systems. The major innovation in the present work is the addition of a ‘dissipativity loss’ that enforces attracting behavior within a shell around the data, and thereby tends to enforce globally attracting properties. Essentially, the dissipativity loss tends to penalize larger flow vectors away from the data (why this works was not fully clear to me, however). This enables to learn highly challenging systems like the strongly turbulent Kolmogorov flow.
strengths_and_weaknesses: Strengths:
In general I find this a sophisticated, concisely written and well presented contribution to the field from which I learned various things. This specific idea of enforcing dissipativity seems novel and useful to me.

Weaknesses:
... at the same time, for many systems this global form of dissipativity seems to be a quite strong and often too restrictive assumption, see my detailed Q below.
Much related lit. is not discussed, some methodological details are missing, for some results only single examples but no stats are presented.
questions: 1) If I understood correctly, a crucial premise of this approach is that there is indeed only one globally attracting limit set. This will be violated in most biological systems at least, for instance in molecular biology or neuroscience, as well as in many other complex dynamical systems of current interest, like climate models. For many real-world data from complex systems we would not know this in advance. Isn’t this a serious limitation of the approach?
Also, how does the vector field far from the attractor actually compare to that of the true system under this approach?

2) I felt that a lot of relevant literature was left aside (all the work by the Brunton group or Edward Ott’s group for instance, https://arxiv.org/abs/2201.05136, https://arxiv.org/abs/1904.02107, https://arxiv.org/abs/1710.07313, https://arxiv.org/abs/2110.07238, https://arxiv.org/abs/1712.09707, https://arxiv.org/abs/1910.03471, https://arxiv.org/abs/2110.05266, https://arxiv.org/abs/2005.10756, https://arxiv.org/abs/1707.01146, https://arxiv.org/abs/2207.02542, or the recent developments surrounding neural ODEs and PDEs, e.g. https://iopscience.iop.org/article/10.1088/1367-2630/abeb90). The authors are not the first to put the focus on invariant statistics; power spectra, Lyapunov spectra, or geometrical statistics have been discussed before in the context of reconstructing dynamical systems. Likewise, although the novelty about Theorem 1 may be its extension to spatio-temporal operators (function spaces), similar ideas I believe appeared before in the works of Funahashi and Nakamura or, more recently, by Raginsky and colleagues (https://proceedings.mlr.press/v120/hanson20a.html and references therein). Also, much stronger competitors are likely to be found among these previous methods than the plain LSTMs and GRUs used in Fig. 4 or the models used in Table 1. Many groups (like Brunton or Vlachas) have tested their approaches on similar systems like the Lorenz-63, Kuramoto-Sivashinsky or Navier-Stokes systems. These, in my mind, would have been the more relevant comparisons. This is especially relevant when the claim is made that other ML approaches fail to capture invariant statistics (lines 255-256).

3) If I understood correctly, the specific regularization solution is partly also motivated by the fact that iterating the MNO across many time steps (eq. 3) may easily lead to divergence. But this it seems to me is really a standard problem in RNN training, especially if the loss is only based on consecutive time steps. Specific algorithms to deal with this issue had been suggested before in the context of dynamical systems, e.g. https://arxiv.org/abs/2110.07238. Would be nice to know how these approaches differ in performance \& properties.

4) More methodological and algorithmic details in the main text would be appreciated. What is the loss function, apart from the novel regularization term? What precisely is the setup and parameterization of the forward NNs and the Fourier NOs, how were hyper-parameters of the architectures determined (especially $\lambda$ in the  dissipativity loss)?

Minor points:

- In my understanding RNNs also learn a Markov operator that is used as in eq. 3, what is the difference?

- Fig. 3 provides just single examples, likewise it appears Fig. 4. It would be good to see statistically more comprehensive evaluations with error bands on many different training runs. Also, it would be  good to see reconstructions of the actual attractors as well (the Lorenz is barely visible in Fig. 3). 

- I found it somewhat difficult to follow parts of sect. 4.2 and Fig. 4, simply cos in my mind too few details were provided. What, for instance, is the ‘residual model’?

limitations: The major limitation I pointed out above was not discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a machine learning framework (MNO) by using neural networks or Fourier neural operators to learn the underlying Markov operator of dissipative chaotic systems. Numerical results show that MNO has lower loss and outperforms the other neural network models, including U-Net, LSTM-CNN, and GRU.
strengths_and_weaknesses: This paper proposes to learn the underlying Markov operator of dissipative chaotic systems using deep learning. Several architectures are investigated in detail. The results is interesting, however, I'm not sure whether learning Markov operator of dissipative chaotic systems is relevant for deep learning. In my view, this paper is more suitable for some dynamical journals, such as SIAM Journal on Applied Dynamical Systems, Chaos, ... .
questions: 1. Could you provide the formula of Sobolev Loss and Dissipativity Loss in Figure 2. I guess that the Dissipativity Loss is given by Equation 5. Then, is the ground truth exactly match the Equation 5? If not, does "lower loss" make sense? I think it might be more appropriate to treat it as a regular term.

2. How about using continuous models to learn continuous dynamics？For example, Neural ODE or its structure-preserving extensions [1]. In [1], they also consider learning Lorenz-63 system. For infinite dimensional PDE systems, in my view, we can directly replace the neural network in a continuous model by a neural operator to obtain a infinite continuous model. 

[1] Haijun. Yu, Xinyuan. Tian, Weinan. E, Qianxiao. Li, Onsagernet: Learning stable and interpretable dynamics using a generalized onsager principle, PHYSICAL REVIEW FLUIDS 6, 114402 (2021)

limitations: Yes, the authors addressed the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This is a very interesting paper and addresses an important challenge in terms of long-term stability of high-dimensional turbulent flow. The authors show that adding a dissipative term in the loss and training it with sobolev loss would improve the stability of the autoregressive data-driven models (here they show that a Markov neural operator) is the best skeleton model. 

strengths_and_weaknesses: The major strength of this paper is that it addresses a difficult problem in the physical sciences that have a lot of stakeholders and look at pertinent and physically justified metrics to support their claim. 

However, the major weakness of this work is that 2 of the 3 systems considered are not realistic enough to show issues with instability (the problem at hand) and the 3rd system is also not the best realistic case. Other papers have looked at more complex systems. The weakness still does not take away a very good set of analysis and transparent results that the authors have shown. Still it needs some work before getting accepted. 
questions: The paper discusses and investigates realistic metrics pertinent to the problem and I've thoroughly enjoyed reading it. However, there are still some questions/clarifications that remain for the authors to justify their claims. Herein, I summarize some of the main issues that I find. Beyond that, there is a large class of literature in weather/climate modeling and fluid dynamics that talks about these issues at large; I suggest that the authors look at some of the data-driven weather prediction papers that talk about long-term instability. Adding that context to this paper would make it even more meaningful for readers who would actually care about why this is an important problem for the physical sciences. 


1. Line 2. -- The issue is not just with time-stepping, but with the fact that because of a positive Lyapunov exponent the initial perturbation grows exponentially. 

2. Line 19-20. There seems to be other work (one at least) that looks at long-term statistics of chaotic systems, e.g. PDFs including the tails that capture extreme events. Even more so, it turns out RCs can do both short- and long-term seamlessly. https://npg.copernicus.org/articles/27/373/2020/. Surely, there could be other papers that do the same. The authors' statement is not accurate here. They are however correct when they say that RCs definitely do not scale for more complex problems. Even so, the authors are still looking at toy systems that RCs excel at (except for the NS case). 

3. Line 25-26 What's the rationale behind calling it "ill-posed", citations to relevant papers are essential. 

4. Line 40. Typo. charges->changes

5. For Lorenz 63, simpler methods are long-term stable, even for KS, and for other more complex Lorenz-type systems as shown above. The first two examples are not interesting or relevant when we want to talk about long-term statistics. 2D Kolomogrov is a great example, and there are many other relevant problems that have been studied for long-term stability along those lines, e.g., QG turbulence, other climate models, and real weather data. There is a big community working on this specific problem. The authors need to look at that literature:
(i) https://arxiv.org/abs/2205.04601
(ii) https://arxiv.org/abs/2202.11214
(ii) https://arxiv.org/abs/2202.07575
(iv) https://dl.acm.org/doi/10.1145/3429309.3429325
(v) https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2020MS002109


In the purely ML community, there are a few other papers as well (who tried and improved stability but eventually couldn't make it infinitely stable) for Kolomogrov turbulence (more complicated cases than the current paper). The manuscript would benefit from putting the problem in the context of this bigger goal. 
 https://arxiv.org/abs/2112.15275

6. The effect of "h" is very interesting in the KS problem. There are a few other papers that have found results that might be contradicting this specific paper, or maybe not if I have misunderstood. But either way, that's good. It generates discussion and the importance of exploring this problem more carefully. However, I feel that the authors should acknowledge those papers as well. While not done in the context of operator modeling, they have been done on more complex, realistic, and practical problems of chaos and turbulence. Here are a few of those papers :

(i) https://royalsocietypublishing.org/doi/10.1098/rsta.2021.0200 (Section 2)
(ii) https://agupubs.onlinelibrary.wiley.com/doi/epdf/10.1029/2020MS002203 (Figure 2)
(ii) https://gmd.copernicus.org/articles/15/2221/2022/ (Section 4.3)

7. I appreciate how the authors interpret stability as something beyond just running the model and not getting "NaN". Stability means that the flow is both physical and non-drifting and has the right variability. However, I am still not 100% convinced that this model can serve as a real long-term stable model for a real complex flow. Here are a few things the authors can show to remove my doubts:
    --- While the model captures the spectrum quite well, it deviates in the high-wavenumbers, an essential reason for instability (Figure 4b). Can the authors show how the spectrum deviates as we move forward in time? Basically, take the best model and plot the output's spectrum at various time steps during prediction and compare it with the truth. Furthermore, I don't think that KS is that complicated a system, wherein "spectral bias" (the fact that you lose high wavenumbers as you integrate in time) is that big of a problem. However, in the N-S case with high Re, this would become a definite problem especially because you have inverse cascade. Also, for KS, I think Pathak et al, PRL, had long-term statistics, without doing anything? Does this KS have the same dimension or is it higher?
   ----  It's great that the authors look at the PC1 autocorrelation curve. It must be pointed out that the authors' model does not capture it, or am I mistaken? I also understand that it's difficult to capture PC1 autocorrelation in KS because, in reality, I don't think PC1 in this system has any physical meaning (e.g., in a real system like Earth's climate PC1 has connections to ENSO and so on..). Still the fact that the model doesn't go absolutely haywire is promising. Yet, the authors should clarify this
  ---  I don't think the PDF is also captured correctly. Please plot PDF in semilog, so that the tails (extreme events) are more prominent, that's what matters. But here, I think even the bulk is wrong in some portions. It's still impressive that you have the right range of values. 

8.  For the N-S case, I have some major confusion. In the main paper (not the appendix), the spectrum is not plotted. In the appendix, I see some images with Re=40. Re=40 is not nearly turbulent enough. That's hardly interesting in the context of turbulence and it's long-term predictability. Even Re=500 is not a realistic set-up. I must say here, that I understand the difficulty to scale up the turbulent flow to high Re, it's both difficult computationally, and probably very hard to predict. But based on the context of the problem, if this model really is going to serve practical problems in terms of long-term stability, the 2D N-S case should have Re=10000 or higher. That's the regime when we have all the complexities of this flow including when the effect of inverse cascade would be prominent. Here're a few papers that look at NS cases in realistic regimes.  
    (i) https://www.pnas.org/doi/10.1073/pnas.2101784118
    (ii) https://www.sciencedirect.com/science/article/pii/S0021999122001528
    (iii) https://www.cambridge.org/core/journals/journal-of-fluid-mechanics/article/abs/subgrid-modelling-for-twodimensional-turbulence-using-neural-networks/10EDED1AEAA52C35F3E3A3BB6DC218C1

9. For the N-S case, the energy spectrum at different time steps needs to be shown for the best model and the baseline. That would clearly show how well the spectrum is captured. It would also only be interesting for a case where the flow is turbulent (Re>10000)

10. Again, the PDFs should be shown in semilog plots to highlight the tails. Also, what's going on with the PDF of KE for the N-S case? It's not really getting captured right? In this case, as well, there's no dominant mode of variability, so PC1 does not have any meaning right? But still, plot the true PC1 autocorrelation plot to show how well/poorly it's captured. 

11. The authors might benefit from looking at another system. A system that has a dominant mode and coherent structures while being turbulent enough. I think the discussion around stability would have far more importance there with PC1, autocorr, etc even if they are not perfectly captured. 

Finally, despite my long review and many questions, I enjoyed reading the paper. The authors have done a great job in addressing the question of stability "correctly"  by looking at the right set of metrics. While there are these lingering questions, once properly addressed this would be an important contribution to the ML and physics community. 
limitations: N/A
review_title: null

Meta-review: metareview: This paper proposes a neural network-based approach to estimate the Markov operator of dissipative chaotic systems. It introduces a novel combination of Sobolev and dissipativity losses. While the reviewers had initial concerns about clarity, assumption and application condition, and the choice of learning Markov operator versus modelling continuous dynamics, the author-reviewer discussion addressed most concerns, and all reviewers agree this work exceeds the bar for publication.

I would encourage the authors to take into consideration the remaining concerns from the reviewers, incorporate key conclusions of the discussions and the limitation of the work in their final version.
confidence: Certain
Decision: Accept