Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces the neural stochastic PDE model to learn solution operators of PDEs, extending previous frameworks of neural CDEs and neural operators. The authors introduce two ways of evaluating the model, based on either solving a system of ODEs, or reformulating to a fixed point problem. The authors perform experiments on multiple stochastic PDE problems, showing that their neural stochastic PDE model is superior over previous baseline models in learning such dynamics. 
strengths_and_weaknesses: Unfortunately, the paper is well outside of my expertise (due to a potential mismatch in the system), and I really do not have the relevant PDE background to analyze the techniques used in the paper. 

Aside from the technical parts, I find the structure of the paper pretty clear - the authors provided good motivations for their method, and derived the theory for evaluating the model in an organized fashion. It would be nicer if the authors can summarize their final algorithm in an algorithm block, which would make it much easier for readers to understand the key steps. 
questions: N/A
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a method to elucidate noise-conditioned spatiotemporal dynamics via the framework of Stochastic Partial Differential Equations (SPDEs). It consists in learning neural-parameterized SPDEs to fit the observed phenomenon thanks to the adaptation of two standard Fourier-based resolution methods used to handle the neural SPDEs (an ODE solver in the Fourier space and a fixed-point method). This allows the proposed model to work on arbitrary space and time resolutions.

The model is then evaluated against several neural operators and neural CDE baselines to solve standard SPDEs in three contexts: based on the initial condition only, conditioned on the intrinsic noise information, and both at the same time. The model especially shows significant improvement w.r.t. the considered baselines on the two latter settings.
strengths_and_weaknesses: ### Strengths

The paper presents, to my knowledge, a **well-motivated, novel model** for a **relevant task**, since handling an underlying noise in the observed dynamics remains a challenge for neural PDE-solving methods. **Numerical results support the relevance** of the proposed model, with the latter being able to accurately predict the studied phenomena, unlike tested baselines. Its **flexibility** is also appreciated, with its two different versions, its ability to handle multiple output resolutions and its versatility in the experiments.

### Weaknesses

I have three main concerns about this paper that are significant enough to nuance my positive recommendation.

Firstly, **the significance of both the model and its results remains unclear**. This is in part due to its SPDE inspiration: the model is specifically designed to integrate the complete noise information in its predictions, but this use case seems limited. Indeed, accessing this noise may only be possible in synthetic controlled settings such as the conducted experiments in the paper. Assessing whether the model is able to integrate partially observed noise would greatly benefit the paper in this regard. Nonetheless, given the underperformance of the model in the absence of noise information, I suspect that this may be challenging for the model as currently presented here.

Secondly, I find that **the evaluation is lacking in some respects**.
 - Some baselines on neural operators might be missing, in particular improvements of FNO: for example, the works of Cao (2021) and Gupta et al. (2021) seem relevant. The evaluation of FNO is also questionable: why is it necessarily unable to handle the $(u_0, \xi) \rightarrow u$ case? Couldn't $u_0$ be appended to $\xi$ at any $t$ as a supplementary information?
 - Moreover, the impact of the two proposed resolution methods is not evaluated: an ablation study (in terms of performance, efficiency, etc.) might highlight their respective merits w.r.t. other standard resolution methods. On this matter, to my understanding, only one of these alternatives (the fixed-point one) is used in the experiments.
 - Finally, I am unsure of the relevance of the task $u_0 \rightarrow u$ in which models are trained and evaluated using a regression objective while the phenomenon is stochastic by nature.

Thirdly, **the writing and presentation of the method should be improved**. The problem is threefold.
 1. The SPDEs are contextualized by a dense background description which is hard to follow for non-experts. Providing more intuitions and simplifying the notations could improve this point: for example, the simplification of lines 148-149 could be done earlier in the document, and examples accompanying the introduction of SPDEs would be welcome.
 2. The presentation of the method in unclear on some points, notably because of the heavy notations but also because important information is scattered across the paper. In particular, Section 3 should clearly state the possible inputs and outputs of the model as well as the learnable parameters (for example, the fact that "Neural SPDE learns a representation of $\mathcal{G}$" is only explicitly stated in Section 4, line 235).
 3. The paper misses a further discussion of related work, either by adding a dedicated section or by specifying in e.g. Section 3 the precise differences with previous work. This would make the original contributions of the paper clearer.

Cao. Choose a Transformer: Fourier or Galerkin. NeurIPS 2021.\
Gupta et al. Multiwavelet-based Operator Learning for Differential Equations. NeurIPS 2021.

### Overall Sentiment

Despite its flaws which are not prohibitive, I believe that this paper introduces an interesting method for the NeurIPS community with appealing performance. Therefore, I am inclined to recommend this paper for acceptance. I am willing to change my evaluation following the discussion phase with the authors and the other reviewers.

**Post-rebuttal update:** The authors' response partially alleviated my concerns (cf. [my answer below](https://openreview.net/forum?id=OGM9dXemmq&noteId=8iHe2W6IV1B)). I consequently raise my score from 5 to 6.
questions:  - Could the authors comment in the ability of the model to handle partially observed stochasticity and the applicability of the considered setting to real-world phenomena?
 - Can the experimental setting be clarified w.r.t. my concerns described above?
limitations: The authors adequately addressed the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The paper introduces Neural Stochastic Partial Differential Equations (NSPDEs) for learning solution operators for SPDEs while being resolution invariant. 
- Two methods of performing inference for NSPDEs are proposed: (1) A method using Fourier Transformations to reduce the NSPDE to a Neural ODE, which can then be solved with existing techniques; (2) a method based on framing the NSPDE as a fixed point problem allowing the use of fixed-point solvers. 
- The model is evaluated on well-known SPDEs and shown to outperform various Neural Differential Equation models as well as the Neural Operators method. 
strengths_and_weaknesses: # Stengths

- The paper proposes a well-executed instantiation of Neural SPDEs continuing the recent research direction of combining deep learning models with various types of differential equations. 
- The two proposed methods for evaluating the NSPDE model provide some degree of choice/flexibility for practitioners. 
- The emprical evaluation shows that the proposed model performs well at predicting the solution of SPDEs. 

# Weaknesses 

- SPDEs are notoriously a very technical subject, however, I believe this is not a justification for the fact that the background section fails (and perhaps does not even attempt) to find the right level of technicality for a machine learning audience. Overall, the background section is not self-contained at all: 
     - The reader is invited to consult a primer on semigroup theory. But no semigroup theory is actually needed to read the main content of the paper. The word semigroup is used only to say that $\mathcal{L}$ is a differential operator generating a semigroup $e^{t\mathcal{L}}$. Why not simply say $\mathcal{L}$ induces a map $e^{t\mathcal{L}}$ and refer to it as a map. If you really want to mention it's a semigroup, you can add a footnote and even define what a semigroup is in that footnote (it's a very short definition). 
     - Line 115 mentions a "mollifier" without ever explaining what that is. Also, what does $\delta^\epsilon * W$ mean? Is that a convolution? 
     - This is followed by a very technical discussion in Lines 116-130 that is of little consequence for understanding the rest of the paper. This discussion could be perhaps moved to the appendix. The space of that paragraph could be saved for further explaining other concepts or adding some figures that help explaining the main concepts. 
     - For Wiener processes, the paper simply references definitions from papers/books. I understand these have been defined properly in the appendix, but I think a short definition or at least an intuitive definition would have been possible in the main text given they are the essence of SPDEs. This would make the main text more self-contained. 
- There is no comparison between the two evaluation methods. First, what is their computational complexity and what are their theoretical advantages and drawbacks? Second, how do they compare in practice / empirically? None of these two types of questions is addressed and from reading the paper, it is unclear which one of the two one should use in various settings. Instead, a minimal set of guidelines backed up by theory and experiments could be provided. 
- None of the tables contain error bars. I see that the checklist says this is because "this would be too expensive for the baseline models NCDE, NRDE and NCDE-FNO". First, the neural operator models do not contain error bars either despite being faster. Second, this claim is not really backed up by data. Could you report the training and evaluation time of those models that made evaluation across multiple seeds difficult/impossible? Realistically speaking, if there was enough compute to run these experiment once, then running them 3 or 5 times should be doable (unless it takes weeks or months?). At least on the systems with one spatial dimensions, I don't think that should be the case. 
- One of the main motivations of the model is that it generalises NCDEs and it can also be applied to irregularly sampled and partially observed data. Correct me if I am wrong, but I believe that none of the experiments in the evaluation section actually consider such a context. Here, I would have expected something like in Figure 1, where data is observed at some sequence of times $t_1, \ldots, t_n$ and the samples might be missing some channels. 

======== EDIT =========

Raised my score to Accept after discussion. 
questions: - See all my questions from the weaknesses section above. 
- I am a bit confused about how $\xi(t, x)$ is supplied to the Neural Operator models and all models more broadly when it is observed. 
   - First of all, what do we assume it is observed more precisely? Do we have full access to the function $\xi(t, x)$ and we can sample it at any point or do we have access to some samples from this function for certain $(t, x)$ pairs. 
   - In the case of FNO, in its original paper, the model takes as input a function over the domain $\mathcal{D}$ and outputs another function over the same domain. As far as I understand, in the setting when $u_0$ is fixed, the input function to the FNO model becomes $\xi(t, x)$. So you are supplying to, FNO samples from $\xi(t, x)$ on a regular grid over $[0, T] \times \mathcal{D}$. Is that correct? 
   - I see that the Neural Operator models were not evaluated in the setting where the input is $(u_0, \xi)$. Couldn't this input be supplyed as function combining these two functions? For instance, the new function could concatenate the two over the spatial domain at $t = 0$ and concatenate $\xi$ with some dummy values for $t > 0$ (or something else along these lines)? 
   
limitations: TLDR: No 
- For limitations, the checklist mentions Section 2 (which is a background section), while Section 5 discusses future directions (not limitations). So I believe limitations have not been properly addressed. 
- Negative societal impacts are not discussed either. I believe this part is treated too superficially despite that fact that PDEs show up everywhere in mechanics and these sort of models could be applied for all sorts of military applications and what not. Therefore, I would invite the authors to put more thought about the potentially dangerous applications of these models.
review_title: null

Meta-review: metareview: This paper provides an extension of Neural PDEs to Stochastic PDE, which is a very important area lying in the intersection between dynamic systems and stochastic analysis. We think the work is well-motivated, novel, and relevant. The author did a good job in the rebuttal to allieviate most of reviewers' concern about the contribution statement, the evaluation, the presentation, and the adaptivity to irregular sampling. We expect the authors revise the paper accordingly in the next version. Please also explain why there are only two baselines in the paper. It would be great to see some discussions about whether the work can shed some lights on learning the unknown SPDEs, beyond solving them.
confidence: Certain
Decision: Accept