Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an extension of Fourier neural operators (FNO) to have operators evolving over time. The paper proposed an architecture to cooperate the time embedding into FNO. The experiments are conducted on synthetic datasets given PDEs and on real-world time series data. The model outperforms alternative methods. There is also an experiment on the robustness to adversarial attacks.
strength_and_weaknesses: ### Strength
- The paper is well-written, and well-structured.
- Theoretical results support the claim on the stability of the proposed method.
- Experiments are comprehensive, covering multiple aspects from learning dynamics and robustness to adversarial attack.
### Weaknesses
- Although the suggested time modulation shows impressive performance, it’s hard to justify why it works well with this specific choice (in Table 1). I’m not familiar with StyleGAN2. So if there is any good justification from StyleGAN2, it would be nice to have a brief discussion of the benefits of such an architecture design.

clarity,_quality,_novelty_and_reproducibility: The means and standard deviations of results in the paper should be provided.
There are some typos but can be improved.

summary_of_the_review: The paper did a good job describing the proposed method and the emperical results are impressive. However, it's not clear what factor leads to the effectiveness of the proposed models.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Similar to as is done in done in diffusion models for U-net layers, this paper applies time-conditioning to FNO layers and demonstrates it applicability on a wide variety of tasks. The paper also proposes a method to normalize the Fourier kernel so that it's more perturbation resistant.
strength_and_weaknesses: **Strengths**

Time-conditioning in the Fourier space as done in this paper is definitely novel. Similarly row-wise normalizing the Fourier kernel is a useful trick to know. The paper applies their method on a very wide variety of problems, demonstrating the general applicability of their approach.


**Weaknesses**

Claims abut lack of expressiveness of ODEs in the second introductory paragraph are incorrect. NeuralODEs are difficult to train in practice on higher dimension problems and there have been many different approaches to try fix some of these issues in various individual contexts. S4 [1,2] class of methods actually try to solve this problem generally while being computationally less expensive. Proposing a "PDE" based framework seems ill-motivated and calling FNO as a PDE solver is also incorrect. A PDE solver will solve the given equation at whatever timestep you want answers for, FNOs first need to be trained on solutions from that PDE solver to make a "PDE surrogate" and can still only "solve" that particular equation and will not generalize to change in the parameter of that equation. Similarly as this paper notices too that FNO has a notion of discrete time and is often trained to recursively make predictions and can't give you predictions at whatever timestep you want. I know the literature in this space is a mess and I don't blame the authors per se but some precision in our language is required for effectively communicating with the differential equations community if you are supposedly solving problems with their approaches.

Mention of U-net based diffusion models is another remiss. There too scalar time is used to condition the different layers of neural network in a manner that is not _too_ different from what this paper proposes. [3] demonstrates how applying the same techniques and architecture is quite effective for PDE modeling (orders of magnitude better than FNO) and also does time-conditioning similar to this paper. 
Overall, expressiveness of FNOs is the reason for superior performance and more details about training strategies and baselines is necessary to actually make sense of results.

In the Image classification experiments it's unclear what CNN model was used as baseline and there are methods to improve CNN style model accuracies too which would ideally be compared as well. I'm not sure if it even should be in main text of the paper, it just seems so disjoint from the title. Moreover it's unclear if this "stabilization" won't hurt performance on difficult 2D PDE modeling tasks like Navier-Stokes.

[1] https://arxiv.org/abs/2111.00396

[2] https://arxiv.org/abs/2206.03398

[3] https://arxiv.org/abs/2209.15616
clarity,_quality,_novelty_and_reproducibility: There are a few typos in the paper. Core idea of conditioning neural-net layers with time is fairly commonly done in diffusion models, transformers, NeRFs etc. these days. However how to do it w.r.t. Fourier layers is new here. Provided code is easy to follow and the results should be easy to reproduce.
summary_of_the_review: Overall there are useful ideas in the paper. However it needs better focus on its identified contributions. Clear discussion of limitations would also be quite useful.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A continuous-time version of NFO is presented, which can learn both ODEs and PDEs. 

strength_and_weaknesses: s: Fantastic results.

s: perhaps a principled method (maybe, not sure)

w: Weak clarity. I didn't understand the method, and hence I can't evaluate it if has value. The paper follows the convention where equations are given, but not explained, exposed or illustrated. The reader is assumed to be an expert in this domain, which limits the papers audience and thus impact. For instance, eq1 is already weakly presented, but afterwards the computation flow, Nemytskii and FNO are gibberish to me. Similarly eq 4 is just given as a blurp with no explanations. I couldn't follow the time modulations. I didn't understand the ablations: why do we care about this? (The stability stuff was very clear). Furthermore, the model itself is not even defined (!), except in fig2 that I can't follow. In experiments I couldn't follow what task was being solved, and whether table 2 was training or test errors, or what the percentages or pred/interp. settings meant in table 3.

w: Poor motivation. The sec 3.1. explains motivation, but I can’t follow what problem is being solved. Apparently ODEs have some issues fitting toy cases, but then we move to PDEs instead (without citing neural PDE works, or comparing to them, eg. Iakovlev'21). The main motivation is perhaps having a model that can handle both ODE and PDEs, but it’s unclear why do we want this.

w: poor contextualisation. The results are fantastic, but there is little contextualisation on how this method differs from others, or what are its limitations. Does this method solve now everything, does it have any weaknesses?
clarity,_quality,_novelty_and_reproducibility: 
Clarity: weak
Quality: High
Novelty: Unclear
Reproducibility: poor, I couldn’t code this.

summary_of_the_review: This paper presents an *DE method that achieves amazing results, but does practically nothing to explain either what is happening or what is being learned or why the method is so good. I don’t think just getting SOTA results is science, one also needs to provide insights or the "why".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an extension to the Fourier Neural Operator, called CTFNO that can continuously condition on time, including a universal approximation theorem for the new architecture, and a stability guarantee that should defend against noisy observations. The conditioning on the time is achieved via the adoption of a positional embedding as used in Transformers (Vaswani et al., 2017), and added block wise via shared networks both to the spatial and to the frequency domain.
strength_and_weaknesses: Strength:
The paper addresses an important topic, namely that many neural PDE surrogates are trained on fixed time grids. Whereas DeepONets like operator learning methods have time dependencies naturally encoded via the trunk net, Fourier Neural Operators do not.

The Gershgorin disc normalization is an interesting way of enforcing stability. 


Weaknesses:
Under the hood, the newly proposed architecture conditions on the time variable. Conditioning is a super important property in various computer vision tasks and natural language translation tasks. As such a lot of different techniques have been developed:  for example, the encoding of the input into a vector representation by using sinusoidal Fourier embeddings as is common in Transformers, or AdaGN (Nichol et al., 2021), which is based on affine transformations of normalization layers via projections of the embeddings. I am not fully convinced with the three presented baseline approaches. They read rather week, but ok, it might just be really hard to condition for FNOs. A different why of encoding parameters in the Fourier space can be found e.g. in Poli et al. Potentially also relevant is https://openreview.net/forum?id=Uk40pC45YJG

Experiment-wise I am missing how the new time modulation improves stability. I would have liked to see long rollouts, and the conditioning of different time windows.  In that sense I have also waited for studies which address the effect of the Gershgorin disc normalization. That is missing in the current version and makes me wonder what the whole theoretical reasoning is about.

Also experiment-wise for e.g. heat equation and Burgers’ equation it is not 100% clear to me what is the input to the neural network surrogate and what is the output, and where the conditioning on time comes into the game. I would have expected curves showing errors for e.g. 100 step rollouts, 200 step rollouts, and so on. Also I am missing how FNO2d is used (input and output?)  I am also missing parameter and runtime comparisons. All in all that makes it really hard to judge the experiments.

Nichol, Alexander Quinn, and Prafulla Dhariwal. "Improved denoising diffusion probabilistic models." International Conference on Machine Learning. PMLR, 2021.
Poli, Michael, et al. "Transform Once: Efficient Operator Learning in Frequency Domain." ICML 2022 2nd AI for Science Workshop. 2022

clarity,_quality,_novelty_and_reproducibility: The paper is written relatively clear, although a red line is missing a bit. There is a lot emphasis on stability and the Gershgin discs normalization, discussion of which is missing in the experiments. I still missing the real intuition why the MuJoCo experiments are done. Altogether there seems to be enough novel and important ideas. In the current version, the experiments are not reproducible, important information and description is missing, 
summary_of_the_review: The ideas are novel and interesting. The current way of how the experiments are presented and compared are weak and hard to judge. I recommend the authors to improve upon the experiments section!
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new neural architecture (called continuous-in-time Fourier Neural Operator, CTFNO), which is based on the idea that the Green function for a (linear PDE) should depend on time. Numerical examples are shown. 

Strengths: Time conditioning network in the FNO is novel, as well as normalization trick by the Gerschgorin circles. The numerical experiments are quite promising.

Weaknesses: The paper is not well-written and is difficult to follow. As pointed by one of the reviewers, it is not authors fault: it is a mess on the FNO literature nowdays. I.e., the standard FNO approach is not even a mapping from a function to a function (see Fanaskov, Oseledets, "Spectral Neural Operators": the non-linearity in the physical space introduces aliasing and additional harmonics), 
so the theorems on the universality of the approach are very questionable. 
Second, the sloppiness of the method: the PDE mentioned is written in the general form, whereas the Green function is written with the shift-invariant case, and only holds for the linear operators $\mathcal{L}$ (since the layer is linear). So, the whole motivational part should be presented in a different way in my opinion. 

The main missing thing in the submission is a simple example that illustrates the case where the time-dependent method will work (significantly) better. Maybe a simple time-series, but the current informal style of writing, together with a specific, quite complicated time modulation architecture, does not make me believe in the new approach.

justification_for_why_not_higher_score: The main reason is that the paper is not well-written. It uses very informal (even annoying language): i.e. writing a general operator and writing the solution as convolution with a shift-invariant, time dependent function does not help. Why shift-invariance (none of the example have this property). Why not general G(t, x, y)? The main efficiency is probably in the architecture design, but it is not clear when it will work. A simple example is needed (maybe even a simple time series).
Decision: Reject