Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a continuous space-time data-driven model for predicting the spatiotemporal evolution of physical phenomena driven by PDE. The method embeds spatial observations independently of their discretization via Implicit Neural Representations and then models continuous-time evolution by a learned latent ODE. The paper claims it can learn from sparse irregular grids or manifolds and extrapolate at arbitrary spatial and temporal locations. 
strength_and_weaknesses: Strength:
The problem is interesting and challenging.  The forecasting model for continuous space-time is important for many real-world tasks.
The model does not rely on fixed discretization on training and test data and can extrapolate at arbitrary spatial and temporal locations.
The extensive experiments on the simulated dataset validate the model’s performance in modeling the evolution of the dynamic system.  

Weakness:
The model decoder only uses the current latent dynamic temporal feature (alpha_t) and ignores previous spatial information. On different spatial locations, the decoder prediction is also independent, which ignores the spatial autocorrelation among samples. 
Some baseline for spatiotemporal interpolation and extrapolation with common machine learning models is lacking, such as the Gaussian process. 
The bi-level optimization training for the dynamic model and auto-decoder seems hard to converge. Some analysis on the training and convergence is needed. 
More evaluations on some real-world spatiotemporal datasets like weather forecasting, air quality forecasting would be interesting to show the generalization capability on real dynamic problems. 

clarity,_quality,_novelty_and_reproducibility: The paper writing is clear and the quality is good. The dataset is publicly available.  The experiments setup and parameters are provided in the paper and appendix. The data and code is available in an anonymous link. 

summary_of_the_review: The paper proposes an encoder-decoder framework to address the continuous space-time data-driven model for predicting spatiotemporal evolution, which is important and challenging. The extensive experiments on the simulated dataset validate the model’s performance and the capability to extrapolate at arbitrary spatial and temporal locations. However, there are several concerns to be addressed:

1. The ignorance of spatial autocorrelation in the framework.
2. There is no analysis of the training convergence. How to ensure the encoded feature reflect the spatial 
Real-world spatiotemporal forecasting evaluation would be more interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes DINO, a model that generalizes to arbitrary spatial and temporal resolutions, beyond the spatial and temporal samples in training. This is achieved via a combination of autodecoding, dynamics model, and hypernetwork with amplitude modulation. It shows in experiment that DINO achieves in general superior performance in generalizing to novel time interval and spatial resolutions, compared to strong baselines.
strength_and_weaknesses: Strength:
The paper is well motivated and clearly written. The method is novel, and the experiment evaluation is thorough. The quality of the paper is very high.

Weaknesses:
1. It is not clear how the method's error accumulate in the long-term evolution for out-t scenario. The long-term evolution is a key challenge in neural-based solvers. A model with excellent short-term error does not necessarily mean a good long-term error (since the error can accumulate and the model can overfit to short-term behavior.

Therefore, it would be nice and strengthen the paper, if the authors show how different models behave where the out-t is range is significantly greater than the in-t (e.g. 5 times, 10 times, 50 times the length). A figure showing the error accumulation is also nice.

2. It is not clear how time consuming is the method. The auto-decoding may need multiple steps to infer the correct alpha. In addition, the latent evolution may be more time consuming that regular time intervals. It would be great if the authors can report the runtime (in seconds) of each method.
2. The author mentioned FNO in the introduction. it would be nice to compare with the proposed method with the strong baseline of FNO.
clarity,_quality,_novelty_and_reproducibility: Clarity: excellent

Quality: excellent

Novelty: good

Reproducibility: good, it provides the code. It would be nice to be more clear about the in-t and out-t range.
summary_of_the_review: In summary, the paper is well-motivated, well-written, novel, and solid (through thorough experiments). The paper would be strengthened if the weaknesses is addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper "Continuous PDE Dynamics Forecasting with Implicit Neural Representations" presents a space and time continuous model for forecasting. This is achieved by coupling an autodecoder to estimate a latent state, a hypernetwork-based emmission model that parameterizes an implicit neural network, and by modeling the devleopment of the latent via a learned flow with an ODE. The method, termed DINo, is compared to baselines on three datasets (2D wave equation, 2D Navier Stokes, and 3D spherical shallow water), and shown to outperform other methods in the flexible setting of changing sampling grids.

strength_and_weaknesses: trengths.
- the paper is very well written, and easy to understand.
- the topic of continuous-time, space-continous modeling is theoretically interesting, and offers more generality than standard machine learning approaches on a grid - hence, is very interesting to the community in my opinion.
- a novel method is worked out, which makes intutive sense. In particalar the proposed separation of time & space, termed amplitude modulation, seems to improve performance.

Weakenesses.
- all datasets are somewhat synthetic. This offers a lot of possiblity to investigate the method (which the authors do), but showing performance on a real-world dataset would significantly strengthen the paper (e.g. on a weather foracasting task, or on a multi-variate time-series task, e.g. from e-Commerce).
- limitations of the method should be worked out clearer: e.g. if I understand correctly, the method cannot easily integrate information from observing several time-points? Another one: the MSE loss on the latents implies uni-modality (a single Gaussian), and determinism - can the model handle multi-modalities and stochasticities in the forecasts?

Small things:
- in several places you write 'extrapolation' (e.g. abstract, p.2. ...). However, while the method can perform extrapolation well in time, due to the ODE, it probably performs not well on extrapolation in space. The case of changing the grid to points within the convex hull of seen ones (e.g. superresolution etc.), I'd consider to be an interpolation task. Hence, please use 'interpolation' for the spatial tasks in the paper.
- Figure 4: from eq. 9 it seems b and \mu are similar - there should be also a plut between W and b then. Also, the z_ts point to b, while in the formula only W is modulated by z.

UPDATE: 
Thanks for the changes made to the paper, which nicely removes some of the limitations, and adds a real-world dataset (although only in the appendix). I upgraded the score accordingly.

clarity,_quality,_novelty_and_reproducibility: - as mentioned the paper is clearly written and of high quality
- as far as I know the detailed approach is novel
- a link to PyTorch code is provided, and details are described, making the model reproducible

summary_of_the_review: The paper is well written, and proposes a novel approach in an interesting domain. The biggest weakness is the lack of comparison on a real-world task.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces DINo, a continuous time and space solver for partial differential equations. The authors proposed to model the solution of the PDE as an implicit neural representation conditioned on the initial condition through an hyper-network. The dynamics is modeled as a learned latent ordinary differential equation.
strength_and_weaknesses: I really enjoyed this paper. Forecasting solutions of PDEs in a continuous manner is a very interesting topic yet very challenging. Such techniques may alleviate the need of heaving meshing procedure in standard physics solver. The proposed approach seems reasonable: while using INR for modeling continuous solutions seems the natural thing to do, conditioning its parameters to the initial condition is not straightforward. Amplitude modulation seems like a very clever and interesting method to tackle the problem. However, I may need additional precisions to fully understand the contribution. 

- I do understand the advantages of using FourierNet with the chosen architecture for the hyper-network in theory. However, I wonder if such gain is verified in practice. For example, could it be possible to replace FourierNet by SIREN or MFN conditioned with an hyper-network (and not directly through the input) ? Could the authors briefly discuss about the other structure they tried ? I have the same kind of question for the auto-decoder: is it actually beneficial over an explicit learned encoder model ?

- I am not absolutely convinced by the "separation of variable" argument in section 4.4. It is not clear to me how separation of variable is relevant in this situation. This technique allows to disentangle a PDE in a set of ODEs, which are easier to solve. Yet, in this framework, the PDE seems to be directly translated into a ODE in the latent space $\alpha_t$, where the dynamics solely depends on the time derivatives. I would appreciate if the authors could clarify this statement.

- Related to my previous remark, I am not sure how the space derivatives (which are needed to solve the PDE) can be embedded in the latent vector $\alpha_t$ such that the entire dynamics can be resumed as a temporal ODE. Does the authors have any insights about what is happening here ? May the auto-decoder somehow pre-computes spatial derivatives (even if the observation is partial) and embed them in a convenient form in $\alpha_t$ ?

I also acknowledge the release of the code, which actually helps me to answer to more technical question about the model (especially, how the partial observation is inputted to the model). However, I think that appendix section D could be more precise.

My most important questions focuses on the experiments. I do appreciate the comprehensive study conducted by the authors, with relevant and numerous baselines, and extensive analysis on the generalization capacity of DINo. Yet, I am still a bit skeptical about the chosen datasets, for three reasons, ordered below in ascending order of importance:

- The main results (table 2 and 3) are computed on uniform grids. I do not see any aspect of DINo that are not compatible with uniform meshes, which are much more powerful and widely used to solve PDEs, especially Navier-Stokes equations. However, uniform grid contains a lot of regularities that both DINo and the baselines may leveraged. I wonder how the proposed method would behave on irregular triangular mesh with complex physics such as the ones introduced in [1].

- Both qualitatives and quantitatives results may indicates that the task on which DINo is evaluated are actually saturated. My doubts are based on the very close MSE obtained in the standard setup $s=100$% and the merely indistinguishable difference between figure 6 and 7. This hypothesis could quickly be rejected by showing additional prediction example from both DINo and I-MP-PDE, maybe with some failure cases. This seems important: if the tasks are saturated, it is unclear how such tasks are relevant for assessing DINo generalization capacities.

- I was quite amazed by the performances of DINo in the 5% subsampling scenario. According to me, these results, and actually many others, may be explained by one the three following hypothesis:
     1. DINo can leverage knowledge for the PDE dynamics to extrapolate un-observed information. 
     2. DINo exploits biaises in the data to infer the state in the entire space.
     3. Datasets are not sufficiently complex and basic interpolation is enough.
In my opinion, (3) is highly possible, as I-MP-PDE seems to perform competitively to DINo in the 5% setup. (3) may be easily rejected by comparing the reconstruction error of DINo on $v_0$ with direct bi-cubic interpolation from the partial observations. If the error on the latter is bigger than the error on the former, then DINo does leverages knowledge either from physics or from regularities in the dataset. Checking the validity of (2) is much more challenging. Maybe the easiest way could be to look at a wider set of trajectories from the dataset and make sure to observe sensible diversity among them. I would really appreciate if the authors could provide an in-depth discussion concerning these points.

I also wonder why I-MP-PDE is absent from table 3, it seems to me that it could be compared with DINo in this setup as well. 


[1] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. Learning mesh-
based simulation with graph networks. In International Conference on Learning Representations
(ICLR), 2021
clarity,_quality,_novelty_and_reproducibility: The paper is well written and seems novel. Reproducibility is possible as the authors released the code. Yet, some additional details could be provided in the appendix.
summary_of_the_review: I really enjoyed the paper and find the overall model architecture quite clever. However, I still have some doubts about the performances of the model compared to more trivial interpolation techniques combined with existing methods. The paper provides a quite extensive comparison with SOTA, yet I am not sure that the chosen task are actually relevant to correctly assessed the model. If the authors provides satisfying answers to my questions, I will be very pleased to increase my grade.

## EDIT
The authors provided very detailed and convincing answers, which remove my fears on this paper. I now think that this work deserves to be presented at ICLR. I still have some concern about the practical performance of the model. Although the datasets are perfectly within the standards of the community, these tasks are already relatively well mastered. I would have given the maximum score if the technical contribution (definitely respectable) had been completed with new results on more difficult datasets than those conventionally used in the state of the art.

It is nevertheless a work of excellent quality, which opens the door to new approaches for the resolution of PDE, and which, I believe, deserves the attention of the community. I increased my rating from 6 to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is proposing a data-driven forecasting for solution for physical models. The main contribution is using implicit models to extend existing (ir)regular grids to continous domains effectively enabling arbitrary and changing sampling grids for both spatial and temporal domain. To handle the complexity hypernetworks using multiplicating fourier blocks is proposed with seperation of variables enabling effective learning. The model is later tested extensively on a large collection of datasets. The paper was reviewed by 4 experts all uniformly agreed on the merits of the paper. Authors addressed the reviews and there was an active discussion. I strongly recommend authors to go over this lively discussion and integrate them to the paper as much as possible before the camera ready deadline.
justification_for_why_not_higher_score: Although the paper has a clear merit to be published and presented to the community, there are still reasons which stops me from giving an oral designation. The resulting models are still much smaller scale than many PDEs tackled in real systems. Hence, although the work is interesting its scaling might still be an issue. Moreover, there is no discussion on computational efficiency. These make me question general applicability.
justification_for_why_not_lower_score: The paper has clearly merit to be published in the conference. Moreover, it has a significant contribution to the larger community of ML for physics/ML for science. Hence, it has merit to be a spotlight paper and shared with the community as a short presentation.
Decision: Accept: notable-top-25%