Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work the authors propose to implicit neural representations to model the spatial representation, while approaching the temporal representation with classical optimization-based approaches to model time-dependent partial differential equations.
strength_and_weaknesses: Strengths:
- Clarity of the mathematical exposition of the approach
- Clarity in the embedding of the presented approaches intersection and background in PDE- and fluid dynamics theory

Weaknesses:
- Poor integration into the current state-of-the-art by omitting many similar approaches, such as Graph Network Simulations [2] which effectively encode the spatial representation with graph neural networks, and similarly use classical time-integrators for the temporal representation. The same applies for the even more modern MeshGraph [3], and Fourier Neural Operator [4], both would need to be compared to the herein presented approach to properly assess the capabilities of the presented approach. The usage of implicit representations for functions has also been explored by Dupont et al. [1] before. Said work would need to be put in context to the presented literature.
- Improperly chosen comparisons to current state-of-the-art models due to an incomplete representation of the literature, see preceding point
- No ablation analyses
- Performance of the approach induces a 30x overhead, hence making it intractable for any practical problem at the current moment

[1] Dupont, Emilien, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. "From data to functa: Your data point is a function and you can treat it like one." In International Conference on Machine Learning, pp. 5694-5725. PMLR, 2022.
[2] Pfaff, Tobias, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. "Learning Mesh-Based Simulation with Graph Networks." In International Conference on Learning Representations. 2020.
[3] Sanchez-Gonzalez, Alvaro, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. "Learning to simulate complex physics with graph networks." In International Conference on Machine Learning, pp. 8459-8468. PMLR, 2020.
[4] Li, Zongyi, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. "Fourier Neural Operator for Parametric Partial Differential Equations." In International Conference on Learning Representations. 2020.


clarity,_quality,_novelty_and_reproducibility: While the exposition is written with a lot of clarity, and does have a very high quality at first glance, it displays a severe lack of nuance in its relation to current work to properly assess its originality. While the present paper still has novelty in aspects, the novel arises from the implicit spatial representations, which are also viewed from the viewpoint of implicit neural representations. Previous approaches such as Graph Network Simulations, and MeshGraphNets have already followed a highly similar approach, where one can surmise that under certain conditions the Graph Network Simulations approach of spatial graph network representations, can be equivalent to the presented spatial implicit representations. As such some of the claims such as "To our best knowledge, computing neural spatial representations on time-dependent PDEs for long horizon tasks with multiple time steps has not been explored, and our work aims to fill this gap." are not supported in literature.

There do in addition exist a multitude of sentence structure errors, and typos which I would dearly recommend to address.
summary_of_the_review: This work present a new approach for implicit neural spatial representations for time-dependent partial differential equations. With the temporal representation modeled classically, there exist a number of similar approaches in existing literature, hence rendering the use of implicit representations the main novelty of the paper. In addition the paper severely lacks adequately chosen comparisons, only comparing to the original physics-informed neural network (PINN) approach from 2019.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes using implicit neural representations as a domain discretization, on which a PDE acts. For temporal evolution, classical PDE integrators are used. Next to requiring less memory for the representations, the model has the capacity to represent small but relevant features accurately. However, in the current implementation, the inference is more than one order of magnitude slower compared to using a classical PDE solver.
strength_and_weaknesses: The idea of using low-dimensional implicit spatial representations as a PDE domain discretization is really interesting because it starts with the assumption that there exists a low-dimensional manifold, on which the physical system lives. I could imagine using such representations to study the structure of chaotic systems such as turbulent flows, see Dynamic Mode Decomposition.

Looking at this paper as a first step in the right direction (I have many ideas about how one could improve this method, but this paper is already a legitimate first step), there are some things that can be improved:
1. Reading the paper for the first time, I didn't get how the inference is carried out up until Section 5, which clarifies that the implicit model is retrained at every time step, which of course is a time intense operation. Could you add an algorithm or just a step-by-step description of the inference step in Section 3, or at least in the appendix?
2. In a pure computer vision paper, I could accept demonstrating performance only through visuals or graphs, but in a PDE paper, I would expect a table with results that other papers can compare against. Could you please add such data (even if it is in the appendix)?
3.  Section 4.2 shows the performance of the method on the "Navier-Stokes Equations" (NSE). Having a solid background in fluid mechanics, I have to say that the *inviscid* NSE are called "Euler Equations" and have quite different behavior than the general NSE. Thus, it is for the least misleading to call Section 4.2 "NSE".
4. In the same line of thought as 3., because the Euler Equations have zero viscosity (=> there is no dissipation), the 2D Taylor-Green system has a stationary solution, i.e. nothing will change at all. This is a nice example as a demonstration that the proposed method doesn't add too many errors as would happen if we use any finite spatial discretization, but this should be stated somewhere. I mean something like "This example demonstrates that our method quite well preserves a stationary solution, but it does not yet show any dynamic behavior. For a dynamical 2D simulation we have the example with the two vortices.". Pointing out the preservation of the stationary solution is indeed a strong argument compared to classical methods, which very often entail the so-called numerical diffusion term.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The only issue I mentioned above concerns the steps of the inference procedure.

The paper is also of decent quality when we exclude the Navier-Stokes vs Euler Equations.

In terms of novelty, I find the idea very promising as I explained in the first paragraph above. And it seems to be the first time to do dynamical updates over implicit representations.

In my current understanding, reproducibility means that the code is provided in the supplementary material (SM), which is not the case here. In the SM I only found visualizations of the benchmarks. Thus, I would say that the results are not reproducible because there is always some hyperparameter that is not in the paper, even if it is just a random seed.
summary_of_the_review: The paper is very insightful, but with some minor details to mend before publication. I would already give the paper a 6, but if my comments are addressed appropriately, it would become an 8.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper proposes an NN solver for the time-dependent PDE. The proposed method uses neural networks to parametrize the time-discretized spatial vector field. The loss function for training combined problem-dependent time integrator, penalty terms by boundary condition, and initial condition, which is the same with the classical solvers, yet one is allowed to choose different spatial samplings per time step.  The method is shown to work on several test problems, ranging from 1D advection function to 3D  Elastodynamics equation.   Overall, the authors show that the proposed method is promising for improving the flexibility and numerical accuracy of the solutions. 

strength_and_weaknesses: 
Strengths:

The studied problem is interesting and important. The paper is well organized and easy to follow.  The numerical experiments have enough details for reproducibility. The visualizations  of the solution and mesh are nicely presented. 


Weakness:

1. Limited technical novelty.  As far as I understand, the contribution of this work is to use neural network to represent the solution, and allows the flexibility of spatial samplings per time step. I would say both ideas are not new and has appeared in many works, and  this paper did not provide further insights on  how to select the sampling set and the design of neural network that yield better results. 

 As authors mentioned: one could representing both the spatial and temporal dimensions via neural networks;  or just the spatial dimension or just the temporal dimension. It seems to me that consider spatial dimension may ignore the temporal correlation among different snapshots of the solution, and need more weights that what is actually need to represent a smooth spatiotemproal field.  


2. The method is not  thoroughly evaluated. The evaluation metric is vague: there is no clear definition on  memory usage, numerical evaluation error metric for solutions and time complexity for each method.  Did the author use the state of the art solver for each example? How about the same type of PDEs but with different parameters and initial conditions. These PDEs may not allow explicit solutions but the classical solvers have theoretical guarantees to obtain the solution up to certain accuracy, so I think it is fine to compare with them to see if the proposed method have advantages in other aspects. 

3. The paper should discuss more on the limitations. For example,  this solver is only for smooth solutions, how to deal with non-smooth case? Also, compared to classical solvers, it  may be more difficult to incorporate certain physical constraints and boundary constraints.

4. The PDE examples are up to 3D where the classical solvers also generally worked well.  It is believed that NN solver has advantages on deal with higher dimension problem (see ref 1) , can the authors comment on the potential of the proposed method on higher dimension problem? 

References:
[1] Solving high-dimensional partial differential equations using deep learning


clarity,_quality,_novelty_and_reproducibility: 
Clarity:

1.  What are the numerical evaluation metric for solution. Do you compare the generalization error?

2.  I was confused about the comparison of the methods. For example,  In 1D example, the hyperparameters for each method are chosen based on equal memory usage, I don't know the meaning of the 'equal memory usage'. It seems to me that in NN solver, one can choose 5000 points per time step while the finite difference method only has 901 points.  It would be great if the author can make tables displaying time complexity, sample complexity for each example and accuracy for each example so it has clear indication of accuracy and efficiency. 


summary_of_the_review: To summarize, the paper proposed a NN solver that utilize neural network to parametrize the time-discretized spatial vector field and the adaptive meshes per time step.  I would recommend authors to present a more comprehensive and informative comparison with the existing methods, and more insights on why the proposed method has advantages and its possible limitations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work,  a novel design of a time-dependent PDEs solver is proposed. It is achieved via a neural network and based on recent works on implicit neural representations (MLP + SIREN). This method is an alternative to a a well-known family of spatial discretization methods.
strength_and_weaknesses: The proposed method is straightforward either from modeling or implementation standpoints. It allows to encode the spatial information through a neural network where its weights store  informations of downstream tasks  in an implicit manner.  In contrary to classical  numerical discretization scheme where their success is highly related to the quality of mesh including the spatial sampling and the complexity of some local regions especially near the boundary layer,  implicit neural representation is independent of the number of points and spatial location since the learned weights are global and as a result affect the vector field globally. Moreover, implicit neural representations are powerful to mitigates the spectral biais of neural networks by leveraging high frequencies information and capturing well fine-grained information without scarifying the generalization capabilities of neural networks. The proposed work is also interesting if we look at the tedious task of numerical solver to build adaptive representation by considering different sampling (remeshing) strategies where their choice highly depends on the task, implicit neural representation are adaptive by construction w.r.t weights of the neural network, hence it is agnostic to the neural network architecture.
Another interesting aspect of this work is spatial gradient computation achieved through auto-differentiation. In traditional methods such VFM, FEM, higher order gradients requires higher order basis function such as Chebyshev polynomial which is very expensive in practice (complexity grows with the targeted order to achieve the desired accuracy). In the proposed work higher order gradients are cheap to get as neural implicit representations can achieve sufficiently high order easily by construction thanks to periodic function.

One of the drawbacks of the proposed work is in the computational time. The authors are aware of it and give some directions to tackle that in future works.

The claims are corroborated with experimental study including different physical systems such as Advection equation, N-S equations, and Elastodynamics equations. The results are in general satisfactory.

I have some considerations and questions:

1/ In N-S experiments, what is the Mach and Reynolds number ?

2/ Have you tested or thought about SIREN variants including Multi-Scale implicit neural representations ? The latter could be more representative since Physical systems are multi-scale by construction.

3/ Implicit Neural Representation are based on periodic functions. I am wondering if the good results you obtain on physical systems described with periodic boundary condition is intrinsically related to this property of periodicity of the neural network and boundary conditions. Have you any results on the same physical systems but described with non-periodic boundary conditions ?

4/  It would be interesting to have more quantitative results and comparisons with recent PINN variants.

5/ One other experiment that would be important to conduct is to assess the generalization capabilities of the proposed model on complex geometries (especially near the boundary layer).  For instance, considering the datasets and tasks described in "Learning mesh-based simulation with graph networks".
  

clarity,_quality,_novelty_and_reproducibility: The paper is very clear, well structured and easy to follow. The work is relatively original as it is a combination of different existing works but very important for the ML&Physics community.
summary_of_the_review: The overall work is satisfactory. However,  the experimental part could be drastically consolidated with more challenging tasks and comparison with existing works.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use spatial-coordinate neural networks as an alternative to the spatial representations used in traditional solvers for time-dependent partial differential equations. This work reformulates the PINN-like PDE solvers in a time-dependent way. It calculates the spatial derivatives as in PINNs at each time step and optimizes an INR w.r.t a physical loss involving a discretized time derivative at each time step. The proposed approach extends effectively PINNs beyond the limited temporal domain.
strength_and_weaknesses: Strength:
- The approach uses coordinate neural networks that can be evaluated at any location in space, which is more flexible than commonly used solvers defined on grid or meshes. 
- The optimization is more reasonable than the original PINNs for time-dependent PDEs: the residual loss based on time-stepping introduces the causal temporal relation into the optimization.

Weaknesses:
- As the paper proposes a new meshless PDE solver, it is then necessary to mention the background of the solvers of the same type. Also, the comparison to the existing meshless PDE solvers other than PINNs is somehow limited. If other meshless PDE solvers turn out to be as effective as the proposed method, can the claim about the trade-off between wall-clock runtime and other advantages still holds?
- Comparison w.r.t approches proposed for the same objective of going beyond the training temporal horizon of PINNs is needed, e.g. (Wang & Perdikaris, 2021). 
- Question about the comparison with the PINNs in Section 4.2 inside the same training horizon: are PINNs trained as in the original paper (Raissi et al., 2019)? How the PINNs perform with sequence-like training compared to the proposed method (e.g. Krishnapriyan et al., 2021), which is closer to the setting of the paper?


References:
- (Raissi et al., 2019) Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
- (Krishnapriyan et al., 2021) Characterizing possible failure modes in physics-informed neural networks
- (Wang & Perdikaris, 2021) Long-time integration of parametric evolution equations with physics-informed DeepONets
clarity,_quality,_novelty_and_reproducibility: Clarity: The idea is clearly exposed in the paper. 

Quality: The proposed model is tested on different equations, but with limited comparison w.r.t existing neural approaches and numerical solvers.

Novelty: Somehow novel with respect to PINNs.

Reproducibility: Hyperparameters and architecture details are given in Appendices. However, no source code is given.
summary_of_the_review: This paper introduces the time-stepping into the original PINNs framework, which appears to outperform PINNs and some traditional solvers in many use cases. However, due to the limited novelty and comparison with the baseline methods, I think the paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for partial differential equations by using a coordinate-based network instead of a standard grid as the representation for the solution to the equation. This is done by modeling the spatial dimensions as a SIREN MLP, and then evolving the parameters of the network over the time dimension by solving an optimization problem at each time step. This explicit separation allows the solution to better generalize to new times, as the coordinate-based network tends to overfit on bounded signals. The paper demonstrates that this works for a variety of partial differential equations, and that despite the slower speed, the approach outperforms grid-based solvers in terms of accuracy and memory consumption.
strength_and_weaknesses: In my opinion, the main strengths of the paper are:
- The paper is written clearly, and communicates the proposed method well. The claims made in the paper are well supported by the experiments, and are well explained. The relationship of the proposed method to the related work, grid based methods and PINNs, is clearly outlined so the contribution and differences proposed are extremely clear.
- The method seems to result in noticeable qualitative improvements in the solutions to various PDEs. The evaluation is thorough, and is done across three different PDEs demonstrating that the result is not just overfit for a particular PDE.

In my opinion, the main weaknesses of the paper are:
- The amount of novelty is slightly limited. It seems like the method builds on the SIREN architecture heavily, and the only main contribution is the separation of the temporal dimension and treating it differently than the spatial dimensions. Do the PINN network baselines also use the SIREN architecture, and then just treat the time dimension as an input to the network? If not, this baseline absolutely needs to be compared to in detail in order to prove that this contribution of separating time and spatial dimensions is critical for performance. Overall, I would think the paper should make more clear that this is the main contribution over the prior state-of-the-art in using coordinate-based networks for PDE solutions.
- There are very limited quantitative comparisons. While the qualitative comparisons are very nice and do show that there is a difference in performance, it’s difficult for me to understand the level of the failure without some kind of quantitative metric for each of the experiments. Addition of some quantitative results would definitely increase the strength of the paper by demonstrating that the method proposed really improves on the performance of the prior work.

clarity,_quality,_novelty_and_reproducibility: The paper is written very clearly and is very high quality. While the method is heavily based on prior work using SIRENs, there is some technical novelty and the paper shows that it leads to increased results (some more could be done to further drive this point home). The paper seems reproducible.
summary_of_the_review: Overall, I find the paper to be good: it very clearly proposes and describes a novel tweak to the existing SIREN architecture for PDE solutions (treating the temporal and spatial dimensions differently), evaluates this over a wide variety of different PDEs, and seems to demonstrate that the method performs significantly better. Since the technical contribution is based very heavily on the previous work, I think that the main key for the paper should be to demonstrate that this tweak does significantly improve the results. The paper does do this, but it could definitely be made stronger with some more explanation of what the baselines are (is PINN a SIREN), and some quantitative results to demonstrate more improved performance alongside the qualitative results.

POST REBUTTAL UPDATE:
After reading the author response to my and other reviews, I am inclined to maintain my score. I appreciate the additional comparisons added in the Appendix which highlight the increase in performance (and differences in runtime, which I didn't recognize at first look of the manuscript without these tables). This improves my opinion of the work. However, I still believe that the amount of technical contribution is limited since the method is very similar to PINN with SIREN, and only offers increased performance at the expense of additional compute time. Overall, I still think it is a good paper that could be accepted and is above the threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a parametric solver for PDEs that makes use of implicit neural representation (INR)-based models for replacing the usual discrete mesh representations used by classical solvers. The INR parameters are made time dependent.  An initial network is trained using initial and boundary conditions, the parametric approximation of the solution at time t is then input to a solver - as for classical solvers, the differential equation is known - and INR parameters at next time step t+1 are inferred by solving an inverse problem, the procedure is iterated and the INR parameters are thus updated for further solves. The evaluation is performed on a variety of problems.

All the reviewers appreciated the paper but many of them highlighted a limited novelty and asked for more quantitative experiments. The authors did a good job at clarifying some points of the method, they added quantitative evaluations for a better performance comparison with classical solvers and an additional comparison with a related baseline. This was acknowledged by the reviewers but considered as insufficient by several of them for changing their opinion. The paper would benefit from more extensive comparisons with other ML based baselines that also exploit the knowledge of the PDE equation.

justification_for_why_not_higher_score: Limited novelty and insufficient evaluation comparison with alternative methods.
Decision: Reject