Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Review: 

This papers proposes an algorithm to learn a model for spatio temporal data assumed to be described by a stationary spatio temporal PDE.

Data considered in the paper consists of vectors y(t) of observations at time t and a fixed set of spatial indices x.
A model for discrete vector y(t) is proposed in the form of coupled ODEs (one for each x_i) with a sparse coupling arising from a neighbouring graph on spatial inputs x, and sharing the same transition function. 
This transition acts on relative local spatial information and on the absolute function values in a neighbourhood.

The continuous time ODE can be solved using classing ODE solver. The model is fit to data using the adjoint method to backprop through the solution given a squared loss.
The ability of the model to learn is evaluated on data generated from PDEs.

+ves: 

+ The proposed model is sensible and the paper motivates and describes the method well.
The obtained sparsity of the ODE coupling makes the method scalable. Preserving a continuous time dimension is useful 
if data is irregularly sampled

+ Once learned, the model is quite flexible -> adapts to new grids, adapts to various time intervals
 
Concerns: 

- The paper does not describe well the setting in which it is applicable. Spatio temporal data could be observations at random space and time locations. One has to read all the details to understand this while it should be highlighed.
For example, the data arrives in vectors of observations at input (x, t) for different t.
The method would not be able to deal with missing data (a partial vector y(t)) etc.. , the method assumes stationarity.

- The methods cite other approaches that take a different route to the problem (e.g. for example learning the parameters of PDE directly) such approaches have their scaling issues but inherit from half a century of research on approximate solutions to pdes, and come with guarantees. Does this method come with any guarantees? A discussion on this aspect would be useful.


- For infinitesimal time steps, it makes sense to use only local information to build the differential F.
But for longer time steps (think diffusion) you would need information that span further away to get accurate results.
Along these lines, how would a model trained on fine time grid deteriorate as you test it on data with bigger time steps

- I m surprised of the experiment about removing the positional information. Removing it makes no sense when one has stationary PDEs in mind. Same with the noise: if you add noise, performance decreases.



Overall, I like the method and think it can be very useful to members of the community,  but I find the paper lack a broader perspective when describing it.
The PDE connection could be used to discuss intuition on where it works or fails much more and to guide
the model experiments and validation.
Instead, the current writing makes it sound a black box engineering solution has been proposed and is
 tested with no real guiding principle.

For this reason, I am not a big proponent of the paper but do not oppose acceptance.




review_title: Interesting method, disappointing description and discussion

final_score: rating: 6: Marginally above acceptance threshold
review_content: This submission proposes extensions of PDE-net that relax some constraints that could help extend the range of applications of this approach. First, rather than fixing a spatial discretization in the form of a grid, the authors use a Delaunay triangulation to represent the domain. The updates to the nodes of this triangulation are performed using a message-passing GNN framework which couples neighboring nodes. Secondly, the authors use a classical adjoint method to allow for arbitrary time-discretizations (though this may be much more expensive in practice).

I am not an expert on the experimental side of this area, but my sense was that the performance of the approach is relatively good, especially compared to other methods when the time step is large. For small time steps, however, PDE-Net is multiple orders of magnitude better than the present approach, a fact that should have been thoroughly discussed in the text rather than just mentioned, because the discrepancy is so large. 

All the experimental results are presented as, essentially, measures of the training error. My sense is that the point of this methodology would be to meaningfully extend the time-horizon over which a given PDE could be intergrated, so I would have liked to see how the integrator performed outside the test set. 

Sparsity is mentioned in the title and the "contributions" but appears basically nowhere in the main text. I did not actually see in what regard the data was sparse or see a clear theoretical justification as to why the MPNN approach would be superior on sparse data. I suppose the argument is that with arbitrary spatial and time discretization, the method still able to be formulated whereas PDE-Net requires dense spatial discretizations and time discretizations to train. However, the fact that the method could in principle be used on sparse data is not a demonstration that it works on sparse data. How does PDE-Net perform in the 2 and 4 time point cases?

review_title: PDE learning with few constraints

final_score: rating: 6: Marginally above acceptance threshold
review_content: - Summary

This paper presents a graph neural network model for learning to model PDE dynamical systems. Given its graph structure and its continuous time formulation (employing the adjoint method for differentiation/backpropagation), this method allows the usage of samples placed arbitrarily in space and time.
 

- Pros

The continuous-time nature of the model allows for the usage of irregular time sample points.

Previously proposed methods either would not work on continuous time, or unstructured grids, or would not be applicable to settings with unknown governing PDEs. This work combines all these features. 

The graph-based representation used makes the proposed method invariant to translations and rotations in the spatial domain.



- Cons

Similar graph-based methods that used continuous-time to model differential equation dynamics had been previously presented, e.g. GNODE. The novelty of the proposed method might be limited.

The test cases are simple and the experimental details are somewhat lacking for a full evaluation of the results (more details below in the additional comments). 



- Reasons for score

[Edit: Score updated, see discussion below]

Overall, given the "cons" described above, notably the potential lack of strong novelty in the proposed method, and the lacking experimental description and results, I am for now classifying this paper as marginally below the acceptance threshold. 
On the positive side, the method seems to perform favorably when compared to other baselines, in comparisons that are actually favorable to the other methods (e.g., using regular grids). Moreover, the method performs well on the tasks it is tested on.
However, I'm concerned with some uncertainties I have regarding the experimental section and the presented results. These are discussed below in the comments.
Moreover, the proposed method centers around using message-passing neural networks to model the differential equation dynamisc. As mentioned above, previous methods had already proposed the usage of graph neural networks with continuous time for the learning of differential equations, and I am not sure that the addition of spatial mesh information to such a graph neural network constitutes a significant enough modification at this point. 
Despite the concerns above, I am open to reading the authors' responses and the reviews/comments and changing my opinion depending on how those affect my current uncertainty.

 

- Additional comments

I believe an important element that is missing from the description of the experiments is a clearer of how much do train and test set actually differ? This would be important to understand how hard the tasks being performed are. Clearly, if training and test set are too similar, the results lose a lot of power.
Morever, since we also dont see any traning vs test plots, it is also hard to see how much performance is different between these two are. (I am not claiming such a plot would be necessary, but merely that given the otherwise lack of information in this direction, it would be helpful information.) I am aware that the appendix includes a description of how the initial conditions for the data are generated, but lacking more information these are hard to grasp intuitively to be able to judge the tasks.

Moreover, the error in model rollouts over time seems to spike at the beginning and then quickly flatten out. It seems strange that errors would spike up initially and then not compound significantly over time. Do the authors have any intuition as to why this is the case? Is it maybe a consequence of the data samples reaching a sort of steady state after some time? If so, wouldn't this weaken the case being made for a continuous-time model?


How would the Delauney triangulation being employed deal with possible obstacles present in the spatial domain? For example, an airfoil might have its opposing boundaries connected by edges (since they are close in space), even though that would supposedly be a solid. Would these types of solids have to be manually specified when extending this method to such scenarios? (This is not a "drawback", of course, it would be expected of most methods that such object boundaries would have to be defined.)


What integrator is used for the experiments? 





review_title: Review

final_score: rating: 7: Good paper, accept
review_content: The paper proposes to use graph-based networks for evaluations of PDEs with continuous time formulations. In contrast to existing works on continuous time ODE formulations with graph structures, the proposed networks incorporate relative spatial information in order for the network to evaluate spatial derivatives in addition to the temporal dynamics. A key argument for this setup is the flexibility in spcae (via the graph nets) in addition to a variable time step. 

The proposed setup is evaluated for the relatively simple PDE cases, a convection diffusion case, a pure diffusion case, and another advection diffusion case for transport. Despite the simplicity, the authors make an effort to illustrate the behavior of their method with an ablation study, and to compare to previous work. Here, they compare to PDE-net, which was originally proposed for system identification, and is used for predictions over time instead here. In addition, they compare to the GNODE approach by Poli et al., which omits the spatial information, but already incorporates ODEs into graph nets. For the latter, the authors demonstrate that for simple cases (pure diffusion) the GNODE approach does a good job to identify dynamics purely over time, while including advection terms significantly increases the error without spatial information. This is good to see and makes sense.

For the PDE-net comparison, I was wondering how the 3 time step sizes were incorporated into the PDE-net. Isn't this "by-default" a fixed time step architecture? Was it changed to receive the time step as an input, or does Figure 7 show 3 networks, i.e. one per timestep? The appendix unfortunately does not provide any additional details on how the comparison was executed. 

I also have to admit I couldnt follow the argumentation why graph-nets, e.g., from Sanchez-Gonalez et al. '18 aren't suitable in the context of the paper. Sure, they are evaluated on moving positions, but isn't that even more difficult compared to the static locations used here? So when keeping these positions fixed, wouldn't the networks potentially do an even better job than for the moving locations? As a plus, they wouldn't require a Delaunay triangulation or similar meshing step.

While the ablation study is generally nice, I was missing one central component here: as the paper targets continuous time (it's highlighted in the title), I expected an evaluation regarding how much is gained from the introduction of continuous time. The larger time steps of Fig. 3 seem quite trivial, but what I instead hoped to see here was a comparison to a model that simply receives the chosen tilmestep dt as an additional input, and is trained in a supervised fashion with data from a few different time step sizes (i.e. non-continuous, but varying dt). This is maybe what was done for the PDE-net, but the paper is not clear here. I think it would be important to demonstrate the advantages of a continuous formulation, which introduces a significant amount for complexity, over a much simpler training with discrete but varying time steps. 

I hope the authors can shed light on this aspect during the rebuttal, as apart from this relatively central open question I like the paper. Thus, assuming that the authors can clarify this part and show that the proposed method yields benefits, I think this could be an interesting paper for ICLR. It presents an interesting evaluation of PDE-learning with graph-nets, which I would consider to be interesting for many ICLR attendees.

review_title: Nice evaluation of graph-based networks for PDEs with open questions on the side of continuous time

Meta-review: This paper proposes a new method for learning a model for spatio-temporal data described by an (unknown) spatio-temporal PDE. The model learns a continuous time PDE using the adjunct method and uses graph networks to perform message passing between different discrete time steps on a grid obtained with Delaunay triangulation.

The method initially 3 favorable and 1 unfavorable ratings, but convincing responses to some of the raised issues led to unanimous recommendations for acceptance (not all reviewer feedback after the rebuttal has been made public, but feedback has been made to the privately AC on these issues by different reviewers). 

The reviewers appreciated novelty of the method and numerous ablations.

Initially perceived weaknesses were some key experiments on generalization over different grid discretizations; the simplicity of some experiments, and links to different prior art - many of these points have been dealt with by authors in their response.

The AC concurs and proposes acceptance.
Decision: Accept (Poster)