Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper explores the use of deep learning machinery for the purpose of identifying dynamical systems specified by PDEs.

The paper advocates the following approach:
One assumes a dynamic PDE system involving differential operators up to a given order. Each differential operator term is approximated by a filter whose values are properly constrained so as to correspond to finite difference approximations of the corresponding term. The paper discusses the underlying wavelet-related theory in detail. A certain form of the dynamic function and/or source terms is also assumed. An explicit Euler scheme is adopted for time discretization. The parameters of the system are learned by minimizing the approximation error at each timestep. In the experiments reported in the paper the reference signal is provided by numerical simulation of a ground truth system and the authors compare the prediction quality of different versions of their system (eg, for different kernel size).

Overall I find the paper good, well written and motivated. The advocated approach should be appealing for scientific applications of deep learning where not only the quality of approximation but also the interpretability of the identified model is important.

Some suggestions for improvement:
* The paper doesn't discuss the spatial boundary conditions. Please clarify this.
* The paper adopts a hybrid approach that lies in between the classic fully analytical PDE approach and the data driven machine learning. I would like to see a couple more experiments comparing the proposed approach with those extremes. (1) in the first experiment, the underlying model order is 2 but the experiment allows filters up to order 4. Can you please report if generalization quality improves if the correct order 2 is specified? (2) On the other side, what happens if no sum (vanishing order) constraints are enforced during model training? This abandons the interpretability of the model as approximating a PDE of given order but I am curious to see what is the generalization error of this less constrained system.

Nit: bibtex error in Weinan (2017) makes the paper appear as E (2017).
review_title: Good paper on the use of deep learning for modeling and identifying dynamical PDE based systems

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses complex dynamical systems modelling through nonparametric Partial Differential Equations using neural architectures. It falls down within the context of a recent and growing literature on the subject.

The most important idea of the papier (PDE-net) is to learn both differential operators and the function that governs the PDE. To achieve this goal, the approach relies on the approximation of differential operators by convolution of filters of appropriate order.  This is really the strongest point of the paper.

Moreover, a basic system called delta t block implements one level of full approximation and is stoked several times.
A short section relates this work to recent existing work and numerical results are deployed on simulated data.
In particular, the interest of learning the filters involved in the approximation of the differential operators is tested against a frozen variant of the PDE-net.

Comments:
The paper is badly structured and is sometimes hard to read because it does not present in a linear way the classic ingredients of Machine Learning, expression of the full function to be estimated, equations of each layer, description of the set of parameters to be learned and the loss function. Parts of the puzzle have to be found in the core of the paper as well as in simulations.

About the loss function, I was surprised not to see a sparsity constraint on the different filters in order to select the order of the differential operators themselves. If one want to achieve interpretability of the resulting PDE, this is very important.

I also found difficult to measure the degree of novelty of the approach considering the recent works and  the related work section should have been much more precise in terms of comparison. 

For  the simulations, it is perfectly fine to rely on simulated datasets. However the approach is not compared to the closest works (Sonoda et al., for instance).

Finally, I’ve found the paper very interesting and promising but regarding the standard of scientific publication, it requires additional attention to provide a better description the model and discuss the learning scheme to get a strongest and reproducible approach.
review_title: A promising approach on nonparametric modelling of partial differential equations with deep architectures that requires more details.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Authors propose a neural network based algorithm for learning from data that arises from dynamical systems with governing equations that can be written as partial differential equations.  The network architecture is constrained such that regardless of the parameters, it always implements discretization of an arbitrary PDE.  Through learning, the network adapts itself to solve a specific PDE.  Discretization is finite difference in space and forward Euler in time.  

The article is quite novel in my opinion.  To the best of my knowledge, it is the first article that implements a generic method for learning arbitrary PDE models from data.  In using networks, the method differs from previously proposed approaches for learning PDEs.  Experiments are only presented with synthetic data but given the potential for the method and its novelty, I believe this can be accepted.  However, it would have been a stronger article if authors have applied to a real life model with real initial and boundary conditions, and real observations. 

I have three main criticism about the article: 

 1.   Authors do not cite Chen and Pock’s article on learning diffusion filters with networks that first published in CVPR 2015 and then authors published a PAMI article this year.  To the best of my knowledge, they are the first to show the connection between res-net type architecture and numerical solutions of PDEs. I think proper credit should be given. [I need to emphasize that I am not an author in that article.] 
 2.   Authors emphasize the importance of interpretability, however, the constraint on the moment matrices might cripple this aspect.  The frozen filters have clear interpretations. They are in the end finite difference approximations with some level of accuracy depending on the size and the number of zeros.  When M(q) matrix is free to change, it is unclear what the effect will be on the filters.  Are the numbers that replace stars in Equation 6 for instance, will be absorbed in the O(\epsilon) term?  Can one really interpret the final c_{ij} for filters whose M(q) have many non-zeros? 
 3.   The introduction and results sections are well written.  The method section on the other hand, needs improvement.  The notation is not easy to follow due to missing definitions.  I believe with proper definitions — amounting to small modifications — readability of the article can substantially improve. 


In addition to the main criticisms, I have some other questions and concerns: 

  1.  How sensitive is the model?  In real life, one cannot expect to get observations every delta t.  Data is most often very sparse.  Can the model learn in that regime?  Can it differentiate between different PDEs and find the correct one with sparse data? 
  2.  The average operations decreases the interpretability of the proposed model as a PDE.  Depending on the filter size, D_{0}u can deviate from u, which should be the term that should be used in the residual block.  Why do authors need this?  How does the model behave without it? 
  3.  The statement “Thus, the PDE-Net with bigger n owns a longer time stability.” is a very vague statement.  I understand with larger n, training would be easier since more data would be used to estimate parameters.  However, it is not clear how this relates to “time stability”, which is also not defined in the article. 
 4.  How is the relative error computed?  Values in relative error plots goes as high as 10^2.  That would be a huge error if it is relative. 
review_title: Novel method, interesting approach, interpretability might be over-emphasized. 

Meta-review: This paper studies the approximation and integration of partial differential equations using convolutional neural networks. By constraining CNN filters to have prescribed vanishing moments, the authors interpret CNN-based temporal prediction in terms of 'pde discovery'. The method is demonstrated on simple convection-diffusion simulations.

Reviewers were mixed in assessing the quality, novelty and significance of this work. While they all acknowledged the importance of future research in this area, they raised concerns about clarity of exposition (which has been improved during the rebuttal period), as well as the novelty / motivation. The AC shares these concerns; in particular, he misses a more thorough analysis of stability (under what conditions would one use this method to estimate an actual PDE and obtain some certificate of approximation?) and discussions about pitfalls (in real situations one may not know in advance the family of differential operators involved in the physical process nor the nature of the non-linearity; does the method produce a faithful approximation? why?).

Overall, the AC thinks this is an interesting submission that is still in its preliminary stage, and therefore recommends resubmitting to the worshop track at this time.
Decision: Invite to Workshop Track