Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 1. Summary: 

The authors propose Neural PDE as an enhanced alternative of Neural ODE, which learns both the governing equations and the target labels by alternating between solving the forward problem and the backward problem.

2. Clearly state your recommendation (accept or reject):

I am leaning towards recommending an accept to this paper, as it proposes a coherent way of solving supervised-learning tasks with Neural PDE.

3. Strong points:

The model is reasonable. Unlike Neural ODE, it does not require numerical integration. It outperforms models including ResNet and ODE-Net in image classfication tasks, and can also generalize to out-of-distribution samples.

4. Ask questions you would like answered by the authors:

a) The main question I have is the relationship to prior work, including Ruthotto and Haber (2019), Long et al. (2018) and Raissi et al. (2019). The authors did mention, for example, that the main difference with Long et al. (2018) is that the latter focuses on scientific problems and only uses a set of discretized points of t. What else are the novelties compared to the previous approaches, besides these as well as alternating between the training of the forward problem and that of the backward problem? It may also be reasonable to compare against those models in the experiments.

b) Any reasons why the boundary conditions removed, and how is the set H of (d, t) pairs selected?


5 Additional comments:

On the topic of physics-informed differential-equations-based models, there are some other works that may be worth referencing:
[1] Greydanus, Samuel, Misko Dzamba, and Jason Yosinski. "Hamiltonian neural networks."
[2] Chen, Zhengdao, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. "Symplectic recurrent neural networks."
[3] Cranmer, Miles, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. "Lagrangian neural networks."
[4] Zhong, Yaofeng Desmond, Biswadip Dey, and Amit Chakraborty. "Symplectic ode-net: Learning hamiltonian dynamics with control."
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper proposed the method of neural PDE as an improvement of neural ODE. In specific, neural PDE considers both the layer and the hidden dimension as continuous variables of the PDE. The new part of neural PDE compared to neural ODE is essentially solving PDE inverse problems (learning PDE from data) in the computational mathematics and engineering community, and the way of learning PDE (by embedding the PDE and initial condition into the loss function via automatic differentiation) is the physics-informed neural network (PINN) proposed in [Raissi et al., JCP, 2019]. The experiments show that compared to neural ODE, neural PDE achieves comparable accuracy but with less forward-pass inference time; but these experiments are not convincing enough.

pros:
- Because the proposed method uses automatic differentiation to handle the derivative as PINN and thus can avoid the numerical integration, and the inference time of neural PDE is less than that of neural ODE.

Major comments:

Results:
- The paper only shows the results on small problems in two tables, which is convincing. To show the performance of this method, the authors should add more experiments and also show more details of the behavior of the method, e.g., training.
- The training procedure is complicated. It includes 4 steps: first train L_T, then L_T + L_C + L_G, etc. It is not clear why the authors train in this way. Are the hyperparameters difficult to tune? How stable is the training?
- The loss includes high-order derivatives of network f, which makes the training much more expensive. What is the computational cost of training? Also, the training trajectories should be added to show the convergence behavior of the loss.

Methods:
- The novelty of neural PDE is replacing ODE in neural ODE with PDE, and adds the PDE loss. But the way of handling PDE loss is exactly the PINN [Raissi et al., JCP, 2019], which is one of the main references in the paper. The authors should state this clearly.
- The authors use the example of Allen-Cahn equation to deliver the message that training with PDE would have better accuracy for extrapolation. However, this example of Allen-Cahn equation is exactly the same example used in [Raissi et al., JCP, 2019]. It might be OK to repeat the example and result from another paper to deliver their message, but the authors should state it clearly. Also, this phenomenon has already been observed in the computational engineering community, e.g., https://www.biorxiv.org/content/10.1101/865063v2 . In fact, I think this part is not the main part of the paper, and can be moved into appendix.
- Section 3 is not well written and the description is not clear.
    - (d,t) are the “symbolic” variables of PDE, and the user can choose their values arbitrary to compute the loss L_G. But they are also the parameters to be trained. Are the (d,t) in L_G the same as the (d,t) used for construct h_last?
    - Is (d, t) the same for all the inputs?
    - Is d in one dimension?
    - It is not clear how the authors select H? How many (d,t) pairs are in H?
    - Are d and t unbounded?
    - It seems that the authors use some points (d,t) in the whole domain to construct h_last; if this is the case, then the “last” is not correct, which usually means the PDE solution at the last time.
- In appendix, why the input dim of f is 6^2x67 not 6^2x67+2? because there are extra inputs of d and t. Why does the network f use ReLU? The derivative f_{dd} would be zero everywhere. The authors should show the details of the network including the (d,t) part.

Minor comments:
- In the introduction, it is not correct that “they studied PDEs in scientific problem domains and do not consider t as a continuous variable but use a set of discretized points of t.” In PINN, t is treated as the continuous variable. The authors should be aware of this.
- Some notations are confusing. For example, d is usually the dimension, but here d is the space variable.
- Why the authors use L_C to denote that initial condition? Why not L_I? since L_B is for “boundary” and L_G is for “governing”.
- The appendix does not have all the information of hyperparameters, e.g., what is the size of H?
review_title: Some novelty in the method; more experiments are required

final_score: rating: 7: Good paper, accept
review_content: The main contribution of this paper is to address the numerical instability issue in neural ODE method when solving the integral problems. To avoid the integral problems, the new methods treat both the layer and hidden dimensions as continuous variable, and solve them at the same time by learning a regression model. 

I also like the idea of learning governing equation and regression model together by an alternating algorithm, which in theory should be better than training a differential equation based neural network with priori knowledge of governing equations. 

My main concern is that to avoid the integral problem in neural ODE, the authors paid the price to treat the whole neural network as a fully coupled system and had to solve for all the layers simultaneously, which needs more variables/parameters to tune. In addition, there is also a number of parameters to add in order to solve for governing equations. All this changes will make the hyper-parameter tuning much more difficult than neural ODEs. The authors should comment on this point during the rebuttal period. 

The new proposed method did give a new insight to handle the numerical instability in neural ODEs, with a bit cost of making model more complicated to tune though. Overall, I would recommend a weakly accept.       
review_title: The paper is an reasonable extension to the idea of neural ODEs. By treating both layer and the hidden dimensions as continuous variables, the proposed method alternately solve the regression model and governing equations that conform to each other, without solving integral problems in neural ODEs. The experiment results showed a good accuracy comparing with neural ODEs. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Post-discussion update: The authors gave a fantastic, thoughtful and exhaustive response that did clarify all of my concerns about the paper. They also updated the paper considerably (making much better), and crucially changed the title to be very accurate the contents.

I like the paper now a lot, but the unimpressive results still stand. The PDE-based image classification performs ok, but also sometimes does not work very well. This would still be ok if insightful analysis of why the model improves would be provided. Unfortunately there is almost none of this, and then the contribution is more in the engineering side than science.

I would not object acceptance, but I would prefer the work to be more complete in this regard first. I raise my score to 5.

----

The paper proposes to solve the forward and inverse problems of PDEs simultaneously (that is, learn both the governing differential, and the forward solution surrogate). This is a dramatic and bold idea, but the paper does not explain why this combination would be a good idea, or what’s the benefit. It’s unclear why is the solution surrogate useful. The experiments show that this provides good results on image classification, but the PDE motivation is lacking. The resulting model does not require any integration, which is a major advantage. However, the paper should be more transparent on discussing the disadvantages of lack of integrals. Without forward solutions, the model is at risk of cumulating errors over time.

The paper seems to borrow its ideas almost completely from Raissi2019, and differences to it needs to be explicated. It seems that this paper takes Raissi2019 method and adds a loss function suitable for image classification. Given that none of the experiments are actually about learning PDEs (they are all image classification), this paper is very misleadingly titled. The method is also very incremental, and seems more like an application of Raissi2019 than an independent research work. 

It’s also difficult to see why one would use a PDE for image classification at all. Labelling small images is already effectively a solved problem. I fail to see the PDE'ness of images.

This work also does not actually learn a “neural PDE”, since the governing equations are assumed to be 16-parameter predefined function, and not a neural function. Only the solution surrogate seems to be a neural network. The title is then misleading also in this regard. 

I also have hard time seeing why not develop this bidirectional method for ODEs first? The paper should do this as an ablation study to first show that it grants some benefits in the simpler ODE case. 

The paper is written in a confusing manner, and lacks presentation polish (typos, language mistakes, strange figure order, lots of dubious statements, etc). The presented methods are also not introduced properly, and it seems that the reader needs intricate understanding of Raissi2019 first.

The experiments show that the PDE-net, ODE-net and ResNet are all equally good at classifying MNIST and SVHN (with no significant differences). The results are missing log-likelihoods, standard deviations and training time analytics. It’s difficult to see what’s the benefit of the method here. In Tiny-experiments the comparison target of MobileNet seems arbitrary. Why compare to a mobile phone -optimized classifier, given that PDE’s surely are far from ideal on such settings. There are no large-scale image classification tasks, nor standard image baseline methods (alexnet, vgg, wresnet). The ResNet comparison is also missing from Tiny. 

The out-of-distribution experiments are excellent, and show the PDE’s improve clearly from ODEs and beat MobileNet. These results are very interesting, and potentially significant. Here more exhaustive experiments should be done, and comparisons to other augmentation methods performed. The authors should also try to give insight why the PDE is more robust to perturbations.

Overall the paper presents an incremental improvement to PDE learning with limited novelty, with unclear presentation, unclear motivation and mixed but partially promising results. The paper needs more work, and should be reworked to be more independent of Raissi2019, and refocused (incl. title) towards the promising application of image robustness, and *why* the PDE are more robust than ODEs in this setting.


Technical comments:
o I do not understand what the dimension “d” means. It does not seem to a dimension at all, but instead a state vector of the state space? The notation is very misleading
o Neural ODE’s do not have particularly small number of parameters (they are often applied in very simple cases or in small latent spaces, but here other NN’s would be simple as well)
o what is “procrastinate”?
o the paper confuses layers and time to be equivalent, this is not true in neural ODEs
o where is eq 1 coming from, and why is the model only restricted to 3rd order (monomial) differentials? Surely a more general PDE definition could have been used
o “general purpose PDE solvers do not exist”: surely they exist, but are too slow to be practical
o eq 6: what is “h”?
o eq 6: why is Raissi2019 performance studied here? I fail to see what’s the relevance of repeating someone else’s work. Is there some novelty here?
o fig4: all methods seem to have very poor fits, given that this is a simple 1D problem with massive amount of data. 
o sec3: h(d,t) should be a function of “h0” as well, and its unclear if this is a true or proxy solution.

review_title: review

Meta-review: This paper proposes a method for regularizing image classifiers by encouraging their hidden activations to conform to a PDE.  This is a reasonable idea, and the authors clearly improved the paper a lot in response to the reviews.  However, the main tasks of MNIST and SVHN classification seem way too easy, and the baselines all need to be tuned to be as fast as possible for a given accuracy, if that's the relevant metric.  I agree with the reviewers that this line of work is promising but that the current paper is not sufficiently illuminating or well-executed to meet ICLR standards.
Decision: Reject