Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Partial differential equations (PDEs) are ubiquitous in science and engineering, and modern deep learning frameworks for modeling PDEs are of interest for scientific machine learning. This paper leverages the method of characteristics to extend the framework of Neural Ordinary Differential Equations (Neural ODEs) to modeling (hyperbolic) PDEs. The method of characteristics allows to transform hyperbolic PDEs into ODEs. Then, tools for learning Neural ODEs can be used to model the transformed PDE. Several different problems are used to demonstrate the performance of the proposed Characteristic Neural ODE framework.
strength_and_weaknesses: This paper introduces a novel and relevant extension for the existing Neural ODE framework, and the presented experimental results are significant. The proposed framework allows to model a broad class of interesting PDEs, i.e., hyperbolic PDEs. I don’t feel that it is a restrictive limitation of the framework that it can handle only hyperbolic PDEs, since this class is of general interest in many problems.  

While the key innovation of this paper is a new deep learning method for PDE approximation, my concern is that this framework could just be viewed as a version of augmented NODEs. However, the experiments seem to suggest that the proposed method gives good results. To me, the strength here is in the interpretation, but I'm guessing from the PDE experiment that you cannot get the NNs to represent what their analytical solution counterparts should be. Still, I think that this paper is a good step in the right direction.  Here are things that I would like to see addressed, and that I feel will help to improve the paper: 


* The presentation of the mathematical exposition can be streamlined in several places to make the paper better readable. I feel that sometimes some unnecessary terms/notations are introduced without additional explanation. For instance, the middle term in the first equation of the related work is more confusing than helpful. I don’t get why it is important to explicitly state that the right-hand side is a function of (u,t). If this notation is important to understand later concepts, then it should be mentioned / discussed with a sentence within the text. I would try to keep things simple so that the paper is more accessible to a broader audience.  


* The related work is discussing that a residual unit can be modeled as an ODE in the limit. While in theory that is fine, this statement is problematic in practice. Typically, you can’t choose $\Delta t$ arbitrarily small, if you want to train a model in finite time. Hence, while there is some algebraic resemblance, the connection between resnets and the forward Euler discretization of an ODE is problematic. I would suggest citing works that point out some of these issues: [1,2]. 


* Since the presented approach is a method for PDE approximation, I would suggest devoting more room for experimental results in the main body of the manuscript to illustrate the framework, i.e., move some of the experiments from the appendix to the main text and expand the discussion. Additional experiments for PDEs in the appendix would be welcome. I am saying this, because I don’t feel that the proposed method is of interest for image classification. Also, the image classification experiments ignore mention of state-of-the-art Neural ODE frameworks [3,4]. For instance [4] achieves about 90% test accuracy on CIFAR-10 with a similar parameter count, and 99.6% test accuracy on MNIST with only 18k parameters. These works should at least be cited in the related works.  


* The quality of the plots can be improved. Instead of having solid confidence bands, I suggest using transparent confidence bands so that it can be seen how large the overlap is. 


* Lastly, I would add a short discussion about the differences / advantages of the proposed approach compared to physics-informed neural networks. I think that the proposed approach is more general for modeling PDEs. So, it would be nice to point this out.  

 

[1] Ott, Katharina, et al. "Resnet after all: Neural odes and their numerical solution." International Conference on Learning Representations. 2020. 

[2] Krishnapriyan, Aditi S., et al. "Learning continuous models for continuous physics." arXiv preprint arXiv:2202.08494 (2022). 

[3] Zhang, Tianjun, et al. "ANODEV2: A coupled neural ODE framework." Advances in Neural Information Processing Systems 32 (2019). 

[4] Queiruga, Alejandro, et al. "Stateful ODE-Nets using basis function expansions." Advances in Neural Information Processing Systems 34 (2021): 21770-21781. 
clarity,_quality,_novelty_and_reproducibility: This work introduces a novel method for PDE approximation. The presented ideas and theory are sound and interesting. Overall, the paper is of high quality, though clarity can be slightly improved by streamlining the presentation and organization of the paper.
summary_of_the_review: In summary, the presented ideas are solid, and I feel that the innovations are novel and significant. The proposed framework connects Neural ODEs to an interesting class of PDEs that are relevant in many real-world applications. Hence, I have no doubt that this work has the potential to inspire future research in this area. Overall, this paper is of comparable quality compared to several related works that have been recently been accepted at ICLR and other ML conference.  Given that the reviewers address my comments, I recommend to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed Characteristic Neural ODE (C-NODE), which extends the idea of Neural ODE to combine with PDEs, and enables the model to learn a family of models (rather than a single Neural ODE model).

Furthermore, the authors demonstrated that C-NODE has a nice property, that it can model intersection trajectories (similar to augmented Neural ODE), in contrast Neural ODE losing such capability. The authors also derived the formulation for continuous normalizing flow with C-NODE (in the PDE settings).

Finally, the authors validated the performance of C-NODE with experiments.
strength_and_weaknesses: Strength:
1. The idea to extend Neural ODE to PDE via Characteristic equations is novel. It great expands the application of Neural ODEs.
2. The theoretical derivation is solid. (Though I only loosely checked the proofs the theories, they look pretty convincing to me).
3. The paper is well-presented and easy to follow, especially in the method section, the authors started from the general background of characteristic equations for PDE, then extend them to the Neural ODE case.

Weakness:
1. Could the authors expand a bit more on the difference between augmented NODE and C-NODE? The formulation of C-NODE essentially models the original NODE in a higher dimension, which looks close to augmented NODE. So I wonder what is the key difference.
2. Experiments
    Despite the nice theory and presentation, the paper lacks comparison with some stronger baselines. For example, [1,2,3] achieved slightly better results on CNF on Cifar10, though they are focused on numerical solvers rather than PDE, it's worth mentioning in the paper. 

PS: I doubt if the slightly worse result of C-NODE is because the model is smaller than in the literature. I would be happy to increase score if the author show better results with a model of comparable size.

[1] Zhuang, Juntang, et al. "Adaptive checkpoint adjoint method for gradient estimation in neural ode." International Conference on Machine Learning. PMLR, 2020. 
[2] Matsubara, Takashi, Yuto Miyatake, and Takaharu Yaguchi. "Symplectic adjoint method for exact gradient of neural ode with minimal memory." Advances in Neural Information Processing Systems 34 (2021): 20772-20784.
[3] Zhuang, Juntang, et al. "MALI: A memory efficient and reverse accurate integrator for Neural ODEs." arXiv preprint arXiv:2102.04668 (2021).
clarity,_quality,_novelty_and_reproducibility: Quality:
The paper is of high quality, with novel idea and solid theory.

Clarity:
The paper is well presented and clear.

Originality:
The paper is novel and pretty original
summary_of_the_review: The paper is novel, well presented and solid in theory. Though a little bit weak in experiment, but it could come from that the model size is small. Overall is a very good paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript introduces a neural ODE (NODE) type model inspired by the method of characteristics from PDE theory, which is therefore referred to as characteristic NODE or C-NODE. Compared to vanilla NODEs the resulting parametric dynamical system can exhibit intersecting dynamics, which can be desirable for certain applications.
This result is strengthened by showing that the proposed model is more expressive than vanilla ODEs in the sense that it can compute arbitrary homeomorphisms as input-output maps. 
Further, a formula for the evoluation of the log density under C-NODEs is presented, which makes the model applicable to density estimation problems. 
The proposed model is tested on image classification, time series prediction, density estimation as well as approximation of the solution of a quasilinear hyperbolig PDE.

strength_and_weaknesses: **Strengths:**
* The proposed model is based on an important theoretical tool in the theory of PDEs. 
* The expressivity of the proposed model is studied theoretically
* The experiments showing performance gains of the model cover a variety of different problems and include comparisons against a variety of different methods.

**Weaknesses:**

* On the side of the experiments, I believe the computation time should be added for a reference, since the number of parameters does not completely describe the complexity of training. Further, in the limitations it is stated that the proposed model exhibits instabilities during training. A discussion regarding this should already present in the experiments section. 

* The experiment on the representational power when it comes to approximating solutions of PDEs seems not well desgined. As far as I see it, C-NODEs are shown to exhibit lower generalization error compared to NODEs when used to regress the solution of a hyperbolic PDE. This however does not imply that C-NODEs are more effective in approximating the regressed function. 

* One weakness of the paper is the motivation to use a model inspired by hyperbolic PDEs. The main argument given is its ability to „transport data into different regions in state space“, which is also highlighted by the theoretical expressivity results. Where I agree that this is indeed a favorable property, this is already the case for simple augmented NODEs. To me it is not clear whether the improvements in the experiments are actually due to an underlying hyperbolic structure in the data of the experiments. That being said, I think this is fine, since the manuscript only speculates about this and does not offer this as an explanation.

clarity,_quality,_novelty_and_reproducibility: **Clarity:**
The description of the method could be improved overall. To me – being not an expert on the method of characteristics – it is not entirely clear how the proposed method works. In particular, it is not clear how the coupled ODE (4)&(5) is solved in praxis: I assume that in general one will solve iteratively for the solution in both $x$ and $\symbf u$ simultaneously, however, in Algorithm 1, not update of $x$ appears. Further, the domain of $u$ is not entirely clear: it is introduced as a function on $\mathbb R^k$, I assume it should be a function of $\mathbb R^k\times\mathbb R$ and later it (or rather its Jacobian) has arguments $(x, \symbf u)$. Similar inaccuracies are common in the manuscript, which makes the definition of the model hard to follow. Overall, I believe the manuscript requires some polishing in order to improve readability. 

**Quality:**
The paper provides a nice extension of NODEs and I appreciate the inspiration from a method from PDE theory. The experiments -- apart from Subsection 5.4 -- are well designed and compare the method in a variety of settings to various baselines.


**Novelty:**
The idea is intriguing and appears novel to me. That being said, I am not an expert on NODEs.

**Reproducability:**
The reproducability could be improved. For example, for the classification problem, neither the loss nor the architecture nor the hyperparameters are reported. Further, the code is not available.

summary_of_the_review: The manuscript proposes an interesting new NODE type model, which is inspired by the method of characteristics for hyperbolic PDEs. 
Overall, I believe that the idea is intriguing and promising. In its current form, I am mostly concerned about the clarity of the paper (the introduction of the model is sloppy or at least not very clear) as well as the reproducability. I believe that both of these points should be addressed prior to a publication of the manuscript.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to model the evolution of latent variables in continuous time models as the solution of first-order partial differential equations. The authors builds upon the method of characteristics for solving PDEs and show that their approach provides both theoretical and practical advantages. In particular, they show that their method can model interesecting trajectories and can still benefit from the adjoint method for backpropagation. Experimentally, the method outperforms classical NODE and ANODE in a variety of tasks (classificiation, time series classification as well as NF). 
strength_and_weaknesses: ## Strengths

- The method is nicely presented and the overall paper is clear. The method is also well motivated, using the method of characteristics for solving PDEs. 

- The experimental section is solid, with various setttings (spanning from classification, to time series modeling and normalizing flows). The results indicate improved performance of C-NODE.

## Weaknesses


From the paper, I read two motivations : the ability to keep the adjoint method for training and the ability to model more complex dynamics than NODE (in particular intersecting trajectories). Yet, I think these are currently under-developped in the paper.

*About the adjoint*

- For the adjoint method, from experience, the improved memory complexity comes at the cost of significantly higher computational time, making it therefore unpractical. Therefore, this motivation is quite weak in my opinion. What is more, the cases used in the experimients don't seem to be requiring the adjoint. method for training either. I would therefore recommend to make this adjoint argument more compelling by showing a case study where the adjoint is indeed necessary.

*About the ability to model intersecting trajectories*

- The ability to model intersecting trajectories seem to arise from the fact that you include the first time point $g(\mathbf{z})$ in the input of the dynamical system function. In my sense, that alone suffices to allow for intersecting trajectories. If you agree, this comes, in my sense, with multiple implications. First, that statement should be made more clear in the text so the reader can better see where this ability comes from. Second, this would represent the most interesting baseline to compare against. Namely, you take a neural ODE and you feed the first $\mathbf{z}$ as input, similiarly as you do for the dynamics of the characteristics. This would be, in my opinion, a more compelling argument for using the method of characteristics framework. 
- What is more, the non-intersecting property clashes with the invertibility statement of section 4.2. Indeed, I do not see how intersecting can be invertible as they won't be injective. I think this would deserve some clarification. I believe this confusion is due to what you consider is the input (if you condition on $z$, it is invertible$, but not otherwise).


clarity,_quality,_novelty_and_reproducibility: ### Clarity

The paper is very clear. I believe some improvements can be made in the intuition why the space of dynamics is richer in C-NODE (see my comment above about the interesecting trajectories). Clarifications regarding the inconsistency between invertibility and interesecting trajectories is also needed.

### Quality

The work is qualitative, and the experimental section is broad and convicing. 

### Novelty

The novelty comes from the method of characteristics. I believe this would deserve some extra experiments to showcase the value of this theoretical framework (see my comment above).

### Reproducibility

The derivations of the model are clear and should allow for reproducing the experiments.
summary_of_the_review: This work proposes and interesting idea from PDE solving to improve NODE architecture. The experiments are broad in scope. I still have some concern regarding a mismatch between the experiments and the motivations of the paper : regarding the adjoint and the usage of the first input in the dynamics. Some work on the motivation and comparison with more relevant baseline (the one I suggested above) would be appreciated.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper translates results from approximation theory to neural ODEs for the solution of PDEs. The presentation is clear, and the reviewers all broadly agree that this paper should be accepted.
justification_for_why_not_higher_score: There are related results in existing works, this is not a groundbreaking new concept.
justification_for_why_not_lower_score: see above, all reviewers broadly agree that this paper is a valuable contribution to ICLR.
Decision: Accept: poster