Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors investigate the role of the method of characteristics (MoC) on the modeling of the latent variables of NODEs, and propose a new algorithm named C-NODEs. C-NODEs parameterize the latent variables of NODEs as the solution of a family of first-order quasi-linear PDEs along the characteristic curves. The authors prove C-NODEs can learn intersecting trajectories and also are universal approximators of homeomorphisms. Experiments empirically show C-NODEs improve the learning accuracy and efficiency on multiple tasks.
strengths_and_weaknesses: The expressiveness of vanilla NODEs is indeed a problem due to the restriction of ODEs. The paper try to involve the family of PDEs to augment NODEs. The paper is easy to follow. The strength of this paper is the rich experiments and analysis of the properties of C-NODEs. However, the presented work raises several questions. The main weakness is the clarity and insufficient baselines. See questions below.

questions: 
1. This paper uses MoC to parameterize or augment the latent variables of NODEs. However, this parameterization restrict the latent varialbes to the solution of first-order quasi-linear hyperbolic PDEs. Does this hypothesis make sense?

2. Some other papers also propose ways to augment NODEs by parameterize the latent variables, such as ANODE, Second-order NODEs (SONODE). Are C-NODEs better than these methods?

3. In Equation (7) (line 129), the functions $J_x u$ and $dx/ds$ are modeled by neural networks. Using $\text{NN}_1$ and $\text{NN}_2$ to represent neural networks, is it equivalent to $u(x(T)) = u(x(0)) + \int_0^T \text{NN}_1(x,u;\Theta_2) \text{NN}_2(x,u;\Theta_2)ds$? If so, the structure of Equation (4) will not be preserved.

4. Near the line 111, $\frac{d}{ds}u(x(s),t(s)) = \frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} = 0$. If so, by integrating over $s$, we have $u(x(T),t(T);x_0,t_0) \coloneqq \int_0^T \frac{d}{ds} u(x(s),t(s))ds = \int_0^T 0 ds = 0$ ?

typo: line 3 - ``a latent variables"
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: **Edit after Reviewer and AC discussion**

After a thorough discussion with the other reviewers and AC unfortunately I will lower my score to a 6. The main reason is that the most revealing experiment is in Table 1, which is a strong experiment. Revealing how all the baselines change with and without C-NODE. But image classification is not the best experiment for NODEs. Instead this strong comparison should be carried out on all experiments especially Normalizing Flows and Time-Series.


**Original**

This paper adapts the Neural Ordinary Differential Equation (NODE) framework to run along the characteristic curves of partial differential equations (PDES), the new model is called Characteristic Neural ODEs (C-NODEs).

This effectively is a different way to make more expressive NODE models, similar to Augmented Neural ODEs and Neural Controlled ODEs being techniques to improve expressivity and generalization of NODEs.

Proofs are provided to show this improves expressivity. C-NODEs are tested on classification and continuous normalizing flow tasks (density estimation) on MNIST, CIFAR-10 and SVHN. C-NODEs are also tested on time-series and PDE tasks. C-NODEs perform well on these tasks, outperforming NODEs as well as Augmented NODEs, 2nd Order NODEs and IL-NODEs. 
strengths_and_weaknesses: Strengths:

The paper is generally very good, with good contributions and methdology, specifically:
- The theoretical grounding of C-NODEs is good.
- The experiments on the whole are good, experimenting against expected baselines on standard benchmark tests.
- The paper is well written, with helpful examples for difficult/new concepts.

Weaknesses:

The largest weakness is that the paper generally feels a little incomplete, the work is brilliant but there are some experiments that could make it a brilliant paper. Specific examples are:

- There is this term $g(z)$ included in the dynamics, allowing the vector field to be conditioned on the input. It would be quite informative to see how much this term affects the answer. That is, can this term (a learnable function with the same structure) solve the classification problem on its own? This would make a convincing ablation if it can't.
- This term $g(z)$ also seems to give the initial condition for the C-NODE, it might be helpful to visualise the C-NODEs time evolution, if it stays roughly the same $g(z)$ is doing the majority of the work.
- Parameter efficiency is mentioned in the paper, but this doesn't seem to be explicitly tested, again this would make a good appendix study.
- Results are missing for testing Continuous Normalizing Flows (and discrete flows) on SVHN.
- The paper claims the adjoint method is better because it can be used with adaptive solvers due to memory efficiency. The Euler method is used with direct backprop. It would be helpful to show that Dopri5 fails when using direct backprop because memory runs out.
- It is noted that coupling C-NODEs with ANODEs can make training more stable, apart from in the classification experiments it doesn't look like this has been tested.

Aside from this there are some minor typos or tiny mistakes that do not affect the quality of the paper but for the authors knowledge:

- Line 26-27: It might help to clarify the adjoint method is only memory efficient in integration time, and not size of the dynamics function for example.
- Between lines 52 and 53: I would change "would take too much memory" to "would require too much memory".
- Line 85: "then its log likelihood from Chen et al.:" should be changed to "then its log likelihood from Chen et al. is given by:"
- Line 107: Implies $s \in [0, 1]$ when it seems that $s \in[0, T]$
- Line 200: "NFE is a indicator" should be "NFE is an indicator"
- Line 230: "Differential equations are solved using the adjoint method and a Runge-Kutta of order 5 of the Dormand-Prince-Shampine solver." Should be "Differential equations are solved with a Runge-Kutta of order 5 of the Dormand-Prince-Shampine solver and trained with the adjoint method".
- Line 236: "using a lower NFEs" should be "using a lower NFE".
- Line 265: "and can combine with" should be "and can be combined with"
questions: My only question is how best to view these C-NODEs, as they are solved with blackbox ODE solvers. Are they more like ODEs or PDEs? And if they are like ODEs is it possible for a NODE (or an ANODE/Latent NODE) to learn the same dynamics as C-NODE. Is it better to think of C-NODE as limiting the dynamics function so it has to be in the form of a characteristic curve? Similar to how convolutions share weights across linear layers to enforce translation symmetry.
limitations: The paper sufficiently explores its limitations. There is no broader impact statement, it isn't really required here, however, it is good to keep in mind that the applications of NODEs and C-NODEs can include some potentially unethical ones. This is because they can generally be applied to learning time-series and now partial differential equations. As said, this is not the case in the current form of the paper, the work is incremental and not applied in those directions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose Characteristics NODE (C-NODE) which parametrize the method of characteristics ODE of Hamilton-Jacobi equations. The authors provide density estimates for C-NODE. The authors backs up their claim of improving performance with experiments on images and time series.
strengths_and_weaknesses: Strengths:
- The idea of combining method of characteristics and NODE is very straightforward and new.
- The density estimates are theoretical supported and works well on CNF type problem in their experiments

Weakness:
- Section 3 is a little bit messy. It is easy to find out the idea behind, but it is a bit confusing on things like whether they want to solve the PDE numerically or the characteristics ODE numerically; how they will do interpolation after they solve with characteristics ODE, etc.
- Experiments on PDE modeling and time series prediction are synthetic data only. It might be converntional for PDE modeling, for time series prediction, this is not a very convincing experiment.
- For problems without a general PDE view, like image classification and time series prediction, the model is simply combination of equations (5) and (6), which is basically more augmentations on NODE, which may not be something new. Still, this provides an interesting PDE viewpoint that is very inspiring for CNF tasks.

questions: - In algorithm session, only procedures for image processing tasks are discussed. What about the other ones?
- Also in algorithm session, what is your starting x(s=0)?
- Equations like burgers may produce shock or rarefactions during their solution. How do you deal with those?

limitations: MOC only applies to Hamilton-Jacobi equations (not just hyperbolic), but still, it does not apply to all equations and may fail to model some type of problems. This should not be an essential problem since it could be solved with Augmentation.
review_title: null

Meta-review: metareview: This paper proposed to model the evolution of the latent variables to the characteristic curves instead of the original ODEs. Authors proved the new method C-NODE is more expressive than the original NODE. Experiments are conducted on image classification tasks to demonstrate its effectiveness. It will be good explorations to leverage the differential equation theory to improve the NODE algorithms. The insights from the journey will help innovate breakthrough directions in operator learning. During the discussion phase, reviewers had rounds of debates about whether the method is demonstrated effective on standard tasks for NODEs.  Although it is a high bar for exploration style work to achieve SOTA results,  we expect some insights from the investigation. For example, why the former NODE is not expressive enough in certain tasks, what might be the factors in the real tasks influencing the expressiveness, why the image classification task need extra expressiveness, etc. Simulations can also be involved to show these insights in extreme cases. 
confidence: Less certain
Decision: Reject