Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method, called Neural-IVP, for solving partial differential equations (PDEs) using neural networks. They use various helpful techniques to stabilize the PDE solutions, increase the scalability, and improve the representation power of the neural networks. Neural-IVP is a local in time method, meaning that the time dependency is induced by having time-variant parameters in the neural networks. They improve the representation power by using sinusoidal embedding and solving the last layer of the neural networks linearly. In order to increase the scalability and the number of parameters in the neural networks, they propose to use Jacobian-vector-products implemented using automatic differentiation, which makes the complexity of time and memory grow linearly by the number of parameters. 
strength_and_weaknesses: Strength:

- Showing the deficiency of the PDE solvers.
- Suggesting new techniques to overcome the shortcoming of the PDE solvers.
- Increasing the scalability and conditioning of the PDE solvers.
- Proving (via theorems) why singular solutions happen (leading to ill conditioning) due to continuous parameter symmetry.

Weakness:

- Limited applications and examples.
- Lack of evaluation in high-dimensional PDE problems.
clarity,_quality,_novelty_and_reproducibility: The paper is sufficiently clear and novel.
summary_of_the_review: I believe that using neural networks to solve PDEs is an interesting and important problem to pursue. This paper has done a good job in showing the drawbacks of the previous methods for solving PDEs and has tried to address those by using various techniques. However, one main shortcoming of the paper is the lack of high-dimensional problems/examples. One of the selling points of this paper is the improved scalability. So, one expects to see its scalability to high-dimensional settings as well. In the section 5 of the paper, the title say "high-dimensional hyperbolic PDEs" but the paper only shows a (3+1) dimensional spacetime example. I am very curious to see if this method can work in hundreds-dimensional problems where neural ODE methods can work without problem? Or, if there is a limit in achievable dimensions, the authors should be clear about it. Also, reporting the training time versus dimension could also be helpful to see. 

Minor comments:

There are some typos throughout the paper. These are some of them:

- be emploted to computer --> be employed to compute
- due the challenges --> due to the challenges
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes a method which helps improve scalability of the local-in-time ODE solvers.

strength_and_weaknesses: Strengths:
- The reviewer really enjoyed the problem statement in sections 1-3; it gives a good grasp on what the problem is and why the current (ML and non-ML based) solvers struggle to achieve good quality solutions for high dimensional problems, as well as complex initial conditions and for higher-order PDEs
- The problem is really interesting and in many aspects understudied

Weaknesses:
- Some of the details of the descriptions are failing to give the consistent picture of what the method looks like; see comments 3, 4
- Experimental assessment: the current experimental assessment lacks detail on why it is better than the existing solvers; see comment 5

clarity,_quality,_novelty_and_reproducibility: Reproducibility: see question 3 below; there are questions about reproducibility as currently the reviewer thinks the improvements are needed.

Novelty and clarity: The paper seems novel, to the best of the reviewer's understanding, and the first three sections are really well written. On the description of the method and the experimental sections, the answers need to be given. The clarity of the method description needs to be improved (see comments below).

Questions on quality and clarity:

1. (Minor) Dimensionality: page 1, “Rather than compute costs which scale exponentially in the dimension, they instead typically have error” N must be defined 

2. (Minor) Typos should be proofread e.g. “Page 2: These methods has proven successful “
 Also, should local in time solvers be spelled ‘local-in-time solvers’ as it is a compound adjective (see https://proofed.co.uk/writing-tips/a-guide-to-compound-adjectives-and-hyphenation/ for a better explanation of my point)? This question is totally subjective, and the reviewer intends it as an open question, which is in no way is reflected in my score. 
3. While sections 1-3 give an excellent background,  it looks like section 4 does not give the method in one place in all its entirety; the reviewer also thinks that the method does not look reproducible from the description. Section 4.1 discusses a toy problem of a sum of two Gaussian like wave packets on which different  solvers are evaluated; is the proposed model a modified Du & Zaki (2021) architecture, or is it based on Bruna et al’s architecture? What is the compete list of modifications of this architecture?  It could be presented, for example, as some sort of algorithm. What are the hypoerparameters of the model?
4. In section 3, the description states that the computational complexity of the model is O(p^3 + p^2 n) and memory O(p^2 + pn); what is n and what is the exact architecture for which this assessment is made? 
5. For the more complicated scenario, described in section 5, there need to be a comparison of models (with Bruna et al (2022) and Du& Zaki(2021)) and ablation studies: which of the proposed improvements actually make the impact on the solution: sinusoidal embedding, head fine-tuning etc? How does it work for different parameterisations of the equation? Up to what dimensionality is the solver still able to persist with stable learning, and at what dimensionality the proposed numerical scheme collapses? Is it possible to give more examples, e.g. some second-order PDEs or problems with the complex conditions, where the standard solvers fail (see Section 3 )

summary_of_the_review: The reviewer thinks that while the introduction and the summary of the challenges of scalability of the local-in-time solvers are really clear and well written, the details of the proposed method and  be experimental assessment could be improved (see comments above).

==
Updating the scores to recommend acceptance as the authors, in my view, sufficiently addressed the comments
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper developed Neural-IVP, an ODE based IVP solver, to solve initial value PDEs. Neural-IVP is based on the local in time methods, but the authors show that local in times methods have issue of representational power and numerical stability. Neural-IVP is developed by addressing these two issues.
strength_and_weaknesses: Strengths
- The scalability and numerical stability issues of local in time methods are analyzed.
- New techniques are proposed to address those issues.

Weaknesses
- A few techniques are used together to improve the local in time methods. Each of these techniques is not totally new. It is also unclear how important of each technique.
- There is only one example in this paper. Many more examples are needed.
- Even for this example, there are not sufficient convincing results. There are no quantitative errors.
- In the paper, it claims about “high dimensional” PDEs. However, the example in Section 5 is a 3D problem, not high dimensional at all. Examples of at least 10 dim are required.
- If time dependent 3D problems are “high dimensional”, then many existing papers have solved time dependent 3D problems by deep learning methods, such as physics-informed neural networks (PINNs). A lot of relevant literature review is missing. 
- There are no comparisons with other methods (either deep learning or conventional methods) in terms of accuracy and speed.
- In Introduction, “Rather than compute costs which scale exponentially in the dimension, they instead typically have error …”. There are no definitions of N and epsilon. This sentence is not a rigorous mathematical statement. Also, references are required.
- In Introduction, “more that 3 dimensions”. A typo here.

clarity,_quality,_novelty_and_reproducibility: The paper is clear.
summary_of_the_review: The paper proposed extensions of local in time method, but the experiments are not convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper is on the theme of developing better PDE solvers using deep learning techniques. The paper points out that techniques where a neural net globally parameterizes the spatio-temporal solution are susceptible to "catastrophic forgetting" as they use a single set of time-dependent weights. Using time-varying weights leads to the "local in-time" methods, but they are not very robust. The paper addresses these issues by developing a "Neural IVP" method that allows large capacity networks to be used while retaining stability and well conditioning. 

Strengths: Proposals to improve PDE solvers are insightful: using fast matrix multiplies and the use of last-layer linear solves and sinusoidal embeddings improves representational capability; using regularization, restarts and last layer finetuning, instabilities are effectively mitigated.   

Weaknesses: Restricted to low dimensional PDE solutions though claims are made on ability to handle "high dimensional" problems. Limited in experimental results - weak in terms of comparisons to other methods in terms of accuracy and speed. 


justification_for_why_not_higher_score: The weaknesses listed above make this paper fall short of accept with spotlight. 
justification_for_why_not_lower_score: Paper merits acceptance given the reviews and the growing interest around using deep learning techniques to improve the state of the art in solving PDEs.
Decision: Accept: poster