Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper argues for the combination of a neural network with an incomplete description of a system (a hybrid model), in their case, an incomplete PDE, rather than a fully data-driven approach that only relies on a neural network. They compare the two approaches on a set of systems described by thermodynamics and fluid dynamics. 

The authors consider a supervised setting where the complete PDE, or solutions for it, are given and used for training the neural networks with an MSE loss. In this setting, their experiments suggest that the hybrid models outperform the data-driven approach by a significant margin. In addition, they argue that hybrid models can lead to faster solving of the PDEs than relying on the complete description when it is available. 
strength_and_weaknesses: # Strength
The idea of using the physical understanding of the studied phenomenon, rather than an entirely data-driven one, makes a lot of sense to me. In addition, the systems studied in this paper seem novel to the hybrid learning literature and constitute solid test scenarios for assessing future developments in hybrid learning. 

# Weaknesses
Overall I find the contribution very limited. Combining a neural network with equations from physics is a long-standing idea in the community. This paper does not introduce anything subtle regarding a good strategy to combine these equations with neural networks. 

The experimental validation of their approach also seems limited in terms of the train/test scenarios considered. In addition, it is unclear how the authors selected the best models and hyperparameters, as no cross-validation or validation set is discussed. This also reduces the solidity of the presented results. For instance, I am pretty sure that at some point, the data-driven approach could catch up with the hybrid approach if the number of train scenarios increases. An analysis of when this (might) happen would have been valuable to motivate the hybrid approach for these problems.

It is unclear to me whether the parameters of the PDE are also learned at training time. If not, this makes the proposed approach's applicability very limited as, in most cases, if we can observe a system and have little understanding of the physics behind this also means that we do not know the corresponding parameters of the incomplete physical description. I have probably misunderstood something there, but I would encourage the authors to clarify this in the paper. Overall, this paper would benefit from a real-world demonstration of the proposed approach. Indeed, the contribution is not methodological. Thus I would argue that this should contain a more solid empirical evaluation.

As a remark, I also feel that some confusion is made between the "model" and the "inference" (in the case of PDEs, the solver). The neural network aims to complete the model, create a more accurate description of the real world, and create a better model. The solver is related to how we make inferences given a model; the terms complete and incomplete solvers do not make sense to me.
clarity,_quality,_novelty_and_reproducibility: # Clarity
The paper's clarity is ok but should be improved to be accepted for publication, in my opinion. For instance, see my confusion regarding some of the points mentioned in the weaknesses of the paper.

# Quality
I did not find the experimental validation solid enough to trust the empirical conclusion made by the authors entirely. For example, the authors do not explain how they selected the different architectures and training hyperparameters. 

# Novelty
The novelty is limited. The main contribution is to showcase a simple hybrid approach to original test scenarios.

# Typos and minor remarks
- What is u in eq (1)?
- The second paragraph before section 3: don't -> do not
- The paragraph before section 3: Why don't you discuss Takeishi and Kalousis 2021 (https://scholar.google.com/citations?view_op=view_citation&hl=fr&user=rqF9bAsAAAAJ&citation_for_view=rqF9bAsAAAAJ:hC7cP41nSMkC)? I am not an author of this paper, it is a genuine remark. I also wonder why you do not compare to an approach similar to APHYNITY (https://scholar.google.com/citations?view_op=view_citation&hl=fr&user=rFaxB20AAAAJ&sortby=pubdate&citation_for_view=rFaxB20AAAAJ:IZKZNMMMWs0C).
- Eq (3): a point at the end of the equation is missing.
- Eq (5): the upper limit should be t + m.
- unfiform-Busen -> Uniform-Busen.
summary_of_the_review: Overall I recommend rejection of this paper for the aforementioned reasons. In particular, the lack of novelty of the approach and the limited reliability of the experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper aims at deploying a deep learning approach in combination with partial differential equations (PDEs) with known yet incomplete physical information. 
For example, a deep learning model can be employed when the information on the system to be solved is limited but additional data are available. A common benchmark is represented by turbulence modelling in computational fluid dynamics where the full PDEs descriptions are too expensive to be solved computationally. Therefore, the PDEs can be rewritten as a sum of an incomplete PDE description plus another term representing the physics arising from the unknown information (reaction rate terms). Correctly representing this additional term is of crucial importance to solve the task. 

Contrarily to standard Neural PDE solvers, this method aims at combining physical incomplete PDE information with some additional machine learned correction due to the unknown part of the PDE description. In this work, the latter is represented by the chemical kinetic mechanism of non reactive flow simulation. The complete (reactive) simulation is used as a baseline. 
The downstream task is to solve a simplified version of the Navier-Stokes equation for a specific system where the chemistry is removed from the analytic description (i.e., is unknown).

A neural network is trained to learn correction fields that would correct the incomplete (non reactive) PDE equations where both source terms and reaction rates have been removed.
The work compares three baselines: Ground truth, Pure Data Driven approach, Fourier Neural Operators (state-of-the-art neural operator method).

The framework is clearly explained in figure 2 where the PDD and the NN-Hybrid approaches are compared in B1 and B2 respectively. 
The 3 different setups on which the approach was tested are carefully explained.  

A few comments/concerns are in place:

- The following sentence is a bit unclear to me ‘Its objective is to learn to model the effects of the unknown chemistry using the neural network parameters θ given an input flow 
 State […]’. What is it meant by ‘[…] using the neural network parameters \theta […]’?
- In the last row of table 1 some baselines are missing (i.e., FNO and NN.) I presume this is because they never converge to the target value? It would be helpful to write something about it in the table caption. 
- On page 7 below figure 4, in the second paragraph the authors define the relative displacement. Right afterward, they mention x_t and \tilde{x}_0. I cannot relate these quantities easily to the definition and this may be overall a bit confusing. I would suggest to keep the definition of relative displacement general \tilde{x}_300 -> \tilde{x}_t and they specify that t=300. As it is now the discussion might not sound very linear and straightforward. 
- I would suggest the authors to add additional plot to figure 6 to visualise the performance of other baselines like FNO and NN. From the discussion of table 1 is already clear they perform worse than NN-PDE, but visualising them (as it’s done in figure 12) might be useful.
- In section 5.3 the author only discuss the full NN method without mentioning the NN-PDE. I am wondering it that is intentional. If so, there’s a reason for that? Are there any limitations that prevent NN-PDE to be applied for this task? 
- I suggest the authors to make the labels explicitly in figure 2 saying what’s the PDD and what’s the NN-hibridy approach for an easy first-sight intuition.
- The caption of figure 2 does not explain what is S. Though this is done in the main text would be useful to repeat it in the caption below the figure. 
- I’d rename section 2 something like ‘Literature Review’ or ‘Related Work’. To me ‘Background’ sounds more like if you are introducing the physics behind the work.  
- Missing full stop at the end of equation 3. 
- Few typos/stylistic issues were found throughout the manuscript. I’d recommend the authors to run a spellchecker for making sure to detect them.
strength_and_weaknesses: The problem tackled by the paper is of great practical relevance. It is common in the physical sciences to face numerical PDEs where the complete description is either very expensive, from a computational perspective, or non accessible. Therefore, when a full analytic description is available, as well as reference, experimental, data, one can combine a numerical PDE solver for an incomplete description  of the full PDE with a correction term which can be learned by a NN upon training on some appropriate data. 

On the other hand, physics-informed ML is a quite well-established avenue of research. It is not at all surprising to me that combining incomplete physical information of an incomplete PDE description with a NN that learns phenomena acting on different time scales enhance the performance. Therefore, while I am not too impressed nor surprised by the results, I do indeed see some value in the method and I thereby recommend consider this manuscript for acceptance.  

As a further suggestion the author may study some transfer learning properties, i.e., how well the correction NN would perform when being trained on a specific set of initial condition but used in different setups. 
clarity,_quality,_novelty_and_reproducibility: - The paper is very clear. 
- The writing style is appropriate and engaging. 
- The method presented in the manuscript is understandable also for people which are not familiar with the specific application. 
- While I am not domain expert, to my knowledge there are no other references where a NN is trained to correct for some lack of information in the PDE description. 
- The method is described in a clear and accessible way making it appealing and deployable to different other downstream tasks involving complicated PDEs.  
- The authors assert the code will be released upon acceptance. However, the manuscript provides quite a high level of details which I imagine  would make the results easily reproducible. 
summary_of_the_review: To incorporate physical prior information when training a neural network to solve a physical problem is known to be a successful approach. 
In this paper the author propose to learn an incomplete description of a PDE and correct a posteriori using a NN for removing two terms which in general act on different time scales. Doing so, the PDE solver happens to be much more efficient in learning the dynamics of the incomplete system. Contrarily,  when learning the full dynamics of the complete PDE description (pure NN approach) the interplay between different time-scales makes the learning task much more difficult. 

The idea of separating the PDE and learn part of the dynamics in a supervised learning fashion and correct the PDE result a posteriori is indeed a valuable and interesting contribution to the community. 

Despite not being a domain expert, I still do see some potential in this work, especially in future developments. 
For these reasons and all the considerations from above, I would recommend to consider this manuscript for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the task of learning continuous dynamical systems from data using a hybrid (machine learning + PDE solvers) approach and compares the performance to purely data-driven methods. In particular, authors consider a regularly occurring in practice setting where only a partial knowledge of the governing equations are available. Authors use reactive flow systems (specifically various flame simulations) as the test ground for evaluation and comparison. They find that by incorporating physical priors in the form of a partial PDE solver improves the performance of the learned model and is able to accurately capture the dynamics of the system and works well for test simulation settings. Additional findings include observation that learned hybrid models result in less stiff updates compared to the underlying ground truth solver.
strength_and_weaknesses: Strengths:
* Authors consider a dynamical system that is more complex than that commonly studied in learned simulators literature.
* The considered method is tested against relevant baseline approaches.
* Paper is clearly written.

Weaknesses:
* Standard deviation estimates of the quantitative results (e.g. Table 1) are O(1) wrt. relevant mean values. While visualizations support the conclusions based on the trend of the mean MAPE and MSE, I find the results somewhat hard to trust.
* An important time subsampling hyperparameter has not been considered for data-driven models.
* AFAICT datasets are not made available to perform independent evaluations/tests.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and provides a decent account for the choices made to set up the test problem. The use of existing framework for fluid dynamics solver (phiflow) is a plus. I did not find any reference to the source code/datasets; and given the complexity of the setup (numerical PDE solver + ML model) I think it’s quite hard to independently reproduce the results based on the paper alone.
summary_of_the_review: I enjoyed reading the paper in general, as the authors address an important question of whether physical priors help/needed to model continuous dynamical systems well. I find the step up in complexity of the simulated system a bonus and in general think this is an important research avenue.

I’m generally leaning towards accepting this paper (score of 8), assuming some flaws and weaknesses can be addressed. (hence the initial score of 6).

My (1) primary concern is related to the evaluation metric and error analysis. Unless I’ve missed something, it is not clear how significant the mean values of MAPE and MSE are given the standard deviations. Is there a way to present the results in a way that clearly indicates that hybrid approach generally improves over data-driven methods? Maybe using a histogram of errors over different sample trajectories / random seeds would provide a more complete picture of error distribution? The fear is that coarse summary might hide stability problems of the purely data driven approaches which would misrepresent their performance as there are numerous methods to address stability issues.

My (2) concern is related to the time-step used for data-driven predictions. In general, data-driven approaches often subsample the predictions in time, which result in better accuracy and efficiency. (e.g. see Fig. 3 “Stachenfeld et al., 2022” (reference used in the paper)). Given that the underlying ground truth solver involves stiff dynamics it could turn out that data-driven models are evaluated in a highly suboptimal regime where they are tasked to predict a very slowly varying solution.

Finally, I think the paper would read better if the stakes for “primacy” were omitted. Arguably a number of previous works learned various closure models in a tandem with differentiable solvers and present work just extends it to a more challenging setting, which is important in its own right.

I’d be happy to provide an additional set of minor comments regarding potential improvements wrt references after primary concerns are addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the case of incomplete physics for numerically solving Partial Differential Equations. It uses a neural network to complete the missing terms and more accurately forecast the evolution of the dynamical system. The proposed approach is evaluated on different variants of a reactive flow problem governed by Navier-Stokes equations. 
strength_and_weaknesses: Weaknesses

The paper claims that it is the first of each kind that learns to complete physics in a PDE setting. Nevertheless learning with incomplete physics models using neural networks to complete the missing part has been studied rather extensively. For example: APHYNITY, Yin et a, Journal of Statistical Mechanics: Theory and Experiment (also cited in beginning of the present submission)  and ICLR2021; Physics integrated VAEs for robust and interpretable generative modelling, Takeishi & Kalousis, NeurIPS, 2021. APHYNITY tackles forecasting tasks, similar to the current submission, for physics systems that are described by ODES/PDES. It models the underlying dynamical system with two additive components, one given by the known physics and a neural network modelling the unknown parts. Physics VAE learns generative models where the VAE decoder also decomposes, to a more general composition than additive, of an incomplete physics component and a neural network. The paper should at least discuss differences from these two works. 

With respect to the experimental evaluation it is not clear to me how the neural network baseline has been trained. It seems that it learns state-state transitions. How is this done? does the model during training sees only consecutive state pairs? or does the model evolve in a longer horizon, i.e. over a sequence of states, and then the model update would have to happen over such sequences, accounting like that for longer temporal dependencies and error compounding. From what I understand the neural network baseline is trained over state state transitions, which, if I am correct, puts it at a clear disadavantage with the hybrid model. More appropriate baselines will explicitly model for long term dependencies, e.g. a solver where there is no physics but just a neural network, neural ODEs, models with recurency. The two papers above provide a number of such baselines. 

Strengths
There is a detailed evaluation of the model in a complex scenario exploring different extrapolation scenarios. 

clarity,_quality,_novelty_and_reproducibility: The paper is mostly easy to follow. I have some doubts with respect to the originality since it seems to me that APHYNITY and Physics VAE address very similar, of not the same settings. 
summary_of_the_review: This is mostly an application paper that proposes to learn to complete incomplete physics models using neural networks in the context of reactive flow problems. My main concerns with the paper is how it relates to previous work that also learns to complete incomplete physics models as well as the fact that, at least to my understanding, the baselines that are considered are weak.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers learning dynamical systems using hybrid ML + PDE solvers. They consider a setting were the governing equations are partially known and modeled by the solver, while the ML component complements for the unknown part. The hybrid model is motivated by a specific problem, the modeling of reactive flow systems, more precisely flame simulation. Experiments compare the proposed approach with pure ML approaches and show large improvement by incorporating the physics.

The question addressed in the paper has already motivated several contributions in the physics aware ML literature. The originality of the paper is to consider a problem more complex than the ones classically used in this literature. The approach demonstrates the relevance of the approach when compared to pure ML methods and even some benefits compared to pure simulation. The appreciation by the reviewers is mixed. This may be due to the chosen approach which does not develop a ML view point with for example the development of new ML-physics ideas, but focuses on solving a specific problem. The main concerns raised by some reviewers are the absence of comparison with closely related approaches, the incremental novelty from the ML side and insufficient baseline comparisons. Overall, one considers that the paper addresses an important question and has several merits but in its present form it is probably more adapted to an applied science venue than to a ML conference. The applicative scope should probably by widened for a submission to an ML audience.

justification_for_why_not_higher_score:  Incremental novelty from the ML side and insufficient baseline comparisons
Decision: Reject