Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: I really enjoyed this paper. It's one of my favorite papers from 2020. Authors: thank you for writing it!

The paper proposes a framework for jointly fitting the parameters of a physical model (such as the parameters of an ODE or PDE) and learning a neural network to model the error or residual of this physical model. The idea is to find a physical dynamics model $F_p \in \mathcal{F}_p$ (e.g. where $\mathcal{F}_p$ is a set of PDEs with different parameter values) and a neural residual dynamics model $F_a \in \mathcal{F}_a$ (e.g. where $\mathcal{F}_a$ is a hypothesis class of neural networks) which minimize the norm of $F_a$ while constraining the composed dynamics $F = F_a + F_p$ to agree with observed data. Interestingly and importantly, the paper proves the minimum-norm decomposition of the observed dynamics into physical model dynamics and neural residual dynamics is unique, given a condition on the geometry of $F_p$. This condition is that $\mathcal{F}_p$ should be a Cheybshev set: a sufficient condition is that it is a closed convex set in a strict normed space. To me, this condition seems very mild.

The paper goes on to show that this method, termed APHYNITY, produces significant gains in predictive accuracy over purely learned methods, purely physics-driven methods, and other forms of combining physical models and learned models. The gain is most significant when the physical model is incomplete. When it is complete there may be some small advantage due to the neural residual accounting for some discretization error; regardless there is no harm, and the learned neural residual is very small. 

The paper also shows that APHYNITY produces better parameter estimates for the physical model in the presence of incomplete physics. This seems important. Our physics models are always approximations and in many interesting applications (climate/atmosphere, bioengineering, mechanics of materials, etc) commonly used physics models may have an interestingly-sized gap with reality. When the identified parameters, not the predictions, are needed for some downstream task such as decision making, APHYNITY could help with better parameter ID.

I thought this paper was clear and well written. The main paper presents an easy-to-follow story, the appendices contain plenty of detail, and when while reading the main paper I wanted more detail on specific points, it was usually easy to follow links to the correct appendices. (Should be true of all papers, but often isn't). 

---
Thoughts and feedback: 
- It would be useful to see some simple visual demonstration of the effect of APHYNITY on parameter estimation in incomplete models. (I should note the paper is already quite long and thorough, though).
- The integral trajectory-based approach (section 3.2, 3.3, motivated in Appendix D) used to fit the parameters of both NN and physics model seems like the "right" way to do this to me. Nonetheless it would be interesting to see numerical comparison with the alternative (supervision over derivatives).
- I wonder if the integral trajectory-based approach, vs the supervision-over-derivatives approach, can be related to traditional methods for parameter ID in ODE?
- It seems to me that the authors do a good job explaining and relating to prior work on learning physical systems. However, with both the density of recent literature in this space and the decades of work combining ODE solvers and function approximators, if there are missing references (papers on similar work that the current submission does not cite), it's quite likely I would not have noticed.
- It would be interesting to know if this could help figure out *in what way* a physical model is misspecified, and guide design of an approximate physical model that better captures reality. I suspect yes, although it might simply boil down to this method having a better estimate of the residual than if one just does a least-squares fit with the physical model.

---
Typos:
- "bayesian" -> "Bayesian" (end page 2)
- "si" -> "is" (page 3)

---
I think this is a well written paper likely to be of interest to a large number of ICLR attendees, with some important novel contributions, and that the method proposed has a good chance of being widely adopted in the subfield of ML+physical simulation. 
review_title: Great paper about jointly identifying parameters of physical models and modeling the residual with a NN

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper outlines a method for forecasting and parameter estimation when you have a partial physics model (possibly with unknown parameters) and time series data. This is a hybrid approach where the data-driven (deep learning) approach only learns the parts not accounted for by the physical model. A key feature is being able to decompose the problem in such a way that the data-driven model only models what cannot be captured by the physical model. The parameters of these two models must be fit jointly so that the physical model's parameters are more correct. They prove existence and uniqueness for this decomposition. 

Strong points:
I think this point of making the decomposition unique (and thus better fitting the physical model's parameters) is intuitive and a good one. To my knowledge, other work on hybrid models do not have theory showing this. (But I haven't extensively read the literature.)

The authors included extensive experiments with multiple versions of each of three test cases, plus several baselines. They also conducted ablation studies for three features of their method: 
- using a decomposition to ensure that the data-driven model only learns what the physical model cannot contribute
- the trajectory (integral) approach instead of training on estimated derivatives
- using adaptive weights in the objective function

Weak points:
I think that the claim "We show that APHYNITY can perform similarly to complete physical models by augmenting incomplete ones, both in forecasting accuracy and physical parameter identification." is overstated. I compared APHYNITY for incomplete physics to the Param PDE or True PDE with complete physics in Table 1, and this claim seems true for the wave equation only.

I think that the related work section should acknowledge that people have been using neural networks for hybrids with physical models since the 1990s. Example: [1]. This is sometimes called "gray box" modeling or "hybrid" modeling. Since this section only mentions very recent work, I'm not very confident in the paper's claim that they are the first to have a principled decomposition with existence & uniqueness.

I think that "Propositions 1 and 2 provide, under mild conditions, the theoretical guarantees for the APHYNITY formulation to infer the correct MB/ML decomposition, thus enabling both recovering the proper physical parameters and accurate forecasting." is overstated. The theory is about the decomposition existing & being unique, not about this method being able to find that decomposition. 
1. The theory makes no assumptions about the form of F_a (the data-driven component). Showing that the decomposition is possible is not the same as showing that there exists a neural network that could represent F_a. The different versions of the universal approximation theorem for neural networks that I've seen all require that the neural network is approximating something reasonably well-behaved (such as continuous). I could imagine that the parts of a dataset that cannot be modeled by an ODE/PDE might also be non-continuous, etc.
2. Further, even if the universal approximation theorem applied here, that would show that such a neural network exists, not how well any existing training methods can find that neural network.

I think "as expected, ||F_a||^2 diminishes as the complexity of the corresponding physical model increases, and becomes almost null for complete physical models." is too strong. It does decrease, but I wouldn't call 0.14 and 2.3 "almost null." 

I personally can't vouch for the theory (in Section 3.1 & Appendix A & B), as I'm having a hard time following it. I hope that another reviewer can. Some things that would help me:
- The intuition would be easier if it's mentioned earlier what \mathcal{F} is in this paper's examples. L^2? It would be nice to put it before Section 3.1, right when the notation is introduced. Similarly, it would help if an example of \mathcal{F}_p was given early on. 
- Some examples are given in Section 3.1 & the appendix of conditions that make a set Chebyshev. However, since I was having a hard time knowing which sets & spaces were important in this paper, I was having a hard time understanding which conditions were important. 

Since reviewers aren't required to read the appendix, and it's quite long, I read it less closely. However, it contains the proofs and backs up many of the nice claims in the paper. This means that quite a bit of content in the paper is less vetted. 



Other clarification questions:

I may have missed it, but does the paper describe how parameters are fit for the PDEs (physical models)? Example: ParamPDE(a,b) for reaction-diffusion. 

Does the requirement of \mathcal{F}_p being Chebyshev mean that the parameters of the ODE/PDE need to be closed on one end? In other words, we couldn't search for a parameter in (-infinity, infinity)? 

Are the results in Table 1 test errors? (as opposed to training or validation?)

It's not clear to me what you mean by the "partially observable" setting.

For the ablation studies, we're presumably comparing these variants to the full method's results, as described in Table 1, right? However, there are some discrepancies between Table 1 and Tables 5 & 6. Some of the numbers don't match and one of the wave equation cases is not included in Table 5. The "Augmented True PDE derivative supervision" case is also missing for the wave equation in Table 6. 
 

Minor points:

It seems that the definition of the set of observed trajectories (in the beginning of Section 3) is not right. 
- Your observations are for a finite set of values t in [0,T].
- Your observations are not across all solutions X for the dynamics: just at a finite number of IC/BCs. 
- For the reaction-diffusion & wave equations, X is also a function of spatial x, and the data is collected at finitely many spatial points. 

I think instead of defining a set \mathcal{A} on page 3, it would be clearer to use the space \mathbb{R}^d? I don't think F maps into just the set A, where A is defined as the set of values that X takes. Couldn't the derivatives have values that F doesn't reach?

On the bottom of Page 3, the question of the minimum being well-defined is explained as the existence question. However, I'm used to "well-defined" in various mathematical contexts meaning roughly "a unique answer." Is there a different meaning of "well-defined" in this context?  

Summary: I tentatively recommend accepting this paper. Their method has quite a few empirical results showing improvements. People are interested in combining physical models with machine learning in a variety of scientific application areas, so I could see this being a well-used method. However, I have listed some points above that I would like fixed or clarified. I also hope that someone else can vouch for the proof.  


[1] Rico-Martinez, R., J. S. Anderson, and I. G. Kevrekidis. "Continuous-time nonlinear signal processing: a neural network based approach for gray box identification." In Proceedings of IEEE Workshop on Neural Networks for Signal Processing, pp. 596-605. IEEE, 1994.

review_title: tentative accept

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ### Summary of my understanding

The authors propose a method of function fitting for differential equations. They premise that a model $F$ is the additive combination of $F_p$ and $F_a$, which denote physics and augmenting parts, respectively. The functional form of the physics part, $F_p$, is given in accordance with prior knowledge, whereas the augmenting part, $F_a$, is modeled by neural nets. The proposed method follows the principle of least action of $F_a$, and the authors suggest solving a constrained optimization problem via a method of Lagrange multipliers. They show numerical results on three PDE/ODE-governed systems.

### Evaluation

I really enjoyed reading the paper, which is well written. The motivation is clearly presented. The related work can be more detailed given the recent active studies on physics + ML but seems sufficient from the viewpoint of ODE/PDE fitting. The proposed method is simple yet reasonable, and the experiments are enough supportive to see the superiority of the method. The ablation study in Appendix F is also very interesting. Possible improvement, which the authors would be aware of, can be found in the lack of experiments on real-world datasets. (I understand the difficulty of finding illustrative real-world examples in this kind of problem, but I couldn't stop pointing out it.) Overall, I think this is certainly sound work.

review_title: Sound work

Meta-review: The authors propose a method for modeling dynamical systems that balances theoretically derived models, which may be grounded in domain knowledge but subject to overly strict assumptions, with neural networks that can pick up the slack. All reviewers were enthusiastic about this work, appreciating its balance of mathematical rigor and experimental assessment. One concern was that this paper follows on decades of related work, which was difficult to adequately summarize. However, changes made throughout discussion phase did address these concerns.
Decision: Accept (Oral)