Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a way to interpolate between gradient descent and Gauss-Newton's method for solving nonlinear least squares problems arising from physics informed training. The authors cite past work that physics informed training is often ill conditioned, so gradient descent often performs poorly. They give a classic example of ill-conditioning, and show that gradient descent converges slowly, while Gauss-Newton quickly saturates neuron activations.

This motivates them to introduce Half-Inverse Gradients (HIG), which interpolates between GD and GN using the SVD. The authors then try the method on several test problems in scientific computing: Control of nonlinear oscillators, Poisson, and control of a Quantum dipole. They compare Adam, HIG, and GN.
main_review: The paper starts out with a very strong overview of past work, and cites Holl et al. (2021) for the motivation that existing optimizers do not perform well on joint optimization of NNs and physics. It would be very helpful if the paper was a little more self contained in the motivation. I see several references to "physics solvers" and "ill-conditioning" but no real motivation as to why these are related. The authors give an example of an ill-conditioned quadratic, which helps motivate why incorporating neural networks with ill-conditioned problems could lead to difficulty, but it's not clear why using "physics solvers" causes this.

Furthermore, I would like to see "physics solvers" to be defined a little bit more explicitly in the beginning of the paper. I was not able to quite understand until reading the appendix and all the case studies. I now assume that you mean a time-stepping scheme that you can differentiate through end-to-end (or using adjoint methods).

Section 2:
- It would be very helpful if you made vectors more explicit in the notation.
- e.g. In the "Physics optimization" subsection, you say "the sum reduces to a single data point". This confused me for a bit, until I realized $x$ and $y$ are vectors (but even then MSE would reduce to a sum of squared terms over the dimensions of the states).
- Equation (3) would be helpful to emphasize that you're using a pseudoinverse, not a standard inverse.
- It took me some time to understand that batch dimension is the number of points in the target state.
- Did you experiment with different configurations of $\kappa, \beta, \tau$? It would be interesting as ablation experiments to disentangle the benefits of these parameters. In particular, does most of the benefit come from $\beta$? Or is there also a benefit in setting $\kappa$ to an intermediate value?

Nonlinear oscillator:
- It might be useful for people who are less familiar to write out the actual time-stepping you get with Hamilton's equations. Maybe this can go in the appendix.
- How many Adam steps are you able to perform in one HIG step? It would also be useful to see number of steps required vs batch size.

Poisson problem:
- Correct me if I'm wrong, but it seems like this setup is different from the other ones in the sense that it's a surrogate training problem, rather than optimizing through a physics solver. Is that correct? I think it would be helpful to make this clear.
- The pretraining is interesting. Would GN perform well when pretrained with Adam?
- re: "the Poisson problem is relatively simple, requiring only a single matrix inversion": Arbitrarily ill-conditioned quadratics can also be solved by a single matrix inversion. I'm not convinced that is the reason for the observations.
summary_of_the_review: The paper proposes a potentially interesting method to interpolate between GD and GN, and is certainly moving in the right direction of a very important problem in scientific machine learning, but the paper as it is right now is a little confusing to read. It would also benefit a lot from some ablation experiments showing the benefit of each individual part of their methods. I vote for marginally below threshold, but am willing to change my score if my concerns are addressed.

Let me know if you have questions about my review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This manuscript introduces a new optimization fashion bridging the Gauss-Newton method and the vanilla gradient descent method in the middle for physical optimization with neural networks. Traditionally, when a neural network is used for a physical problem, it will inevitably be affected by the unbalanced magnitudes dramatically. This manuscript proposes HIG, which is a middle point of two popular and individually advantageous optimization methods. It also provides a nice guideline on how to choose the hyperparameters and examine the efficacy of HIG on a concept-illustrating synthetic problem and three realistic physical problems.
main_review: I personally feel excited about this direction. Data-driven physical learning has been a hot area recently where progress is being made in multiple parallel paths including problem formulation, network architectures, loss specifications, etc. The optimization paradigm has been an unavoidable component in deep learning, and this work fits into the place as the analogy of its counterparts in vision/NLP problems. This manuscript has shown convincing theoretical intuition and validated on valuable physical problems, nevertheless, it arouses my curiosity in several directions:

1. I found this work strongly parallel with a recent pre-print [1], which I would like to refer to as a concurrent work to this manuscript. I am well aware that a comparison is not required but would like to hear more qualitative discussion.
2. It seems to me that the loss function in the toy example should be $l(y,\hat{y},\lambda)=\frac{1}{2}(y^1-\hat{y}^1)^2+\frac{1}{2}\lambda(y^2-\hat{y}^2)^2$, or did I miss something? Also the notation (superscripts and subscripts) are confusing.
3. The computational cost seems high. The Jacobian inference, in spite of being vectorized for a subset of modules, can be a memory monster. I expect some results and discussions on the memory footprint. Also, I wonder if the method can benefit from the Krylov subspace method where an iterative method is used for speedup.
4. My experience with Adam tells me learning rate scheduler is a good friend of first-order methods. I wonder if one is used for Adam baselines in the experiments. If not, how does it perform?
5. A very common trick in training neural networks is adding normalization layers (in recurrent cases, layer normalization is more popular). How does this method deal with normalization layers? Does it require it? If it does, how to update it during training?

[1] Holl, Philipp, Vladlen Koltun, and Nils Thuerey. "Physical Gradients for Deep Learning." arXiv preprint arXiv:2109.15048 (2021).
summary_of_the_review: This manuscript fills a blank for optimization in physical deep learning with a nice and clean method. The presentation is relatively clear to me. I would recommend an accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the training of neural networks for physical simulations. By distributing the burden equally between network and physics, the paper presents the half-inverse gradients (HIGs) method. Experiments show its advantage for achieving a faster and more accurate minimization.
main_review: Strengths:
The paper presents an efficient method for training neural networks for physical simulations. The idea is simple and is easy to understand. The written and organization of paper is clear and easy to follow.

Weaknesses:
I have some concerns on the experiments.
1. It is better to compare the batch size of HIGs with the Gauss-Newton method.
2. The truncation threshold \tau is important for both accuracy and efficiency. It is better to discuss its effect in more details. Also, how to set \tau for a problem? 
3. What does \gamma refer to throughout our the paper?
4. Just Below Eq.(2), "f" should be "$f$".
summary_of_the_review: The paper is well-written and organized. The idea is simple and effective. It is better to discuss more details on the settings of the hyper-parameters in the proposed method.
review_title: null

Meta-review: Thank you for your submission to ICLR.  The reviewers and I are in agreement that the paper presents a substantial contribution to the field at the intersection of differentiable simulation and ML methods.  In particular, the half-inverse method is compelling, non-obvious, and hints of a nice path forward towards the goal of practical differentiable simulations within models.  Overall I'm happy to recommend the paper be accepted.
Decision: Accept (Spotlight)