Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors look at solving inverse problems with a neural network training scheme. They use a scale-invariant physics solver to compute the gradient update, and then train the neural network using this information. They show that this SIP training approach can converge faster to inverse problem solutions over optimizers such as Adam and SGD for various physical problems.
strengths_and_weaknesses: Strengths:

- Solving inverse problems is a long-standing problem in science and engineering, and the authors propose an interesting technique to integrate the physics solver with a neural network to address this challenge
- The method seems to converge more quickly compared to other optimization approaches, and can reconstruct solutions more accurately
- The paper is well-written for the most part

Weaknesses:
- The authors consider an infinite data setting, so it isn’t entirely clear what the generalization benefits of this method are
- The authors compare to mainly first-order optimizers and one second-order optimizer. It would be interesting to see how well the Newton’s method-based second-order optimizers do here, as are often used in solving numerical inverse problems (L-BFGS, etc.). There are also some promising results using other second-order optimizers, as well as exploring the use of starting with a first-order optimizer and then switching to a second-order optimizer. 
questions: - What is the advantage of the SIP solver vs. using the adjoint method as a way to back propagate through the weights?

- Can you clarify the problem setting better for the problems (such as Poisson and heat equations)? That is, exactly what the parameters being learned correspond to (i.e., by solving the inverse problem).

- How does the method compare to traditional numerical inverse solvers?

- Could the authors clarify what you mean by the network weights are not being altered?
limitations: Limitations are discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes scale-invariant learning for solving inverse problems. The state of the art machine learning techniques used to infer solutions to inverse problems rely on first order optimizers such as SGD or Adam. The highly non-linear behavior in physical processes result in suboptimal optimization direction. This work tackles this issue by proposing a 2 stage process which combines higher order optimization methods with Deep Learning. Specifically, their method uses the solutions from a scale invariant physics solver as targets for the neural network during training. Finally, they demonstrate that their method is much more effective than the existing methods on numerous problems such as - poisson's equation, Heat equation, Navier-Stokes equation.
strengths_and_weaknesses: Strength:
* The 2-stage SIP pipeline Clearly outperforms the traditional training pipeline that uses gradient based optimizers to optimize the joint problem NN \odot P. 
* This work has provided convergence proof for their algorithm.
* Empirically demonstrate the effectiveness of their work on problems such as - poisson's equation, Heat equation, Navier-Stokes equation.

Weakness:
The authors have only compared their methods only against optimizers such as SGD, Adam, AdaHessian which seems a bit 1-dimensional. Currently the story is that SIP is better than the very traditional training pipeline and to me, the contributions seem limited to some extent. I believe the authors can paint a stronger picture if they compared their method against other popular approaches such as - neural operators [1] and Physics Informed Neural Networks [2]. I'm happy to increase my scores if the authors could compare their methods against these baselines.


[1] Kovachki N, Li Z, Liu B, Azizzadenesheli K, Bhattacharya K, Stuart A, Anandkumar A. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481. 2021 Aug 19.

[2] Karniadakis GE, Kevrekidis IG, Lu L, Perdikaris P, Wang S, Yang L. Physics-informed machine learning. Nature Reviews Physics. 2021 Jun;3(6):422-40. 


questions: The authors have only compared their methods only against optimizers such as SGD, Adam, AdaHessian which seems a bit 1-dimensional. Currently the story is that SIP is better than the very traditional training pipeline and to me, the contributions seem limited to some extent. I believe the authors can paint a stronger picture if they compared their method against other popular approaches such as - neural operators [1] and Physics Informed Neural Networks [2]. I'm happy to increase my scores if the authors could compare their methods against these baselines.


[1] Kovachki N, Li Z, Liu B, Azizzadenesheli K, Bhattacharya K, Stuart A, Anandkumar A. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481. 2021 Aug 19.

[2] Karniadakis GE, Kevrekidis IG, Lu L, Perdikaris P, Wang S, Yang L. Physics-informed machine learning. Nature Reviews Physics. 2021 Jun;3(6):422-40. 
limitations: The authors have adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Inverse problems that involve physical systems usually have strongly varying gradients. This may lead first-order optimizers like SGD or Adam to compute suboptimal optimization directions. To handling this issue, the paper embeds a scale-invariant inverse problem solver into the gradient-descent-based learning pipeline. More specifically, the paper reformulates the objective function in the observed outcome space to that in the physical parameter space. It needs to compute the inverse of the physical process for each training step. Some theoretical and experimental results are provided.
strengths_and_weaknesses: Strengths:

The paper presents a new approach to obtain scale-invariant updates of physical parameters, i.e., when rescaling the physical parameters, the optimization will behave the same way. Namely, it uses the objective function in the physical parameter space instead of the observed outcome space. Some acceleration tricks and convergence analyses are also presented. It is a useful complement to scale-invariant optimizers, e.g., Newton’s method, for high-dimensional parameter spaces. 

Weakness:

See the comments for the "Questions” below.

questions: After reading the response and the comments from other reviewers, most of my concerns are addressed. So I increase my score to 6. My remaining concern is the approximate derivation in subsection 3.1. If there are no approximations, the performance is worse or better?
==================================================
There are some comments and suggestions.
1.	Highly related methods are missing for comparisons. There are many optimization methods to avoid compute the inverse of Hessian matrix. For example, the Hessian-free method. James Martens. Deep learning via Hessian-free optimization. ICML, 2010. It is better to include these methods for comparisons.
2.	The paper assumes that the physical process P has a unique inverse and is sufficiently well-behaved. It is better to give plentiful examples. 
3.	At lines 134-147, the paper approximates P^{-1} with its first update. It needs more elaborations on this. What about iteratively solve P^{-1} precisely? 
4.	At line 249, x has a square.

limitations: The paper did not have the limitations and potential negative societal impact.
review_title: null

Meta-review: metareview: Reviewers agreed that solving inverse problems is a long-standing problem in science and engineering, and the authors propose an interesting technique to integrate the physics solver with a neural network to address this challenge.
confidence: Certain
Decision: Accept