Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper questions the status quo of using L2-loss for optimizing PINNs. Theoretical results show that a learned solution to HJB is (Z1, Z2, Z3)-stable iff the physics-informed loss uses Lp loss with sufficiently large p. Empirical results partially support the theory and show that optimizing L-inf loss will result in lower L1 error in the case of a linear LQG problem. As L-inf loss is challenging to optimize with Adam, the paper proposes an adversarial training algorithm that indeed outperforms optimization of L-inf with Adam. 

strengths_and_weaknesses: Strength:
- The work is significant. The broader field of learning surrogate models of PDEs can have wide-reaching impacts in physics, chemistry, fluid dynamics, biology, climate modeling and more. Within the field of phyisics-informed machine learning, the authors have identified and addressed a very important research question: Does there exist theorems that connect properties of PDEs to the optimal choice of PINN loss function. The choice of HJB equations is important and developing the theory on HJB equations only should be sufficient for acceptance. 
- The authors provide rigorous and sound theory to prove that a learned solution to HJB is (Z1, Z2, Z3)-stable iff the physics-informed loss uses Lp loss with sufficiently large p.
- The authors clearly state the research question [L34-35a] and include necessary background math, e.g., Def. 4.2.

Weaknesses: 
- One of the authors main contributions seems to be the notion of (Z1, Z2, Z3)-stable. However, it is still unclear to me 1) if (Z1, Z2, Z3)-stable actually is a novel concept that the authors came up with, 2) what the concept can and cannot be used for, and 3) how the concept relates the 'stability' in PDEs (see Qestuions 5-6). 
- It is challenging to fully evaluate originality of the work, because the related works section is very sparse. I have inluded some questions to evaluate originality and am willing to raise my score.
- The empirical evaluation does not fully support the theorems, nor does it fully answer the research question. I have included some questions to evaluate the quality of the empirical section.

questions: Results and Impact:
1) The authors evaluate the approximation quality of their method on L1 error (Table 1). What is the justification for only using L1 to empirically define a "good approximator of the exact solution" (L34)? Indeed, the proposed L-inf loss results in a lower L1 error than L2 loss on a high-dim. linear LQG problem. But, it is unclear to me if L-inf loss will also result in a lower L2 error. I would be willing to raise my score with a comparison of L2 error and a discussion of it. 
2) Can a general statement be made that L-inf loss will be a more appropriate loss choice for high-dimensional problems?

Related Works:

3) What is the broader novelty wrt. prior works that use deep learning for HJB? The related works state that there exists "several works" that solve HJB with deep learning, yet the authors only mention two works [L82-90]. A quick Google shows that there is other works by, e.g., Nüsken and Richter, 2021; 
4) There exist works that use GANs instead of L2 loss. What is the advantage of using L-inf over the GAN loss?
5) Is (Z1, Z2, Z3)-stable a new concept developed in this paper or does it already exist in the literature? If so, what is the citation for (Z1, Z2, Z3)-stable.
6) It is confusing to me that authors use the word 'stable' to talk about approximation quality in Sec 4. More specifically, how does (Z1, Z2, Z3)-stable relate to the general definition of stability in PDEs [L93-95]? An "equation is stable if the solution of the perturbed PDE converges to the exact solution as the perturbations approach zero [6]"

limitations: - There is no discussion of limitations nor of negative societal impacts.
- It would be very helpful if the limitation section could list and analyze the assumptions in Definitions/Theorems 4.1-4.4 and discuss what the extra steps would be to adapt the Definitions to other classes of PDEs beyond HJB.
- Are the theorems truly applicable to *all* variations of the HJB equations?
- What does (Z1,Z2,Z3)-stable mean and what does it *not* mean?
- It would be helpful if the authors mention that, while the work is very theoretical, it could be used to improve numerical modeling of applications that violate the NeurIPS ethical guidelines.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the objective function in physics-informed learning for HJB equations. Authors propose the concept of stability to analyze what is a good choice of loss function. The theoretical result challenges the common practice of $L^2$ and suggests that $L^p$ with large $p$ gives stability guarantee. The authors propose an adversarial training algorithm to minimize $L^{\infty}$ loss based on their theoretical finding.  The proposed method demonstrates superior empirical performance in the simulated high-dimensional problem.
strengths_and_weaknesses: Strengths:
1. This paper is well-motivated and well-written. Stability analysis is important for using physics-informed learning to solve PDEs. 
2. The theoretical results advance the understanding of loss design in physics-informed learning.  
3. The proposed method demonstrates superior empirical performance compared to other baselines in the simulated high-dimensional problem. 

Weakness:
1. Adversarial training is time consuming and it introduces more hyperparameters to tune. 
2. The experiment only considers two special cases (including the one in the appendix) of HJB equations which have close form solutions. The one in the appendix doesn't compare with other baselines. The empirical evidence is not strong enough to me. 
3. The theoretical results suggest Lp loss with a large p guarantees better stability than L2. However, the results in Table 2 turn out to be the opposite. Explanation in the Section 6.2 seems vague. 
4. The setting seems restricted. More details are provided in the questions. 
questions: 1. In the stochastic differential equation given by Eq (4), the output of control function has the same dimension as the state. Also, the control function only depends on time. But I think the control variable usually has different dimension from the state variable and the control function should be a function of both time and state. The setting described in Eq (4) seems too simple compared to what I found in the literature equation 2.10 in section 2.4 [1].  Is this a standard setting in optimal control? 
2. Given the results in Table 2 and what authors said in page 5 -"$L^p$ and $L^{\infty}$ -norm behave similarly when $p$ is large", it seems that minimizing the exact $L^{\infty}$ loss may have poor performance. The reason why the adversarial training works better than $L^2$ seems to be that adversarial training fails to approximate $L^{\infty}$ -norm well. Is it possible to check 
3. Given the training objective in Eq (8), shouldn't the objective functions in algorithm line 5,7,8 use absolute value instead of $L^2$? 

[1] Lu, Q., and Xu Zhang. "A mini-course on stochastic control." _Control and inverse problems for partial differential equations_ 22 (2016): 171-254.
limitations: Listed in the weakness and question sections.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the choice of loss functions when using neural networks to solve partial differential equations. The authors establish positive and negative results showing the conditions under which the problem of solving a Hamilton-Jacobi-Bellman equation is stable/unstable. Based on this, an $L^\infty$ training approach is proposed. Experiments demonstrate that the proposed method has better performance than existing methods.

strengths_and_weaknesses: Strengths: 

The results of this paper are solid and comprehensive. 

The paper is well-organized and easy to follow. 

The experiments clearly demonstrate the advantage of the proposed $L^\infty$ loss over the original PINN methods.



Weaknesses: 

This work focuses on a quite specific class of HJB equations and requires $\bar{c} \leq 2$ for the stability result to hold, which seems to require $\alpha_i \geq 2$ in the cost rate function.

Table 2 in the ablation studies seems to contradict the theorem results. Some more discussions may be quite helpful here to explain why finite but larger values of p lead to worse performances.

questions: I suggest that the authors could address the weaknesses mentioned in the strengths and weaknesses section.
limitations: This paper does not have any potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Te present work is concerned with the application of physics-informed neural networks (PINNS) to the solution of high-dimensional Hamilton-Jacobi-Bellman equations (HJB). They show that in a wide range of cases directly applying the standard squared penalty to HJB and boundary conditions does not result in a stable problem, meaning that the PINN loss can go to zero without the PINN solution approximating the true solution. In order to overcome this problem, the authors propose to use the higher-order $L^p$ penalties. They show that higher-order $L^p$ penalties achieve stability under a wider range of conditions, with $L^\infty$ being the extreme case. They then design an adversarial training procedure to train with the $L^\infty$ loss in practice and show that it improves the relative accuracy of the solution compared to existing PINN variants. 

### After rebuttal:
The authors have addressed my concerns appropriately. I now recommend acceptance. 
strengths_and_weaknesses: Strengths: The method is well-motivated and seems to perform well in practice on the problem of interest.

Weakness: As detailed under "questions", the present version of the paper leaves doubts regarding some of the claims. The experimental validation seems to be in a setting not covered by the theory and the most common diagnostics, such as error vs time, are missing.

I want to emphasize that I think that this has the potential to be a solid paper suitable for acceptance. I am giving a somewhat lower score right now but hope to be able to raise it if the authors can clarify my concerns.  
questions: 1. In the theorems 4.3 and 4.4, the target accuracy is measured in the $W^{1,r}$ norm, meaning in terms of the $L^r$ norm of both the solution and its gradient. However, in the experiments, the error is measured in terms of just the $L^p$ norms of the solution. Thus, experiments do not actually seem to illustrate the theoretical results as advertised.

2. Following up on 1, are similar results true when measuring solution accuracy in $L^p$ spaces (instead of sobolev spaces)? If not, the authors should provide more motivation accuracy in sobolev spaces is important. 

3. Did you find that different learning rate schedules had an effect on the accuracy of Adam? 

4. The error plots of Figure 4 are not very meaningful instead, or complementing them I think it would be important to show learning to show time series of the training and test error. This would allow to distinguish difficulties of the finite batch setting to accurately approximate the infinite-batch-training loss from difficulties in optimizing the finite batch loss, from the problem of stability that is motivating the author's technique.  
 
limitations: As discussed above under "questions", there presently seem to be gaps between theory and experiments that are not commented on in the text.
review_title: null

Meta-review: metareview: The reviewers reached a consensus that this paper meets the bar for being accepted at NeuRIPS, and therefore the AC recommends acceptance. Please refers to the reviews and author's responses for reviewers' opinion on the strength and weakness of the paper. 
confidence: Less certain
Decision: Accept