Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This work proposes Differential Equation GAN (DEQGAN) for solving differential equations using generative adversarial networks (GAN) where the generator learns to output the solution and the discriminator tries to learn a good loss function. The authors compare DEQGAN against classical PINNs over twelve ordinary differential equations (ODEs) and partial differential equations (PDEs). The experimental results show that DEQGAN can achieve multiple orders of magnitude lower L2 error than PINNs.
strengths_and_weaknesses: Strengths: the paper is well-written and easy to follow. DEQGAN demonstrates superior empirical results compared to classical PINNs. The fact that DEQGAN doesn't depend on predefined distance such as L1, L2 can be useful in some problems.

Weakness:
1. The Related Work section misses some important discussion [1, 2]. The authors should discuss the relationship to works and place the contribution of DEQGAN in the context of prior works. For example, SA-PINNs [1] can also be viewed as learning a loss function.
2. The authors state that the L2 loss lacks theoretical justification but miss the important explanation on why and when L2 loss can be problematic. Furthermore, DEQGAN do not have a theoretical justification of the learned loss function either.
3. The loss function in DEQGAN is borrowed from generative adversarial network (GAN). The original GAN work designs its loss function such that generator is minimizing Jensen-Shannon divergence between the generated distribution and target distribution given optimal discriminator dynamics. However, the Jensen-Shannon divergence doesn't make sense in DEQGAN because the problem is to solve a differential equation with unique solution. Even with Gaussian instance noise, minimizing the divergence between two Gaussians will just be minimizing the L2 loss. I do not see why a generative model is needed here.
4. The equations considered in the experiments are artificial. Only one specific initial condition is used for each equation, which raise the concern that the authors may pick the initial conditions to put DEQGAN in advantage in the experimental results.
5. DEQGAN has training instability issue. To solve one equation, it needs to repeat runs multiple times and filters out the ones with poor performances.

[1] McClenny, Levi, and Ulisses Braga-Neto. "Self-adaptive physics-informed neural networks using a soft attention mechanism." arXiv preprint arXiv:2009.04544 (2020).
[2] Daw, Arka, M. Maruf, and Anuj Karpatne. "PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics." Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.
questions: 1. The authors motivate the need of learning loss function by saying that L2,L1 loss functions lack theoretical justification. However, the author challenges the standard practice in physics-informed learning without explaining why and when L1, L2 loss functions can be bad choices. If L2 loss is problematic, how is the learned loss function better than L2 loss? Can the authors provide theoretical justification for it?
2. How fast is DEQGAN compared to PINNs and numerical solvers? Can authors provide time complexity comparison against PINNs and its improved variants such as SA-PINNs?
3. Only one specific artificial initial condition is used for each equation. How will DEQGAN perform if the initial condition is a random function sampled from a random field?
limitations: 1. Lack of theoretical justification of the loss function and how the disciminator can learn a better objective function.
2. DEQGAN has training instability issue. To solve one equation, it needs to repeat runs multiple times and filters out the ones with poor performances.
3. The equations considered in the experiments are artificial. Only one specific initial condition is used for each equation, which raise the concern that the authors may pick the initial conditions to put DEQGAN in advantage in the experimental results.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a new way to solve differentiable equations by leveraging GAN-based adversarial training to “learn” the loss function for PINNs.
strengths_and_weaknesses: Strengths
-Provides a novel method to solve Differential Equations with GANs in an unsupervised training setup. They do not use the solutions to the equations.
-As can be seen from the results it improves the performance on the Differential Equations task by multiple orders of magnitudes.
-They authors also integrate different effective methods in Section 3 for stable training of the GAN.
-Good paper presentation and a lot of details for the experiments in the Appendix. I also like the 


Weaknesses
-No discussion for mode collapsing. I understand that this is a different application of GANs but it would be good to have a discussion on mode collapsing. Maybe the proposed GAN can provide different solutions for some equations.
-No theoretical analysis or guarantees are provided in the work. GANs in general they do not have a lot of guarantees but a theoretical analysis would be much appreciated.
questions: -It would be good to see an ablation study where they see the effectiveness of the “tricks” that they used to train the GAN effectively because some of them may not be needed.
-Can you make some comments on mode collapsing as a I mentioned on the weaknesses above?
limitations: The authors have addressed the limitations on the Conclusions section. The work is not applicable for negative societal impact in my opinion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method to solve ordinary and partial differential equations using GANs. Following in the tradition of physics-informed neural networks (PINNs), the flow equation is parametrized by a neural network. The parameters of the neural network are optimised by replacing a hand-tuned loss function such as the mean-squared error, with a discriminator and an adversarial loss. This method appears to provide improved performance over PINNs trained with hand-tuned loss functions and numerical solvers, on several well-known differential equations.
strengths_and_weaknesses: Strengths:
1) The paper is very well-written, well-structured and easy to follow. The method and results, for the most part, are described in clear detail.
2) The ablation study, and discussion of methods to ensure stable training were extremely interesting and relevant, given that GANs are known to be difficult to train.
2) The extensive results in the paper make a convincing case for using GANs over other methods for solving differential equations -- in particular, with respect to the variability of the solutions from other methods across different problems.

Weaknesses:
1) Many of the details of training, hyperparameters etc., included in the methods section (sections 3.2.1, 3.2.2, 3.4), while important for reproducibility, distract from the main message of the paper, and hinder readability.
2) Some of the results require a lot more elucidation: e.g. the details of the ablation study in section 5.2 are mostly relegated to the appendix, and thus make the import of the results hard to parse; the plots in Figure 3 are barely mentioned or described in the text. 
3) The paper could improve with a more extensive discussion of the limitations of the method (the conclusion mentions non-interpretability of the discriminator, but not much more).
4) The paper claims to present two "methods" to improve robustness in training: residual monitoring and addition of instance noise. While it is important to document the techniques used to stabilised training, neither of these are nescessarily novel contributions in the context of GANs. Furthermore, it is not clear whether these measures would be equally effective without spectral normalisation, skip connections in the generator, or with different choice of variance for the instance noise, or a change in the 25% training iteration threshold for residual monitoring.

questions: As outlined in the previous section, I believe the paper makes a strong case for using GANs to solve differential equations. Some concrete suggestions to address the weaknesses:
1) It would be good to move the details of sections 3.2.1, 3.2.2 and 3.4 to the appendix, and move details of the ablation study to the main paper. Furthermore, sections 3.3 and 4.1 describe training / engineering choices to a level of detail that hinders readability, and can certainly be condensed.
2) It would be good to have a more extensive discussion of Figure 3. Moreover, Figure 3 appears to show only simulations from the GAN -- it would be good to have some examples in of the simulations from other methods as well, in order to contextualize the GAN performance beyond the reported MSE numbers.
3) While the methods residual monitoring and instance noise are certainly important details for reproducibility, it is not convincing that they would work without the other hyperparameter or architectural choices. It would therefore be good to modify the discussion of these two "methods", and to present them as potential avenues for stable training rather than a contribution of the paper.

Questions:
1) The performance of the different methods for solving differential equations are compared solely on the basis of MSE. Is there a particular reason for choosing this metric? -- it is particularly surprising that there is no justification provided for this in the paper, since the introduction makes a strong point that the choice of similarity metric typically used to optimize parameters for the PINNs are arbitrary. It would also be good to see whether the GAN performance is equally good under other similarity metrics e.g. L1, or MMD 
2) In line 181-182, "We use the ReLU function...noise should not be used." -- this logic is not clear. Why should $L_g < L_d$ on any particular iteration indicate a better generator? -- this could also be due to initialisation, especially at the beginning of training.
3) How does the coarseness of the grid from which $t, x$ are sampled affect the performance of the GANs over other methods?

Minor comments:
1) The notation in section 4 is confusing in places: $G(x)$ and $\Psi_{\theta}(x)$ are used somewhat interchangeably; in line 179-181 $L_g$ and $L_d$ are not defined.
2) In Figure 2, a legend is not required in every panel
3) In Figure 3, the dotted lines in (a) and (b) are note identified in the legend; the colourbars have no label in (c) and (d)  
limitations: It would be good if the discussion could address the following issues:
1) While the discriminator is not interpretable, the generator has potentially the same issues. What scientific insight is gained from an (effectively) black box generator modeling a flow equation?
2) While the GANs do provide an improvement over existing methods in terms of performance, it would be good to have an idea of the associated computational costs, sample efficiency and training times compared to the other methods.
3) How does this method scale with the number of dynamical variables, the size of the coarseness of the $t, x$ grid?
review_title: null

Meta-review: metareview: This paper presents a new method for solving differential equations using generative adversarial networks to "learn the loss function" for optimizing the neural network. After the discussion, the reviewers still have a few major concerns: 

(1) The authors' claim that the existing methods lack of theoretical justifications. However, the paper does not provide a sufficient justification on their proposed method, either, which makes their key motivation of the paper questionable.

(2) Some important baseline methods are missing in the comparison as well as references. The authors should improve their literature survey.

(3) The computational challenges of solving PDEs mainly lie in high dimensionality. Most of existing deep-learning based PDE solves, including PINN, attempt to demonstrate the benefit of using deep neural networks for approximating high dimensional functions or operators. However, the experiments only consider low dimensional PDEs, which not difficult to solve, and existing numerical methods can solve them efficiently without deep neural networks and complicated tuning. 
confidence: Certain
Decision: Reject