Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use evolutionary sampling to mitigate failures in training PINNs. The main idea is based on the premise that solution of a PINN propagates from boundaries towards the interior regions. In contrast to existing work, Evolutionary sampling is able to adapt to the changing PDE residual error landscape while being computationally efficient. Experiments are conducted on multiple challenging PDEs, and the results demonstrate the effectiveness of the proposed approach.
strength_and_weaknesses: Strengths:
1. Well written, proposed method is sound for the most part.
2. Experimental validation is conducted on multiple problems that are challenging for standard PINN methods and the proposed method is effective in practice.

Weaknesses:
1. I am concerned about the practical viability of the idea of adaptive sampling. In real-world scenarios how realistic is to expect the PDE solution to be available at arbitrary space-time coordinates on demand? I imagine these are obtained through sensor measurements, which may be too constrained to obtain at arbitrary points or too expensive to obtain.
2. While the proposed method is based on a population of samples, there are no mutation or crossover (interaction) operators as you would expect in a typical evolutionary algorithm.

Other Comments:
- Not a weakness per se, but all the references to the appendix in the main paper were missing. This should be an easy fix.
clarity,_quality,_novelty_and_reproducibility: - The exposition in the paper is very clear. I enjoyed reading the paper.
- The quality of the paper in terms of ideas, explanations, analysis, experiments and writing are very good.
- Code is available for the paper. Although I did not test it, the code along with the experimental details in the paper should be sufficient to reproduce the main results of the paper.
summary_of_the_review: I am positive about the paper, except for the probable impracticality of real-world adoption. The paper is very well written, and the experiments and analysis are thorough. Most importantly, the method seems to be effective across a range of PDEs.

Post Rebuttal Update:
I read the other reviews and the author's responses. I will maintain the initial rating for the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes "the propagation hypothesis" as an explanation of why some (collocation) sampling schemes might cause PINNs to fail.
To mitigate the issue, the authors propose an _evolutionary_ sampling strategy, that keeps points with high error in a pool and resamples others. They also incorporate a time-weighting scheme to propagate the importance of samples from $t=0$ onwards. The authors show that their method outperforms prior art in 3 datasets when the number of samples used is small.
strength_and_weaknesses: ## Strengths
* Authors formulate an intuitive concept to identify failure modes of PINNS;
* Authors propose a simple strategy that improves significantly the sample efficiency of PINNS;
* Ideas are clearly presented and the work is easy to follow.

## Weaknesses
* The idea of focusing on regions of high error is not exactly novel in PINNs;
* While the "propagation hypothesis" is intuitive, there is no formal characterization for it in the work;
* The experimental campaign is shorter than I would expect from a paper with a  very simple idea and no theoretical guarantees;
* The theory is inconsistent (and perhaps wrong) --- see details below.
* The experiments lack a more thorough discussion. For instance, do you have an intuition on why Evo is capable of solving the Eikonal equations in some cases (Figure 8) and other methods not? 

###More about the theory

For **theorem 3.1**, there are two issues: 

1. The step from lines 5 to 6 on page 11 of the appendix multiplies by $V^{-1/2}$ but should multiply by $V^{1/2}$ since $p(\mathbf{x}_r)^{-1} = V$;

 2. The paragraph before theorem 3.1 is misleading. It says that sampling proportional to a power of $\mathcal{R}$ is equivalent to changing the $L^p$ loss in the norm. However, minimizing $\mathcal{L}^2_r(\mathcal{Q}^k)$ is not the same as minimizing $\mathcal{L}^2_r(\mathcal{Q}^k)$  (or a power thereof) os not the same since $Z= \int_{\mathbf{x}_r \in \Omega} |\mathcal{R}(\mathbf{x}_r) |^k d\, \mathbf{x}_r$ depends on network parameters (and therefore is not a constant). 

For **theorem 4.1**, states that, as the number of iterations of Algorithm one approach $\infty$, their loss reverts to an $L^\infty$ loss. However, on page 5, the authors state that training PINNS with $L^\infty$ is undesirable and can lead to $oscillatory behavior between different peaks of the PDE residual landscape". Does this mean we should always use a small number of iterations in Algorithm 1? This aspect deserves further discussion.

clarity,_quality,_novelty_and_reproducibility: Clarity: Overall, the paper is well written. 

Quality: I have some concerns regarding the theory (listed above).

Novelty: Somehow,  proposals are mostly reappraisals of previous techniques for PINNs (i.e., weighting based on error and enforcing "physical causality").

Reproducibility: The authors also provide code and experiments seem reproducible. 
summary_of_the_review: Given that the novelty is somehow limited and my concerns with the implications of the theory & the short experimental campaign, I recommend rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel sampling strategy for collocation points to improve the training performance of PINNs. The strategy is based on the propagation hypothesis, which suggests that information from the boundary of the computational domain has to "propagate" to its interior, and that this propagation sometimes fails during PINN training. The authors show that the approach indeed leads to more stable training.
strength_and_weaknesses: The paper is excellently written, and the arguments for the Evo strategy are clear. The approach is simple, intuitive, and yet effective. The experimental evidence is good, although I would appreciate results with more benchmarks.

The two weaknesses I see are that 1) the benefit of Causal Evo is not shown or discussed in the experiments, and 2) that comparisons with some parts of the literature are missing entirely or difficult to evaluate.

## Ad 1)
I understand Evo as a method that mitigated training failures, rather than one that improves performance for a successfully trained PINN. Thus, from that perspective, Evo and Causal Evo perform equivalently in the shown experiments. It would be interesting to see an example where Evo fails because causality is not respected, but where Causal Evo successfully trains the PINN.

## Ad 2)
I am missing other approaches that have been proposed to mitigate training failures. These approaches include domain decomposition, loss weighting schemes, the architecture from arXiv:2109.09338, or the regularization term proposed in arXiv:2112.05620. It would further be interesting to see if and how the sampling scheme can be combined with the former methods. Further, the comparison is difficult to evaluate because the chosen benchmarks show little overlap with what has been proposed in the literature. For example, CPINN was evaluated in the original paper on a set of benchmarks that does not intersect with the benchmarks chosen in this paper. I would be interested to see how Evo performs on these benchmarks.

## Minor
- In the paragraph "Propagation Failure", the authors argue that trivial solutions can become attractive for PINNs. This has been made precise theoretically in arXiv:2109.09338 and experimentally in arXiv:2203.13648.
- References to the appendix are broken.
- The computational overhead of the comparison methods is not clear ("Challenges with Potential Remedies"). Similarly, for Evo it is not clear how $\mathcal{P}_i^s$ changes throughout training (i.e., is it of fixed size, or is rather $\mathcal{P}_i$ of fixed size?).
- The four salient properties seem somewhat artificial and post-hoc. For example, the gradual accumulation property is not well justified. As an alternative property, it seems equally intuitive that initially the collocation points should concentrate around the initial and boundary conditions (ensuring successful learning locally, from which propagation can continue), while at later stages the collocation points should be uniform (ensuring uniform accuracy across the computational domain). Thus, a priori, the CPINN sampling strategy seems equally powerful to mitigate propagation failures as Evo. It is even more interesting to see that the CPINN fail for the systems for which Evo succeeds, and I would appreciate an evaluation of Evo on the benchmarks that were chosen by the original authors of CPINN.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and of high quality. The novelty is somewhat limited by the fact that several sampling schemes have been proposed so far, and that the experimental evaluation is limited to only two systems (one with two different parameterizations). Code and datasets are available, which makes the paper reproducible.
summary_of_the_review: A clear paper with an intuitively appealing, simple, and yet effective method to mitigate training failures in PINNs. To judge if Evo is a major contribution towards PINN training requires more evidence.

*EDIT:* Raised my score according to the discussion with the authors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: For this paper, we have two "borderline" (though slightly positive) reviews and one clearly negative review. On the positive side, all reviewers agreed that the main idea is well-motivated and that the paper is well-written. All other aspects, however, have been discussed in a more controversial way. 
Severe concerns include the somewhat missing conceptual novelty, not fully convincing comparison studies and  problems with the main concept of the "propagation hypothesis". In the end, none of these two reviewers finally wanted to assign a clearly positive score, and none of them was willing to champion this paper.

A third reviewer was highly critical from the beginning, mentioning  issues with the theoretical foundation (Theorems 3.1 and 4.1). It seems that none of these fundamental points of criticism could be addressed convincingly in the rebuttal, and this reviewer clearly expressed that the paper was "not good enough". Actually, I agree with most of the critical points raised, and I think that this paper generally lacks a clear and transparent theoretical basis. Therefore, I vote for rejection.    
justification_for_why_not_higher_score: There are too many open questions regarding the validity of the formal concept proposed.
Decision: Reject