Adversarial Adaptive Sampling: Unify PINN and Optimal Transport for the Approximation of PDEs
Authors: Kejun Tang
Jiayu Zhai
Xiaoliang Wan
Chao Yang
Venue: ICLR 2024 poster (2024)
Keywords: adversarial adaptive sampling
optimal transport
neural network approximation of PDEs
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Solving partial differential equations (PDEs) is a central task in scientific computing. Recently, neural network approximation of PDEs has received increasing attention due to its flexible meshless discretization and its potential for high-dimensional problems. One fundamental numerical difficulty is that random samples in the training set introduce statistical errors into the discretization of the loss functional which may become the dominant error in the final approximation, and therefore overshadow the modeling capability of the neural network. In this work, we propose a new minmax formulation to optimize simultaneously the approximate solution, given by a neural network model, and the random samples in the training set, provided by a deep generative model. The key idea is to use a deep generative model to adjust the random samples in the training set such that the residual induced by the neural network model can maintain a smooth profile in the training process. Such an idea is achieved by implicitly embedding the Wasserstein distance between the residual-induced distribution and the uniform distribution into the loss, which is then minimized together with the residual. A nearly uniform residual profile means that its variance is small for any normalized weight function such that the Monte Carlo approximation error of the loss functional is reduced significantly for a certain sample size. The adversarial adaptive sampling (AAS) approach proposed in this work is the first attempt to formulate two essential components, minimizing the residual and seeking the optimal training set, into one minmax objective functional for the neural network approximation of PDEs.

Published as a conference paper at ICLR 2024
ADVERSARIAL ADAPTIVE SAMPLING: UNIFY PINN AND OPTIMAL TRANSPORT FOR THE APPROXIMATION
OF PDES
Kejun Tang; Jiayu Zhai; Xiaoliang Wan Chao Yang
PKU-Chaner PKU-Changsha Institute for Computing and Digital Economy
Institute of Mathematical Sciences, ShanghaiTech University Department of Mathematics and Center for Computation & Technology, Louisiana State University &School of Mathematical Sciences, Peking University & PKU-Changsha Institute for Computing and Digital Economy tangke code. pku. edu. cn, zhai jyŁ aaf@shaanhaitecch edu. cn x ilwanmmath. edu, chao.yangêpku. edu. cn
ABSTRACT
Solving partial differential equations (PDEs) is a central task in scientific com- puting. Recently, neural network approximation PDEs has received increas- ing attention due to its flexible meshless discretization and its potential for high- dimensional problems. One fundamental numerical difficulty that random sam- ples in the training set introduce statistical errors into the discretization of the loss functional which may become the dominant error in the final approximation, and therefore overshadow the modeling capability of the neural network. In this work, we propose a new minmax formulation to optimize simultaneously the approxi- mate solution, given by neural network model, and the random samples in the training set, provided by a deep generative model. The key idea is to use a deep generative model to adjust the random samples in the training set such that the residual induced by the neural network model can maintain a smooth profile in the training process. Such an idea is achieved by implicitly embedding the Wasser- stein distance between the residual-inducer distribution and the uniform distribu- tion into the loss, which is then minimized together with the residual. nearly uniform residual profile means that its variance is small for any normalized weight function such that the Monte Carlo approximation error of the loss functional is reduced significantly for a certain sample size. The adversarial adaptive sampling (AAS) approach proposed in this work is the first attempt to formulate two essen- tial components, minimizing the residual and seeking the optimal training set, into one minmax objective functional for the neural network approximation of PDEs.
INTRODUCTION
Partial differential equations (PDEs) are widely used to model physical phenomena. Typically, ob- taining analytical solutions PDES is intractable, and thus numerical methods (e.g.. finite element methods (Elman et al. 20144) have to be developed to approximate the solutions of PDEs. However, classical numerical methods can be computationally infeasible for tigh-dimensiinal PDEs due the curse of dimensionality or computationally expensive for parametric low-dimensional PDEs (Xiu & Karniadakis 2003) Ghosh al. 2022 Yin al.) 2023 Zhai et al.) 2022). To alleviate these difñ- culties, machine learning (ML) techniques, e.g., physics-informed neural networks (PINN) (Raissi jet al. 2019) and deep Ritz method (F Yu 2018), have been adapted to approximate PDEs as sur- rogate models and have received increasing attention (Han al., 2018 Zhu & Zabaras 2018 Zhu (ct al., 2019 Weinan 2021; Karniadakis et al. 2021). The basic idea of deep learning methods for approximating PDES is to encode the information of PDEs in neural networks through proper loss functional, which will be discretized by collocation points in the computational domain and sub-
sequently minimized to determine optimal model parameter (Raissi et al. 2019 E & Yu 2018 Sirignano & Spiliopoulos) 2018 Zhu et al. 2019).
"Co-first Author

---

Published as a conference paper at ICLR 2024
The collocation points crucial to effectively train neural networks for PDEs because they provide an approximation the loss functional. In the community of computer vision or natural language processing, it is well known that the performance of ML models is highly dependent on the quality of data (i.e., the training set). Similarly, if the selected collocation points fail to yield an accurate approximation of the loss functional, it is not surprising that the trained neural network will suffer large generalization error, especially when the solution has low regularity or the problem dimension large. As shown in (Tang et al., 2023; Wu et al. 2023), if the collocation points in the training set are refined according to proper error indicator, the accuracy can be dramatically improved. This is similar to classical adaptive methods such as the adaptive finite element method (Morin al. 2002] Mekchay & Nochetto 2005). In this work, we propose new framework, called adversarial adap- five sampling (AAS), that simultaneously optimizes the loss functional and the training set seek neural network approximation for PDEs through minmax formulation. More specifically, we min- imize the residual and meanwhile push the residual-indus To do this, introduce deep generative model into the AAS formulation, which not only provides random samples for the discretization of the loss functional, but also plays the role of the critic in WGAN (Arjovsky et al., 2017] Gulrajani al. 2017). In the maximization step, the deep generative model helps identify the difference in a Wasserstein distance between the residual-induce tion and a uniform distribution; in the minimization step, such a difference is minimized together with the residual. This way, variance reduction is achieved once the residual profile is smoothed and the loss functional can be better approximated by a fixed number of random samples, which yields a more effective optimal model parameter, i.c., more accurate neural network approximation of the PDE solution.
distribution to a uniform distribution.
distribu-
The main contributions of this paper are as follows.
We unify PINN and optimal transport into an adversarial adaptive sampling framework, which provides a new perspective on neural network methods for solving PDEs.
We develop a theoretical understanding of AAS and propose a simple but effective algo- rithm.
2 PINN AND ITS STATISTICAL ERRORS
The PDE problem considered here is: find u E F : + R where F is proper function space defined on computational domain Ω E R.P, such that
Cu(x) = s(x), bu(x)= g(x), Ve E an,
Va T Ω
(1)
where is partial differential operator (e.g., the Laplace operator A), b is boundary operator (e.g., the Dirichlet boundary), is the source function, and g represents the boundary conditions. In the framework of PINN (Raissi et al. 2019). the solution u of equation is approximated by
neural network up (x) (parameterized with 0). The parameters 0 is determined by minimizing the following loss functional
J (ue) = (,(u)) + +u)) with J,(uo) / |r(x; dxx and Jo(us) = Jmn 1b(x;; Rx)
/ /
where r(x:; 0) = (u)( - s(x), and b(x; 0) = (x) g(x) are the residuals that measure how well uo satisfies the partial differential equations and the boundary conditions, respectively, and 7 > 0 is a penalty parameter. To optimize this loss functional with respect to 0, we need to discretize the integral defined in equation numerically. Let Sp {x J,N, and San (23))a i= be two sets of uniformly distributed collocation points on Ω and an respectively. We then minimize the following empirical loss in practice
(i)
N
No
JN (ua)
( 0)
Σ
(x0 0),
N
No.
which can be regarded as the Monte Carlo (MC) approximation of ((ue) subject a statistical error of -1/22 with N being the sample size. Let uo; be the minimizer of the empirical loss JN(ue) (uo)
2

---

Published as a conference paper at ICLR 2024
and ugg be the minimizer of the original loss functional J(uo). We decompose the error into two parts as follows
E (1l10,- -mla) (Iues uo- l1a) + I|u0- u|la
where E denotes the expectation operator and the norm 11-112 corresponds to the function space & for u. One can see that the total error of neural network approximation for PDEs comes from two main aspects: the approximation error and the statistical error. The approximation error is dependent on the model capability of neural networks, while the statistical error originates from the collocation points. Uniformly distributed collocation points not effective for training neural networks if the solution has low regularity (Tang et al., 2022 2023] Wu et al.) 2023) since the effective sample size of the MC approximation of (ug) is significantly reduced by the large variance induced by the low regularity. For high-dimensional problems, information becomes more sparse or localized due to the curse of dimensionality, which shares some similarities with the low-dimensional problems of low regularity. Adaptive sampling is needed. In this work, we propose new framework to optimize both the approximation solution and the training set.
ADVERSARIAL ADAPTIVE SAMPLING
Adversarial adaptive sampling (AAS) includes two components be optimized. One is neural network ug for approximating the PDE solution, and another is probability density function (PDF) model Po (parameterized with a)) for sampling. Unlike the deep adaptive sampling method (DAS) presented in (Tang et al. 2023). in AAS, we simultaneously optimize the two models through an adversarial training procedure, which provides new perspective to understand the role of random samples for the neural network approximation of PDEs.
3.1
A MINMAX FORMULATION
The adversarial adaptive sampling approach can be formulated as the following minmax problem
min max I (uo. Pox)
r² (x; 0 Pa (z)dx + (uo)).
(4)
PVEV
.2
where V a function space that defines a proper constraint on Pa (2z). The choice of V will be spec- ified in sections 3.2 and 3.3 in terms of the theoretical understanding and numerical implementation of AAS.
The main difference between [(u,, Pa) and J (uu) in equation 2 is that the weight function for the integration of (x; 0) is relaxed to (axx) from a uniform one. First all, such a relaxation can also be applied to (.). In this work, we focus on the integration of r² for simplicity and assume that (.). is well approximated by prescribed set San. Indeed, some penalty-free techniques (Berg & Nyström, 2018 Sheng & Yang, 2021) can be employed remove Jz(.). Second, Pa Pa(x:) 0 is regarded as a PDF on 32, and an extra constraint on Pa is necessary. Otherwise, the maximization step will simply yield delta measure, i.e..
/ (2( ()p(x)dx
8(x - xo) = aig max p>0.Ja pda=1
where wo arg max (x; 0). Nevertheless, the region of large residuals is of particular im- portance for adaptive sampling. Third, the maximization in terms of Pa is important numerically rather than theoretically. Indeed, if the statistical error does not exist and the model ug includes the exact PDE solution, the minimum of x² is always reached at 0 long as Pa is positive Ω. To reduce the statistical error induced by the random samples from Pa. we expect a small variance Var(y22) in terms of Pa- If the variance of the Monte Carlo integration for p² is smaller than the variance of y in terms of the uniform distribution, the accuracy of the Monte Carlo approximation will be improved for a fixed sample size, which yields more accurate solution of PDEs (Tang et al. 2023). To obtain small Var(r?), the profile of the residual needs to be nearly uniform. So an effective training strategy should not only minimize the residual but also endeavor to maintain smooth profile of the residual, in other words, the two models ug and Pa need to work together. See Figure [ for an informal description of the approach.
3

---

Published as a conference paper at ICLR 2024
We will model Pa using bounded KRnet, which defines an in- vertible mapping f.(.) 1" + 1 with I = [-1, and yields normalizing flow model. In this work, simplicity. The bounded KRnet can be achieved by adding a lo- gistic transformation layer (Tang al. (202)) new coupling layer proposed in Zeng et al. (002). Let 2 (a) and pz be a prior PDF. We deline Pa.
Ps
consider $ _ for
Two neural network models are sinultaneously trained in the adversarial adaptive sampling framework. The residual is min- imized and finally becomes "uni- form", while the collocation points are updated and finally become nonuniform.
Figure 1:
Pa (æ) (Ja
Depending on priori knowledge the problem, the prior Pz (2) can be chosen uniform distribution more general models such as Gaussian mixture model.
3.2 UNDERSTANDING OF AAS
For simplicity and clarity, we remove Jo (ug) and consider
min I (uo.p) ()((()(())d PEV
(5)
We choose as
V = {p(x)]llp(lup S 0, p(x) S M)
where M is a positive number. We define bounded metric
dm (2, y) min{ M, d(x y)}, 0, E RD
where d(x.y) 1| yll is the Euclidean metric in RR, Without loss of generality, let Z be compact subset of R. with total Lebesgue measure 1, and μ and v two probability measures on Ω. The Wasserstein distance dw (n, v) between μ and v for the metric das (x, y)
I (x, y) dr(xx v)
dwa (),
inf xEIl(00)) axa
where II Ω) is the collection of all joint probability measures on Ω x Ω. The dual form (see e.g. (Villani 2003), Theorem 1.14 and Remark 1.15 on Page 34) of dwar is
dww (), sup { ((x) ((x))(((>))))) (x) 0 s ((x)
||das M, and ltip I } (6)
where I|ol is the Lipschitz norm of function ø. We now reformulate the maximization problem as
1² (x:;0)))(())) ()p(a))
sup pev n
1² (2; f)p(x)d / n² (2;; d) / p(x:)dx + / (x: f)dx f p(z)dx
sup peV
(2; /
/ p(x)d14 p(z))dpa
/ p(x)
sup PEV
+ sup NeV
/ (x (1:00) dx.
S(dwar (pr: Ha) ) M)
where H4, and Ma indicate the probability measures on Ω induced by (xx) and the uniform distribu- tion on respectively. It can be shown that the constant M exists if we modify the function space V as
= (l(x)) S p(x) 2 0, p(æ)dx 1},
where p(x) can then regarded as a PDF. It is seen that the upper bound includes both the loss of the standard PINN and the Wasserstein distance between the residual induced distribution H4 and the uniform distribution μ.. For any u, the existence function ů, which has the same total residual loss and uniform residual profile, is theoretically guaranteed (for detailed construction and
4

---

Published as a conference paper at ICLR 2024
these properties of it, please see the proof of Theoremmmin Appendix A.2) This ensures that simultaneously reduce the residual and the Wasserstein distance between the (renormalized) residual and the uniform distribution. Once the residual profile is smoothed, variance reduction is achieved such that the Monte Carlo approximation of J (uo. p) will be more accurate for a fixed sample size. This eventually reduces the statistical error of the approximate PDE solution.
We now summarize our main analytical results. Consider
J ((((())))(())
inf sup I (u, PEV
with the following assumption. Assumption A1. The operator in equation 7 is surjection from function space Ej (IRP to C (2), the class of C functions that are compactly supported on Ω.
In general, (RR can be any function space, such space of neural networks, smooth functions, or Sobolev spaces. And this assumption means for any smooth function f (22). equation (²("") = f admits some solution u". For example, if Laplacian A, the assumption means we can find solution for Au = for any f in (92). With this assumption, we can prove the following main theorem for the min-max problem equation (for the detailed proof, please see the Section A.2 in the supplementary material),
Theorem 1. Let be the Lebesgue measure on RD,, which represents the uniform probability distri- bution on S2. In addition, we assume Assumption [A]
holds. Then the optimal value of the min-max problem equation 7 O. Moreover, there is sequence u functions with r r(u,) o 0 for all n, such that is an optimization sequence of equation namely,
lim J(un: Pn.) 0
for some sequence of functions J satisfying the constraints in equation optimization sequence has the following two properties:
Meanwhile, this
1. The residual sequence (r(wa) of {un ) converges to O in T,? (d4.)
2. The renormalized squared residual distributions
dvn A (2(
So (un dx dp(x))
converge to the uniform distribution μ in the Wasserstein distance dwar.
3.3 IMPLEMENTATION OF AAS
In the previous section, have shown that P., (x) and (2) in equation 4 play similar role as the critic and generator in WGAN (Arjovsky et al. 2017 Guirajani et al., 2017). The generator of WGAN minimizes the Wasserstein distance between two distributions; PINN minimizes the resid- ual; AAS achieves a tradcoff between the minimization of the residual and the minimization of the Wasserstein distance between the residual-induced distribution and the uniform distribution. From the implementation point of view, particular difficulty is the constraint || S induced by the function space V. In this work, we propose weaker constraint that be easily implemented. We consider
J(uo.Pa) (1; ( - 3 / (VaPa()))
min 0
max Per 20, Jq Pa (x)d=11
where we use a H1 regularization term to replace explicit control on the Lipschitz condition. The constraints PDF naturally satisfied because Pa normalizing flow model. Pa (x) > 0 as long as the prior is positive since (-) is an invertible mapping. It can be shown that the maximizer for fixed Ug is uniquely determined by the following elliptic equation
p3 + 2
m Ja (x; 0 dx 0, Ω
Op"
0, a..
On
5

---

Published as a conference paper at ICLR 2024
In the deep learning framework, the neural networks are in general (particularly when solving PDEs) differentiable. So the regularity constraint !|p*| Σ M equivalent to VVp"ll I on compact set f2. Thus, Such a choice is demonstrated to be empirically sufficient since we focus on PDE approximation instead of PDF approximation.
adjust the penalty parameter 3 to implicitly control this regularity.
To update at the minimization step, we approximate the first term of [ug Pu) in equation using Monte Carlo methods:
| (uo()))pa(z)]e 2
uo (2())
(10)
where (6) can be generated from the probability density Po efficiently thanks the invertible mapping Sa(:) .. To update a the maximization step, we approximate [uua.Po Pa) by importance sampling:
m 22
(8) us (2(
(8) Pa Pa( (2
m V xaa (3 2))
J(uo Pa)
-3.
(11)
m
Par
(x
m Pa (x (
where Por is PDF model with known parameters a' and each x' is a sample drawn from Pa Using equation 10 and equation [1 we compute the gradient with respect to 0 and a, and the parameters can be updated by gradient-baset optimization methods (e.g., Adam (Kingma 2017)). The training procedure is similar to GAN (Goodfellow et al. 2014) and can be summarized in Algorithm [ used for importance sampling when computing I Puo,Po).
Ba
where we let Par = Pau in equation i... the PDF model from the last step is
Algorithm AAS for PDEs
Input: Initial Pox and ug. maximal iteration M, batch size m, initial training set Sa,0
(2 1000/:0 ii=1
and San,o = { (8)
a0.
1: for = 0...., MM do
2: for j steps do
3: Sample m samples from SA. and sample m samples from Syn.k
4: Update ue by descending the stochastic gradient of J [ ox) (see equation 10). end for
5:
6: for jsseps do
7: Sample m samples from Sp
8: Update Pa by ascending the stochastic gradient of ox) (see equation
9: end for
10: Generate Sp,t+1 C 52 through Par.
11: end for
Output: ug
4 RELATED WORK
There is lot of related work, and we summarize the most related lines of this work.
Adaptive sampling. Adaptive sampling methods have been receiving increasing attention in solving PDEs with deep learning methods. The basic idea of such methods is to define proper error indicator (Wu et al. 2023 Yu et al. 2022) for refining collocation points in the training set, which sampling approaches (Gao & Wang 2023) (e.g., Markov Chain Monte Carlo) or deep generative models (Tang et al. 2023) are often invoked. To this end, an additional deep generative model (c (c.g.. normalizing flow models), classical PDF model (e.g., Gaussian mixture models (Gao al.) 2022
Jiao et al., 2023)) for sampling is usually required, which is similar to this work. However, there some crucial differences between existing approaches and the proposed adversarial adaptive sampling (AAS) framework. First, AAS, the evolution the residual -induual-nnuced induced distribution has clear path. That is, this residual-induced distribution pushed to uniform distribution during

---

Published as a conference paper at ICLR 2024
training. Because minimizing the Wasserstein distance between the and the uniform distribution is naturally embedded in the loss functional in the proposed adversarial sampling framework. Second, unlike the existing methods, our AAS method admits an adversarial training style like in WGAN, which is the first time to minimize the residual and seek the optimal training set simultaneously for PINN.
distribution
Adversarial training. In Zang et al. 2020), the authors proposed a weak formulation with primal and adversarial networks, where the PDE problem is converted an operator norm minimization problem derived from the weak formulation. Although the adversarial training procedure is em- ployed in (Zang et al., 2020). it does not involve the training set but the function space. Introducing one or more discriminator networks to construct adversarial training is studied in (Zeng et al.) 2022). where the discriminator is used for the reward that PINN predicts mistakes. However, this approach does not optimize the training set but implicitly assigns higher weights for samples with large point- wise residuals through adversarial training.
NUMERICAL RESULTS
We some benchmark test problems presented in (Tang et al. 2023 method. All models are set to be the same as those in DAS-PINNs (Tang et al., (023) and trained by the Adam method (Kingma be the same as those in PINNS Tang et al. 2023). For comparison, we also implement the DAS algorithm (Tang et al. 2023) and the RAR algorithm (Lu al., 2021] Yu et al., (022) as the baseline models. The training of neural networks is performed on a Geforce RTX 3090 GPU with TensorFlow 2.0. The codes of all examples will be released on GitHub once the paper is accepted.
demonstrate the proposed
Ba 2017). The hyperparameters of neural networks are set to
5.1 ONE-PEAK PROBLEM
We start with the following equation which is a benchmark test problem for adaptive finite element methods (Mitchell 2013 Morin et al. 2002):
-Au(x) = s(x) in Ω
(12)
u(æ) = 9(x) on 0N,
where x = [1i, v22] and the computation domain is Ω = [J1, The following reference solution given by
u(z1, x2) = exp (-1000[(a1- - 0.5) + (x2 - -0.5)
which has a peak at (0.5, 0.5) and decreases rapidly away from (0.5, 0.5). The reference solution is imposed on the boundary. The source term s(x) is derived by the exact solution and is listed in the supplementary A.5 A uniform meshgrid with size 256 x 256 in [- 1 is generated and the error is defined to be the mean square error on these grid points. From Figure [a). it can be seen that
AAS method can give an accurate approximation for this peak test problem. Note that the uniform sampling strategy is not suitable for this test problem as studied in (Tang al. 2023). The training behaviour for different regularization parameters (i.e., 8) is shown in Figure a) and Figure 2[b))
can be seen that the error behavior is similar for 3 5, 10, 20. Figure Figure 2 c) shows the evolution of the residual variance and the training set during training for 5, where the variance
decreases as the training step increases and the training set finally concentrates on (0.5, 0.5) with a heavy tail. The comparison of different adaptive sampling methods is presented in Table [ which also included the results of the following test problems.
5.2 TWO-PEAK PROBLEM
We next consider the following equation
Ve(x)V((x)] + v2u(x) = s(x) in Q " [ gxx) on an,
(13)
where æ = [x,, Fo T, t(x) = 27 + 2 and the computation domain is Ω = [31, Following Tang et al. 2023). the exact solution equation 13 is set to be
x221)22) -005 + - -0.5
(0.0 (o.teee00...(++++0..53"33 +(22+0.5))
7

---

Published as a conference paper at ICLR 2024
227
3023
18""
38"
10"
16'
0.
0.
133
23
25
05
10
15
2.0
25
(a)
(b)
(c)
Figure 2: The results for the peak test problem. (a) The error behaviour. (b) The variance behavior. (c) The evolution of the training set.
which has two peaks at the points (0.5, 0.5) and -0..5. Here, the Dirichlet boundary condi- tion on 0 given by the exact solution. From Figure and Vigure3ib) it can seen that our AAS method can give accurate approximation for this two-peak test problem. The error behavior for different regularization parameters (i.e., 3) shown Figure 3(. Figure shows the evolution of the residual variance and the training set during training for decreases as the training step increases and the training set finally concentrates on (-0.5, -0.5) and (0.5, 0.5) with heavy tail.
where the residual variance
130
5.48
10:1
10m
036
ASI 6.0
130
0.0
85
1.2
155
20
Rendin
(8)
(b)
(c)
Figure 3: The results for the two-peak test problem. (a) The exact solution. (b) AAS approximation. (c) The error behavior.
10a
6=5
10
10P
Variance
10'
167
18
10-1
265
Figure 4: The evolution of the residual variance and the training set for the two-peak test problem. Left: The variance behavior. Right: The evolution of the training set.
8

---

Published as a conference paper at ICLR 2024
5.3 HIGH-DIMENSIONAL NONLINEAR EQUATION
In this part, we consider the following len-dimensional nonlinear partial differential equation Au(x) + u(x) - u² (x) s(x), 2: in f2 = [-1,1]
(14)
u(x) g x on 0S2.
The exact solution is u(x)) =0] and the Dirichlet boundary condition A is imposed by the exact solution. The source term s(x) is derived by the exact solution and is listed in the supple- mentary A..5 The error is defined be the same in (Tang et al. 2023). Figure the results of the ten-dimensional nonlinear test problem. Specifically, Figure 5(a) a) shows the error behavior during training for different regularization parameters, and Figure (()) shows the evolution of the residual variance. Figure5jo shows the samples during the adversarial training process, where we select the components æ1 and 22 for visualization. We have also checked the other components, and the results are similar. It is seen that the training set finally becomes nonuniform to get small residual variance. The results of different adaptive sampling strategies for the three test problems summarized in Table
10.
16"
2011.
10
307
(a)
(b)
Figure 5: The results of the ten on-dimmensionll nonlinear test problem. (a) The error behavior. (b) The variance behaviour. (c) The evolution of the training set, Fi 22 plane (3 10).
Table I: Error comparison of adaptive sampling methods
Test problem
PDE equation 12
PDE equation 13
PDE equation 14
Method
PINN
9.74e-04
3.22e-02
1.01
RAR Lu et al. 2021
9.83e-01
DAS-G
lang et all 2023
3.75e-04
1.51e-03
9.55e-03
DAS-R
Tang el al. 2023
1.93e-04
6.21e-03
1.26e-02
AAS (this work)
2.97e-05
1.09e-04
1.31e-03
6 CONCLUSIONS
We developed a novel adversarial adaptive sampling (AAS) approach that unifies PINN and optimal transport for neural network approximation of PDEs. With AAS, the evolution of the training be investigated in terms of the optimal transport theory, and numerical results have demonstrated the importance of random samples for training PINN more effectively.
Acknowledgments: K. Tang has been supported by the China Postdoctoral Science Founda- tion grant 2022M71 1730. Zhai is supported by the start-up fund of ShanghaïTech University (2022F0303-000-11) X. Wan has been supported by NSF grant DMS-1913163. C. Yang has been supported by NSFC grant 12131002.
REFERENCES
Robert A Adams and J John Fournier. Sobolev Spaces, 2nd Edition. Elsevier, Amsterdam, 2003.

---

Published as a conference paper at ICLR 2024
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. PMLR, 2017.
Jens Berg and Kaj Nyström. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing, 317:28-41, 2018.
Weinan E and Bing Yu. The deep Ritz method: A deep learning-based numerical algorithm for
solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
Howard C Elman, David J Silvester, and Andrew Wathen. Finite elements fast iterative solvers: With applications in incompressible fuuid dynamics. Oxford University Press, USA, 2014.
Lawrence C Evans. Partial Differential Equations. American Mathematical Soc., 2010.
Wenhan Gao and Chunmei Wang. Active learning based sampling for nigh-dimensional nonlinear
partial differential equations. Journal of Computational Physics, 475:1 11848, 2023.
Zhiwei Gao, Liang Yan, and Tao Zhou. Failure-informed adaptive sampling for PINNs. arXiv preprint #rXiv22200000273 2022.
Sayan Ghosh, Govinda Anantha Padmanabha, Cheng Peng, Valeria Andreoli, Steven Atkinson, Piyush Pandita, Thomas Vandeputte, Nicholas Zabaras, and Liping Wang. Inverse aerodynamic design of gas turbine blades using probabilistic machine learning. Journal of Mechanical Design, 144(2), 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of Wasserstein GANs. Advances in Neural Information Processing Systems, 30, 2017.
Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505-85100 2018.
Yuling Jiao, Di Li, Xiliang Lu, Jerry Zhijian Yang, and Cheng Yuan. GAS: A Gaussian mixture distribution based adaptive sampling method for PINNs. arXiv preprint 2.03::2333.1844) 2023.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang- Physics-informet machine learning. Nature Reviews Physics, 3(6) 440, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2017.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. DeepXDE: A deep learning library for solving differential equations. SIAM Review, 63(1):208-22) 228,,22281000 2021.
Khamron Mekchay and Ricardo Nochetto. Convergence of adaptive finite element methods for general second order linear elliptic PDEs. SIAM Journal on Numerical Analysis, 43(5): 1827, 2005.
William F Mitchell. A collection of 2D elliptic problems for testing adaptive grid refinement algo- rithms. Applied Mathematics and Computation, 220:350-364, 2013.
Pedro Morin, Ricardo H Nochetto, and Kunibert G Siebert. Convergence of adaptive finite element methods. SIAM Review, 44(4):631-658, 2002.
Maziar Raissi, Paris Perdikaris, and George E Kariaadakis Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
10

---

Published as a conference paper at ICLR 2024
Hailong Sheng and Chao Yang. PFNN: A penalty-free neural network method for solving class of second-order boundary-valin problems on complex geometries. Journal of Computational Physics, 428: 10085, 2021.
Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1364 2018.
Kejun Tang, Xiaoliang Wan, and Qifeng Liao. Adaptive deep density approximation for Fokker- Planck equations. Journal of Computational Physics, 457:1 10000, 2022.
Kejun Tang, Xiaoliang Wan, and Chao Yang. DAS-PINNs: deep adaptive sampling method for solving sigh-dimensional partial differential equations. Journal of Computational Physics, 476: 111868, 2023.
Michael E Taylor. Partial Differential Equations I: Basic Theory, 2nd Edition. Springer, 2011.
Cédric Villani. Topics in Optimal Transportation. Number 58 in Graduate Studies in Mathematics. American Mathematical Society, 2003.
Weinan. The dawning of a new era in applied mathematics. Notices of the American Mathematical Sociery, (8(4):565-571 2021.
Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. A comprehensive study of non- adaptive and residual-based adaptive sampling for physics-ifformed neural networks. Computer Methods in Applied Mechanics and Engineering, 403: 15671, 2023.
Dongbin Xiu and George Em Karniadakis. Modeling uncertainty in flow simulations via generalized polynomial chaos. Journal of Computational Physics, 187(1):137-167, 2003.
Pengfei Yin, Guangqiang Xiao, Kejun Tang, and Chao Yang. AONN: An adjoint-oriented neural network method for all-at-once solutions parametric optimal control problems. arXiv preprint arXiv:2302.0828 02076, 2023.
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems. Computer Methods in Applied Mechanics and Engineering. 393: 14823, 2022.
Yaohua Zang, Gang Bao, Xiaojing Ye, and Haomin Zhou. Weak adversarial networks for high- dimensional partial differential equations. Journal of Computational Physics, 41 411:109409, 2020.
Li Zeng, Xiaoliang Wan, and Tao Zhou. Bounded KRnet and its applications to density estimation and approximation. arXiv:2305.0900 2023.
Qi Zeng, Spencer H Bryngelson, and Florian Tobias Schaefer. Competitive physics informed net-
works. In ICLR 2022 Workshop on Gamification and Multiagent Solutions, 2022.
Jiayu Zhai, Matthew Dobson, and Yao Li. A deep learning method for solving Pokker-Planch equations. In Mathematical and Scientific Machine Learning, pp. 568-597. PMLR, 2022.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional recoder-decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415- 447, 2018.
Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high- high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56 81, 2019.
11

---

Published as a conference paper at ICLR 2024
A APPENDIX
We add the proof of Theorem and additional numerical experiments here.
A.
PRELIMINARIES FROM OPTIMAL. TRANSPORT THEORY
Definition Suppose X is a metric space equipped with the metric d(x, y). and μ and are two probability measures on X. The Wasserstein distance (as known as the Kantorovich- Rubinsseinn metric) d v) between to probability measures μ and for the metric function d(xr, y) is defined to be
inf / =em(xxx) xxxx d(x, y) du (x, y):
dwa(k,, =
where I1(X x X) is the collection of all probability measure on X x X such that
π (AA X) = p(A), +(X B) rBB)
for all measurable sets A, X.
For the analysis of the adaptive algorithm this work, we consider the metric das (2r, y) induced by the Euclidean metric d(x y) = || 2
dm(a: y) min M, d(x, y)}, v, X.
Then the metric (æ, y) is always bounded by M (reachable, namely ||dM llxo M). We denote the Wasserstein distance for d (z, y) by (.
According to the optimal transport theory, the Wasserstein distance can be described by its dual form (see c.g. (Villani 2003), Theorem 1.14 and Remark 1.15 on Page 34).
Theorem
theorem). Let be Polish space and let be lower semi- continuous metric on X. Let || [lipp denote the Lipschitz norm of a function defined as
((x)= Kly))
| = sup
aty d(x, y)
Then
dw (p, suu
{ (())
d)p=))(())) w)(z)) os <(x) S |oo M, and | S }
In this work, restrict ourselves on a compact domain X Ω RDD of learning, and without loss of generality, we assume the Lebesgue measure of Q is 1.
A.2 THE FIRST CONVERGENCE THEOREM AND ITS PROOF
Theorem 4, Let ya be the Lebesgue measure on X, which represents the uniform probability distri- bution on Q. In addition, we assume Assumption Al holds.
Then the optimal value of the min-max problem equation 5 is 0. Moreover, there sequence 1.2 1 of functions with r(u)) * 0 for all such that it is optimization sequence of problem equation namely,
lim (n): 10000
(15)
for some sequence of functions following two properties:
100
V. Meanwhile, this optimization sequence has the
The residual sequence (r(wn))oe of (un) converges to T?2 Td4)).
2. The renormalized squared residual distributions
(2(2 dp
dyn Sq (,((((()x)) do
(16)
converge the uniform distribution p in the Wasserstein distance dwM.
12

---

Published as a conference paper at ICLR 2024
Proof. Consider minimizing sequence Un, 1.2.... of inf / (2((x)) dx,
(17)
where without loss of generality, assume that Jo r² (un dx s n
Now
sup P(
S1
0SPSM
1// 1 ("((((z)))(al) dx
/ y² (((((z)) dx I pla)) dedd I (2(u((z)) de J p(xx) dx
sup Ipllup OSPSM
Ja , (u)( de
I p(z)) dun (2x) p(x)) dx)
/ p(xx) dx)
sup
sup
Upllins11 OSPSM M
pliu S2 OSPSM
(dwar (yrs 10) + M) I r² (u)( dæ.
(18)
By the assumption of the theorem, for each n, distance dw M (Dn,/4) (4) S A. where in is the measure defined as in equation 16 by replacing u, (æ) with ū,, (a:) In fact, for each n, we can find, by partition of unity, sequence of functions in C converging to dg in the Sobolev norm of l (See for example (Evans) 2010)). ) So we can find
find function (x) that the Wasserstein
function Wn, in C2 (S2), such that I| (x) In(x))l S on f2 Since r is surjection, there is some n (z) so that
y (ūm) = wn / y² (2( dæ.
and
1 12 (ūn) dx = (m((x) do Jo 12 (xx) dæ
S (1+ / (1(()) dx) Ss , (un((z))
2 (2)))(z)) dx.
This means (ūn is also minimizing sequence of equation 17 and it yields
p(x) di (x) - / p(x) dx
dwa (Dn,H)= = sup ||
OSPSM
(
sup |l
p(x)
J r2 2- (ūn, (x)) dx La(x) dæ
0SP5M
p(x)) [wn - [=) dx
sup ||
USPSM
M
So we get from equation 18 that
J(ūn, p) S lim 4M / (²(un) do O,
< lim
sup
/
200
|lIL 0SP5.
n
which means that {ū,, 100 is also a minimizing sequence of equation 5 that is,
lim I (ūn, Pa) = 0.,
15
R=000
for sequence of functions (02 C V. Meanwhile, have the following properties of ū.
13

---

Published as a conference paper at ICLR 2024
1. The residual sequence (r(ūn)) converges to 0 in L2 (dp)), since
J (ūn) dæ s (u))) dx _ 0, as n
2
2. The renormalized squared residual distributions
(2b du Jq r2 (Un do
di
converges to the uniform distribution j the Wasserstein distance dwy
A.3 REPLACEMENT OF THE BOUNDEDNESS CONDITION IN THEOREM
For the boundedness constraint for "test function" p in prove that it can be removed in circumstance. And with the following lemma and its following remark, and Theorem 4 we can obtain main Theorem [
which is stated again with its assumption in the following.
Assumption. The operator r in equation 7 is surjection from function space E (R.) C (9)) the class of C functions that are compactly supported on Ω
Theorem. Let μ be the Lebesgue measure on RD,, which represents the uniform probability distri- bution on St In addition, assume Assumption A7 holds. Then the optimal value of the min-max problem equation 7 O. Moreover, there is a sequence [un functions with r(u,) F 0 0.ror all
n, such that is an optimization sequence of equation 7 namely,
lim I [(Un Pn) 71-000
for some sequence of functions {Pr] satisfying the constraints in equation optimization sequence has the following two properties:
Meanwhile, this
1. The residual sequence {r(un)) of ) converges to 0 in L (ddd)..
2. The renormalized squared residual distributions
(2(un) dp(æ))
dun Jn (u)) dx
converge to the uniform distribution p. in the Wasserstein distance dwa.
Although the residue 7 renormalized to probability distribution for the analysis the algorithm, itself is not probability distribution, and not treated algorithm, the "test function") p is seen as sampling distribution density and the residue ,² is just the PDE operator (or any kind of objective function whose minimum is 0). In the implementation, establish as generative model, that is, an invertible transform between an unknown distribution (an adversarial distribution the residual distribution if we think the algorithm GANs) and an reasy-to-sample distribution such as normal or uniform distribution. So we assume be the density function of probability distribution. Under this assumption, we have the following result.
Actually, in the implementation of our
similarity to
Lemma 5. Let Q be a compact subset of R.D. If positive function f Ω + R is K K-Lipschitz continuous, and f is the density function of probability distribution, namely, Saa f dx then there is some constant M (Ω2, K), that M. In other words,
S M, VfES= {f 01llf1le I K, and [ f f = 1}.
Proof. For any x, Ω have
5 f()) f(x f()) f(0) < K| al + f(v) +
14

---

Published as a conference paper at ICLR 2024
where ((2) is the diameter of S2. Taking integral with respect to y over Ω on both sides, have
< (()) (D(())((()) 1, where (9) the Lebesgue measure (volume) of Ω2 that is,
0 f(x) - (D(())) ((2)
So we have
M M(Ω, K) (D
p(
e The converse of this lemma is also true the sense that if f is bounded by some constant M, then the integral Jg f dæ S p(Q), and f can be renormalized into probability density function with constant M (S)). And similar to boundedness for the gradient (or Lipschitz constant) discussed in section 3.3 constant renormalizer will not affect the training procedure.
A.4 DEVIATION OF EQUATION AND ITS SOLUTION
For given r(æ; 0). consider the following minimization problem:
min ((pa) =3 S VaPallds [ (2)))))())) Pa >0
(L, Pa(a)))))) -1):
where the positivity of Pa is guaranteed by the KRnet and À the Lagrange multiplier for the mass conservation of PDF. Assuming that dra on the boundary ØN, where n a unit normal vector 7m on ö2 pointing outward. We have the first-order variation of L(pa) for perturbation function 5p(x)
/
(
8L 23
J Vpa Vópdx -
8pdx + X 8pdæ
-23 (ʃ pp ndr
5p pradl)) 11 ,² Poppd + X I 6p(z))da
59
2,3
8p Pa da jn
(
Ja
r2 8pppa + / 6p(a))da
(28V*pe Pa + + A)8pdx,
5
where applied integration by parts and the homogeneous Neuman boundary conditions. The &L yields 6p
optimality condition
A 0, x E Ω 2 pz) + +7;;;)
(19)
8pa 0, a E an.
ön
From the compatibility condition for Neumann problems, have
/ (2; A)da 0,
(20)
which yields that
/ /x 0)xx.
|22| 0
Assume that a bounded domain with smooth boundary. can be shown that if e H²()) and 00 e Ok+ with & e N, the solution of equation 9 satisfies (Taylor 2011)
||Pa 2 < fl)f)l
where f(x: (A- /223)) and C is general constant that does not depend on According to the Sobolev Imbedding Theorem Adams & John Fournier 2003).
Wh. 0.1 (Ω),
15

---

Published as a conference paper at ICLR 2024
when D = k- 12-1. 1. Thus up to set of measure zero, we have
PPallco.mm S CillPa/llwaao) S CallPallAM(S))
where C1 and C2 are general constants independent of Pa- So Pa is Lipschitz continuous when the boundary and r(æ: are sufficiently smooth. However, this also means that the H1 regularization used in equation 8 inducess a weaker constraint than the Lipschitz condition in Lemma
A.5 SUPPLEMENTARY EXPERIMENTS
About the setting of ((x) and ((r). The source term s(x) is derived by the exact solution, i.e., we can set the source function by plugging the exact solution into the equation to get s(rr). We set g(x) = u(x) since the Dirichlet boundary condition is imposed on AΩ.
Parametric Burgers' Equation. We also test the proposed AAS method using parametric PDEs that are commonly used in the design of engineering systems and uncertainty quantification. Specif- ically, we consider the following parametric Burgers' equation, which a benchmark problem stud- ied in DeepXDE.
du
du
du
vll du )2 +
Ou |
+ u
+ 1
dx
dy
Or
dy
du
dv
vl( Or
dv )2)
+ v
)2 + 0y
Ot
dx
dy
dr
r, E [0, 1). and E [0,1]
where u and v are the velocities along x and directions respectively, and v e (0, 1] is a parameter that represents the kinematic viscosity fluid. Here, the Dirichlet boundary conditions are imposed on all boundaries. The exact solution is obtained as follows.
u(I, (,) 4
4[1 + eexp((-xx 4y ()/(3)]))]
3
1
o(z, v1
4[1 + exp(eeppppp + 4y 1)/(32n))
The problem setup space is æ [t, 2, y, v). i.e. D 4. When v is small, solving this problem is quite challenging. We use the proposed AAS method to train neural network (x to approx- imate the solution over the entire space results, which demonstrate that the proposed AAS method is able to accurately solve this parametric Burgers' equation. We can train the models using the strategy as discussed in Remark 2, i.e., we gradually add the data points to the current training set. AAS with fixed , = 5 means that we use similar training strategy as DAS-G presented in (Tang et al. 2023) with a fixed 3, while AAS with decay 3 = 5 means that B has decay scheme at every 100 stages with decay rate 0.9. Adding the data points gradually to the current set of random samples is more stable than that of replacing all data points.
[t. æ, y. E 1] Figure 6 shows the numerical
10-2
AAS decayy 5
with dacay AAS with replace stroogg
10-4
AAS With roplace 22 200RGgy
10:5
Emor
1002
30
10-3
10-14_
0.0
3.5
1.0
1.5
2.0
255
3.0
35
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Iteration
365
Iberation
Figure 6: The results of the parametric Burgers' equation. Left: The error behavior. Right: The evolution of the variance.
16


Reviews:
[{"review_id": "Rek3puFWf2", "reviewer": "Reviewer_YsSo", "summary": "The paper proposes a new objective to sample collocation points for PINNs adaptively. Specifically, the collocation points for training the PINN are sampled from a normalizing flow with soft Lipschitz constraint, enforced by Sobolev regularization. The PINNs and normalizing flows are optimized in an alternating fashion, where the normalizing flow is trained to maximize the expected PINN residual (w.r.t. to the distribution given by the pushforward of the normalizing flow). This formulation is then connected to the dual form of an optimal transport problem. In this context, it is shown that there exists an optimal solution to the proposed min-max problem where the (normalized) squared residual converges to a uniform distribution in the Wasserstein distance. The approach's effectiveness is further demonstrated on two toy problems and one high-dimensional nonlinear PDE and compared to three related methods.", "questions": "1) Why is RAR not tested for the PDEs in (12) and (13)?\n2) Why can we not use a single backward pass on the loss in (11) and update both $\\theta$ and $\\alpha$ (using decent and ascent, respectively).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1) Tackling adaptive sampling for PINNs using optimal transport seems to be a promising direction.\n2) The proposed objective seems to be novel and can bring numerical benefits on the considered examples.", "weaknesses": "The theoretical as well as numerical contributions need to be significantly improved:\n\n1) Further challenging problems (as in Subsection 5.3), as well as baselines (as enumerated in the section on related works), are needed to judge the performance of the proposed algorithm. \n2) \"[...] which is the first time to minimize the residual and seek the optimal training set simultaneously for PINN.\": Analogous to DAS-PINNs, it seems that the final algorithm is still alternating between optimizing the two networks, see also Question 2) below. In general, it should be made more explicit what the novelty of the present work is and how it theoretically compares to related work.\n3) For the numerical results, training times and standard deviations (w.r.t. different seeds) are missing. Especially given that the adversarial training slows down training.\n4) The loss for learning the normalizing flow seems to use the REINFORCE trick, which, however, is known to suffer from high variance.\n5) Since the residual continuously changes over the course of optimization, it should be better motivated what the advantage of learning a generative modeling is (as compared to just using a method to sample, e.g., according to squared residual)?\n6) It seems that the precise connection to optimal transport remains a bit unclear:\n\t* \"So the minimization step will reduce not only the residual but also the Wasserstein distance between $\\mu_r$ and the uniform distribution\". Since there is only an upper bound shown on page 6, it is unclear why the minimization step is guaranteed to decrease the Wasserstein distance.\n\t* \"The evolution of the residual-induced distribution has a clear path\": According to the theorem, there only *exists* such a path, and it is not clear whether this path is taken by GD. It would be good to have at least plots of the residual evolution for all experiments.\n\n**Minor issues:** \n1) use `\\citep`.\n2) In Eq. (4), min-max should be written on both sides of the equation.\n3) What is a 'proper' constraint?\n4) The last paragraph in Section 3 seems a bit convoluted. The choice for minimal variance could just be motivated by the optimal choice for importance sampling. \n5) It is unclear why we would consider $I^D$ instead of $\\Omega$ when the KRnet is first introduced.\n6) The text for the figures of the training set is barely readable.\n7) typo: resultd", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWYbyiVild", "reviewer": "Reviewer_ZtMG", "summary": "In this paper the authors introduce an adaptive sampling technique that for sampling the input points (also referred to as collocation points) for training a PINN architecture on a domain. The main motivation behind it is that for highly irregular PDEs, uniformly sampling collocation points will not work well and result in high residual error for PINNs. \n\nThe idea is similar to the adaptive sampling techniques that have been introduced in previous work by Tang et al (2023) and Gao et al (2023), however instead of directly minimizing a pushforward or adaptive sampling, the authors in this paper enforce a wasserstein loss to ensure that the residual error is uniformly distributed across the grid. \n\nThat is, the training is done such that the error induced by PINNs is uniformly distributed across the grid, and hence the points are sampled accordingly. The authors operationalize this using a WGAN setup, wherein they train a pushforward map to learn p(x), such the wasserstein-1 loss between the the density of the residual and uniform distribution. This is similar in vein of the adaptive sampling techniques introduced in Gao et al (2023) but the weights are instead learned using a GAN type architecture.", "questions": "I have asked most of the questions in the previous sections.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The idea to use a GAN type loss with to ensure that the density of the residual is uniform across the domain is very interesting. \n\nIn their experiments, the authors are able to do well on very sparse data, i.e., PDEs that have two-peaks, something that PINNs sometimes don’t do well in, and are able to get better results than the previous baselines by Tang et al (2023), thus showing the benefits of using the wasserstein formulation for learning $p_\\alpha$.", "weaknesses": "In general the work is very similar to that by Tang et al (2023), there the authors are using a network defined using a normalizing flows type of an architecture whereas here the authors are using a GAN instead. \n\nThe paper is very hard to read with sometimes the notation and the terms used by the authors for different quantities is not clear. For example\n\n- The authors mention that this methodology achieves variance reduction, what is the proof for that? is it similar in flavor of Tang, et al (2023)\n- In Assumption 1, the authors refer to $r$ as an operator. Is that an operator, or the residual between the loss. If the authors mean that $r$ is an operator, then from which function space to what other function space? (I presume it is U x F → U).\n- In theorem 1, $\\nu$ is used to define the density of the residuals, whereas in the derivation under equation 6, it is defined by $\\mu_r$. I think that they are the same quantity, however if not, what is $\\nu$ and what is $\\mu_r$?\n- It is unclear as to what is being approximated by a KRNet?, since the authors first use $p_\\alpha$ to define the density that they are trying to train, however, there is no mention of it after section 3.2\n- Also, KRNet is not cited in the last paragraph of page 3.\n- In equation 9, what is p*?\n\nFew other questions that I think would help with the understanding of the usability of the techniques are: \n\n- What are the implementation details (in terms of number of parameters) etc of the networks approximating the solution $u_\\theta$ and approximating the density network, i.,e $p_\\alpha$.\n- Since the authors are doing a min-max optimization, that is usually hard to train and get right, esp for for PDEs that may have some advective terms.\n\nWhile the methodology provided by the authors seems to do better than the baselines, given the presentation and the lack of clarity of the precise steps, I think the paper would benefit from a revision", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4jHpgpHaUs", "reviewer": "Reviewer_3BKL", "summary": "A uniform approximation error is crucial in representing functions in a given vector base.  The same applies to compound function representations like neural networks. It also crucial where the errors are measured. The Authors combine both aspects, a  Wasserstein distance between a uniform and error distributions,  and an adversarial adaptive sampling of points for the PINN collocation.\n\nThe loss is divided to an analytical part that reflects the difference of the neural network from the solution and a statistical part that comes from using a finite sample.  By tuning the residual to become uniform, the density of samples becomes optimally non-uniform.\n\nThe method is used for three different PDEs with encouraging results compared to other adaptive learning techniques.", "questions": "Any methods to solve the above problem? A possible remedy could introduce quasi random point distributions where the points do follow the error distribution, but are smoothly enough distributed in oder to avoid computation of the the error on points that are close to each other already without an expectation of significant change in the error distribution.\n\nCould the problem above be solved with couple of iteration steps of repulsive force in the point cloud?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "An excellent way to bring confidence to the inference with good proofs of convergence for the adaptive learning distributions.\n\nPresentation is clear and not hard to follow.", "weaknesses": "Looking at the point clouds one visually can guess that the Delaunay triangulation link length distribution of  the point cloud contains very short distances i.e the point cloud is not locally smooth. Usually building meshes for FEM one prefers same size of elements locally and smaller elements in those areas do not provide more accuracy. \n\nI suppose this is the same for collocation points in PINNs. Currently the point clouds do not look like node distributions for FEM calculus. This is increasingly so, when creating points in the higher dimensions. The amount of wasted calculations increases and may drive this to a curse of high dimensionality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hLriYdiTgt", "reviewer": "Reviewer_CyTK", "summary": "In this article, authors raise a significant problem that linked with numerical difficulty is that random samples in the training dataset introduce statistical errors into the functional loss which may become the dominant error in the approximation, and therefore overshadow the modeling capability of the neural network. A new approach was proposed to optimize both the approximate solution and random samples in the training set by using a min max formulation. This approach is called adversarial adaptive sampling (AAS). The main idea of AAS is minimizing the residuals and meanwhile push the residual-induced distribution to a uniform one.  AAS can be divided into two parts. In the maximization part, the deep generative model helps define the difference between the residual-induced distribution and a uniform one using Wasserstein distance. In the minimization one, this difference is minimized together with the residuals.   Also, they used some benchmark test problems of comparison AAS algorithm with another state-of-the-art algorithms as DAS and RAR.", "questions": "What other types of PDEs can the proposed algorithm be applied to? \nWhat are restrictions of the proposed method?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The motivation is clear, and the method is novel and interesting. Authors clearly described the theory of proposed method and the algorithm.", "weaknesses": "A numerical result is given for one type of PDEs. It would be interesting to see how AAS algorithm performs for other types of PDEs. There is no discussion of further development or limitations of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)