# Meshfree Variational-Physics-Informed Neural Networks (MF-VPINN): An Adaptive Training Strategy
## 3 Numerical Results
### Implementation Details  
The computer code used to perform the experiment is implemented in Python using the Python package Tensorflow [4] to generate the neural network architecture and train the MF-VPINN. Using the notation introduced in Section 2.1, the used neural network consists of \(L=5\) layers with \(N_{\ell}=50\) neurons in each hidden layer (i.e., for \(\ell=1,\ldots,L-1\)); the activation function is the hyperbolic tangent in each hidden layer. For the first iteration of the iterative training, the neural network weights in the \(\ell\)-th layer are initialized with a glorot normal distribution, i.e., a truncated normal distribution with mean \(0\) and standard deviation equal to \(\sqrt{2/(N_{\ell-1}+N_{\ell})}\). Then, for the subsequent iterations, their are initialized with the weights obtained at the end of the previous one. During the first iteration of the training (during the minimization of \(R_{h}^{2}(\cdot;\mathcal{P}_{0})\)), the optimization is carried out by exploiting the ADAM optimizer [51] with an exponentially decaying learning rate from \(10^{-2}\) to \(10^{-4}\) and with the second-order L-BFGS optimizer [52]. Then, from the second training iteration, we only use the L-BFGS optimizer. We remark that L-BFGS allows a very fast convergence but only if the initial starting point is close enough to the problem's solution. Therefore, in the first training iteration, we use ADAM to obtain a first approximation of the solution that is then improved via L-BFGS. Then, since the \(m\)-th training iteration starts from the solution computed during the \((m-1)\)-th one, we assume that the starting point is close enough to the solution of the new optimization problem (associated with a difference loss function with more patches) and we only use L-BFGS to increase the training efficiency. During the \(m\)-th iteration of the training, the training set consists of all the quadrature nodes \(\mathfrak{F}_{\ell}^{i}\), for any \(\ell\in I^{\hat{P}}\) and for any patch \(P_{i}\in\mathcal{P}_{m}\) as defined in (9). The order of the chosen quadrature rule is \(q=3\) inside each triangle. The Dirichlet boundary conditions are imposed by means of the operator \(B\) defined in (6). In this operator, for our first numerical test, the function \(\phi\) is a polynomial bubble vanishing on \(\Gamma\) and \(\overline{g}\) is the output of a neural network trained to interpolate the boundary data. For the numerical test in Section 3.4, instead, \(\phi\) is computed as in [50] and \(g=0\). To decrease the training time, the functions \(\phi\), \(\
abla\phi\), \(\overline{g}\) and \(\
abla\overline{g}\) are evaluated only once at the beginning of the \(m\)-th training iteration and they are then combined to evaluate \(Bu^{\mathcal{N}\mathcal{N}}\) and its gradient (where \(u^{\mathcal{N}\mathcal{N}}\) is the output of the last layer of the neural network). The derivatives of \(u^{\mathcal{N}\mathcal{N}}\) and \(\overline{g}\) are computed via automatic differentiation [53] due to the complexity of their analytical expressions. The output of the model is the value of the function \(Bu^{\mathcal{N}\mathcal{N}}\) and its gradient evaluated at the input points. Such values are then suitably combined using sparse and dense tensors to compute the quantity \(R_{h}^{2}(Bu^{\mathcal{N}\mathcal{N}};\mathcal{P}_{m})\). The sparse tensors contain the evaluation of \(\varphi_{i}\) and \(\
abla\varphi_{i}\) at each input point, whereas the dense ones store the quadrature weights, the vector \(\gamma=\{\gamma_{i}\}_{i=1}^{n_{\text{patches}}}\) and the evaluation of \(\mu\), \(\beta\), \(\sigma\) and \(f\) at the input points. We highlight that all these tensors have to be computed once at the beginning of the \(m\)-th training iteration (updating the ones of the \((m-1)\)-th iteration) to significantly decrease the training computational cost. As discussed in Section 2.1, we assume that all the patches and test functions can be generated from a reference patch \(\hat{P}\).