# Efficient Solving of Boundary Value Problems Using Radial Basis Function Networks Learned by Trust Region Method

Authors: Mohie Mortadha Alqezweeni; Vladimir Ivanovich Gorbachenko; Maxim Valerievich Zhukov; Mustafa Sadeq Jaafar
Journal: International Journal of Mathematics and Mathematical Sciences
 (2018)

---

## Abstract

A method using radial basis function networks (RBFNs) to solve boundary value problems of mathematical physics is presented in this paper. The main advantages of mesh-free methods based on RBFN are explained here. To learn RBFNs, the Trust Region Method (TRM) is proposed, which simplifies the process of network structure selection and reduces time expenses to adjust their parameters. Application of the proposed algorithm is illustrated by solving two-dimensional Poisson equation.


---

## Body

## 1. Introduction

Mesh-free methods of solving boundary value problems have been widely studied in different researches in the last decade [1]. They belong to the class of projection methods [2–6]. In this paper, we are studying one of the most promising mesh-free methods using the radial basis functions (RBFs). In the case of their application, the approximation of the decision is presented as the weighted sum of RBFs where weights are selected in such a way that the approximated solution satisfies the boundary value problem in the selected sample points. The major difficulty of using the mesh-free methods based on RBFs is the nonformalizable selection of basis function parameters. This issue can be overcome with radial basis function neural networks (RBFNs) [7–11]. The process of boundary value problems’ solution using RBFNs comes down to the RBFNs learning. The main difference between the RBFN method and other mesh-free methods is that it adjusts not only the weight of the basis functions but also their parameters [7–11]. In [12], it was suggested to use the Trust Region Method (TRM) for training RBNFs [13]. There are new results to improve the Trust Region Method performance in the submitted article, in particular the use of the Hessian matrix approximation to reduce the computation cost. Two approaches were considered for the initial allocation of radial basis function centers, and more optimal values of Trust Region Method hyperparameters were found. In Section 2, the RBFNs are discussed in order to solve the boundary value problems. Section 3 presents the proposed methodology of learning RBFNs using TRM. Section 4 presents the experimental analysis of the proposed solution to solve the boundary value problems. Finally, Section 5 presents the conclusion of the paper.


## 2. Radial Basis Function Networks

RBFN is a network consisting of two layers [14]:(i)
The first layer realizes nonlinear conversion of an input vectorx=(x1,x2,…,xd)∈Rd.(ii)
The second layer does the linear summation. RBF is used as a conversion function.The output of a network is described by the expression(1)ux=∑m=1Mwmφmx;pm,where M is the quantity of RBFs, wm is the weight of RBF, and pm is the RBF parameter vector F. Gauss’s functions, multiquadrics, and the reverse multiquadrics, among others, are examples of RBFs [14].Consider the process of solving boundary value problems using RBFNs using an example of the boundary value problem, given in the following operator form:(2)Lux=fx,x∈Ω,Bux=px,x∈∂Ω,where u is a desired solution; L is a differential operator; B is an operator of the boundary conditions; Ω is an area of solutions; ∂Ω is a border of the area; f and p are known functions. The solution is approximated by RBFN:( 1 ) Select N domestic and K boundary sample points from the sets Ω and ∂Ω.( 2 ) Determine the structure of the network: the network type, the number of RBFs M, and the RBFs type.Specify initial values:w=(w1,w2,…,wM) are the weights of RBFs and p=(p1,p2,…,pM) are parameters of RBFs (the structure of the vector p elements depends on the type of the functions; e.g., for a two-dimensional Gaussian function, it can be defined as pi=(ai,ci1,ci2), where i=1⋯M¯, ai is the shape parameter (width), and (ci1,ci2) is the position of its center).( 3 ) Learn the network; that is, find such values of w and p that the error functional representing the sum of squares of residuals in sample points reaches the minimum value: (3)
J
w
,
p
=
∑
i
=
1
N
L
u
x
i
;
w
,
p
-
f
x
i
2
+
λ
∑
i
=
N
+
1
N
+
K
B
u
x
i
;
w
,
p
-
p
x
i
2
⟶
m
i
n
,where x1,x2,…,xN∈Ω, xN+1,xN+2,…,xN+K∈∂Ωλ is the selectable penalty factor, which takes into account the residuals in the boundary sample points. The trained network provides the solution at any arbitrary point when its coordinate comes to the input layer of the network.


## 3. Learning RBFN Using TRM

The efficiency of neural network method for solving boundary value problems depends on the efficiency of the method of solving the problem of the error functional minimization (3). TRM is one of the best approaches to solve problem (3) [12, 15]. The key features of this method are as follows: it has the possibility of simultaneous optimization of a large number of parameters; it has a high level of efficiency and convergence even for bad-conditioned problems; it can overcome local minimums; it has the capability of minimization of concave functions, that is, functions with negative definite Hessian matrix; and finally, when the second-order Taylor expansion is used as an approximating function, the problem of the objective function minimization reduces to the problem of minimizing a quadratic functional [13]. All these make TRM ideal for solution of the problem of minimizing the error functional, which often has a large number of parameters to be optimized and many local minima, and it is an ill-conditioned problem.The basic idea of TRM is that, at each iterationk of the function f(x) minimization, where x∈Ω⊆Rn (Ω is the domain of definition and R is a set of real numbers), the function f is replaced by an approximating function mk in the trust region Bk⊆Ω and the minimum of mk is calculated in Bk, which becomes a new minimum of f.Depending on how the decrease predicted by the model is confirmed by the objective function, the decision on the expansion or contraction of the trust region is taken. There is a formal description of the algorithm.Algorithm 1 (TRM).
Step 1 (initialization). Specify an initial value of x0, the radius of the trust region Δ0, the threshold accuracy of the estimated models μ1 and μ2 such that 0<μ1≤μ2<1, the transformation coefficients γ1 and γ2 of the trust region (0<γ1≤γ2<1), and the number of the iterations k=0.
Step 2 (approximation off). Select the norm · and construct a function mk, approximating function f in the area Bk.
Step 3 (minimization ofmk). Select the method of conditional minimization of mk, which allows finding such a step sk that point xk+sk is a global minimum of mk in Bk.
Step  4. This step consists in evaluation of the model accuracy pk that is calculated as follows: (4)pk=fxk-fxk+skmkxk-mkxk+sk.If pk≥μ1, then xk+1=xk+sk; otherwise, xk+1=xk.
Step  5. Change the trust region radius(5)Δk+1∈Δk;∞,ifpk≥μ2,γ2Δk;Δk,ifpk∈μ1,μ2,γ1Δk;γ2Δk,ifpk<μ1.
Step 6 (increasing the number of the iterationsk=k+1). If the required accuracy of the solution has been reached, or k is equal to the maximum number of iterations, or the radius of the trust region is too small, then complete the learning; otherwise, go back to Step 2.
Step 7 (stop). The algorithm is a generalized algorithm of TRM, as it has no instructions on how to construct a function mk, select the norm ·, and what method to use for mk minimization.
Since the error functional is a twice differentiable function, hence the second-order Taylor series can be used asmk model of J; as the norm, we are using the Euclidean norm here. To get the second-order Taylor series, it is necessary to calculate the Hessian matrix, which has a large computation cost. Instead of the exact value of the Hessian, we are using its approximate value, which is a multiplication of the Jacobi matrices H(x)≈G(x)=[J(x)]TJ(x), where the Jacobian of the error function has the form(6)Jq=∂r1∂q1∂r1∂q2⋯∂r1∂qM·1+d+l∂r2∂q1∂r2∂q2⋯∂r2∂qM·1+d+l⋮⋮⋱⋮∂rN+K∂q1∂rN+K∂q2⋯∂rN+K∂qM·1+d+l,where(7)riq=Luxi;q-fxi,1≤i≤N,λBuxi;q-pxi,N<i≤N+K,q=w1,p11,…,pl1,…,wM,p1M,…,plM.Using the second-order Taylor series leads to the need for solving the conditional problem of a quadratic functional minimization. To solve it, Steihaug’s method [16] is used here. This method is a modification of the method of preconditioned conjugate directions, taking into account the restrictions on the solution (the solution should lie in Bk) during mk functional minimization and allowing working with negative definite Hessian matrix.


## 4. Experimental Study

As an example of solving boundary value problems using RBFN, learned by TRM, consider the boundary value problem for the two-dimensional Poisson equation, described in [8](8)∂2ux,y∂x2+∂2ux,y∂y2=fx,y,x,y∈Ω,ux,y=px,y,x,y∈∂Ω.This problem often arises in thermodynamics, electrostatics, and image processing.Let the solution domain be square bounded by the points(0,0) and (1,1), f(x,y)=sin⁡πxsin⁡πy, p(x,y)=0. This problem has an analytical solution ua(x,y)=-1/2π2sin⁡πxsin⁡πy. We will use it to evaluate the accuracy of the resulting numerical solution. There is an error functional of this problem: (9)Jw,p=∑i=1N∂2uRBFxi,yi;w,p∂x2+∂2uRBFxi,yi;w,p∂y2-sin⁡πxisin⁡πyj2+λ∑i=N+1N+KuRBFxi,yi;w,p2.Two experiments were done with different initial values of the network parameters. In both experiments, 144 randomly selected sample points were used to learn the networks. 100 of them were located in the area of solutions Ω, and 44 were located on the border ∂Ω. Gaussian function was used as RBF. In the first experiment, RBFN consisted of 16 neurons, whose centers were randomly distributed in a square area bounded by points (-0.2;-0.2) and (1.2;1.2). The width of the RBFs was randomly chosen from the interval [0.3;0.6]. Initial values of the RBF weights have been chosen randomly from the interval [-0.05;0.05]. Penalty factor λ=1000. TRM parameters were equal to Δ0=2, μ1=0.2, μ2=0.7, γ1=0.5, and γ2=0.7. Note that the presented hyperparameters of both RBFN and TRM were selected manually, although more advanced techniques can be used in the future, like grid search, genetic algorithm, and others.Learning of the network was completed in 8 iterations, with the value of the solution error being equal to1.7·10-4. The solution error was calculated according to the formula of the relative standard error: (10)ε=∑i=1N+KuRBFxi,yi;w,p-uaxi,yi2∑i=1N+Kuaxi,yi2.The error graph is shown in Figure 1.Figure 1
Experiment number 1, error solutions.In the second experiment, the RBFN consisted of 64 neurons, whose centers were located in the nodes of a uniform square grid bounded by points(-0.2;-0.2) and (1.2;1.2); the neurons’ width was constant and equal to 0.5. Initial values of weights of the RBFs were equal to zero. The solution error was equal to 5.3·10-5 and was reached for 15 iterations. The error graph is shown in Figure 2.Figure 2
Experiment number 2, error solutions.As expected, the more the neurons in the network, the more accurately it approximates the desired solution. However, the process of learning such a network requires much more time (for learning the network from the first experiment, it needed two times fewer iterations than training a network from the second experiment). The dependency of the errorε (10) on the iteration number k is shown in Figure 3.Figure 3
The dependency of the errorε (10) on the iteration number k.


## 5. Conclusion

In this paper, a method based on TRM was proposed for RBFNs learning. Approximate values of the Hessian matrix are used to improve its performance. This method allows reducing the time of network’s learning.



---

*Source: 9457578-2018-06-03.xml*