## 5 Neural Networks Approximation in PI Algorithm
### Neural Network Approximation of Optimal Cost in PI Algorithm  
In the implementation of the PI algorithms, we need to use the data of the nominal system and use the least square method to solve the cost function. However, the cost function of nonlinear optimal control problem has no specific form. Therefore, it is necessary to use neural network structure to approximate the cost function, carry out policy iteration, update weights, and then obtain the approximate optimal cost function. In this subsection, neural network is utilized to approximate the optimal cost in the corresponding HJB equation. Based on the continuous approximation theory of neural network [35], a single neural network is utilized to approximate the optimal cost in HJB equation. For matched uncertain systems, suppose that the solution \(V^{*}(x)\) of HJB Equation (10) is smooth positive definite, and the optimal cost function on compact set \(\Omega\) is expressed as  
\[V^{*}(x)=W^{T}\phi(x)+\varepsilon(x) \tag{36}\]  
where \(W\in\mathbb{R}^{L}\) is an unknown ideal weight, and \(\phi(\cdot):\mathbb{R}^{n}\rightarrow\mathbb{R}^{L}\) is a linear independent basis vector function. It is assumed that \(\phi(x)\) is continuous, \(\phi(0)=0\), and \(\varepsilon(x)\) is the error vector of neural network reconstruction. Thus, the gradient of the optimal cost function (36) can be expressed as  
\[\
abla V^{*}=\frac{\partial V^{*}}{\partial x}=\
abla\phi^{T}(x)W+\
abla\varepsilon (x) \tag{37}\]  
where \(\
abla\varepsilon(x)=\frac{\partial\varepsilon}{\partial x}\). On the basis of approximation property of neural network [35; 36], when the number of neurons in hidden layer \(L\rightarrow\infty\), the approximation error \(\varepsilon(x)\to 0\), \(\
abla\varepsilon(x)\to 0\). Substituting (36) and (37) into (9), the optimal control is rewritten as follows  
\[u^{*}(x)=-\frac{1}{2}g^{T}(x)[\
abla\phi^{T}(x)+\
abla\varepsilon(x)] \tag{38}\]  
Assume that \(\hat{W}\) is an estimated value of the ideal weight \(W\). Since the ideal weight \(W\) in (36) is unknown, the cost function of the \(i-th\) iteration in Algorithm 1 is expressed as  
\[\hat{V}_{i}(x)=\hat{W}_{i}^{T}\phi(x) \tag{39}\]  
Using the approximation of neural network in cost function, the Bellman Equation (30) in Algorithm 1 is rewritten as follows  
\[\hat{W}_{i}^{T}\phi(x(t))=\Psi+\hat{W}_{i}^{T}\phi(x(t+T)) \tag{40}\]  
where \(\Psi=\int_{t}^{t+T}[f_{max}^{2}(x)+x^{T}x+u_{i}^{T}(x)u_{i}(x)]dt\). Since the above formula uses neural network to approximate the cost function, the residual error caused by neural network approximation is  
\[\varepsilon_{i}(x(t),T)=\Psi+\hat{W}_{i}^{T}\phi(x(t+T))-\hat{W}_{i}^{T}\phi( x(t)) \tag{41}\]  
In order to obtain the neural network weight parameters of approximation function, the following objective functions can be minimized in the meaning of least square  
\[E=\int_{\Omega}\varepsilon_{i}(x(t),T)^{T}\varepsilon_{i}(x(t),T)dx, \tag{42}\]  
that is \(\int_{\Omega}\frac{d\varepsilon_{i}(x(t),T)}{d\hat{W}_{i}}\varepsilon_{i}(x(t ),T)dx=0\). Using the definition of inner product, it can be rewritten as  
\[\langle\frac{d\varepsilon_{i}(x(t),T)}{d\hat{W}_{i}},\varepsilon_{i}(x(t),T) \rangle_{\Omega}=0 \tag{43}\]  
It follows from properties of the internal product that  
\[\Phi\hat{W}_{i}+\langle[\phi(x(t+T))-\phi(x(t))],\Psi\rangle_{\Omega}=0 \tag{44}\]  
where \(\Phi=\langle[\phi(x(t+T))-\phi(x(t))],[\phi(x(t+T))-\phi(x(t))]^{T}\rangle\). Therefore,  
\[\hat{W}_{i}=-\Phi^{-1}\langle[\phi(x(t+T))-\phi(x(t))],\Psi\rangle_{\Omega} \tag{45}\]  
So far, the neural network weight parameters of approximation function \(V_{i}(x)\) can be calculated. Thus, the update control policy can be obtained from (35)  
\[\
abla V^{*}=\frac{\partial V^{*}}{\partial x}=\
abla\phi^{T}(x)W+\
abla\varepsilon (x) \tag{37}\]0  
According to [32; 33; 35; 36], using the policy iteration of RL algorithm, the cost sequence \(V_{i}(x)\) converges to the optimal cost \(V^{*}(x)\), and the control sequence \(u_{i}(x)\) converges to the optimal control function \(u^{*}(x)\).