## 5 Neural Networks Approximation in PI Algorithm
### PI Algorithms for Robust Control  
For the system with matched uncertainty, the optimal control problem (4) with (5) is considered. For any admissible control, the corresponding cost function can be expressed as  
\[ \begin{split} V[x(t)]&=\int_{t}^{\infty}[f^{2}_{max} (x)+x^{T}x+u^{T}u]dt\\ &=\int_{t}^{t+T}[f^{2}_{max}(x)+x^{T}x+u^{T}u]dt+\int_{t+T}^{ \infty}[f^{2}_{max}(x)+x^{T}x+u^{T}u]dt\end{split} \]  
where \(T>0\) is a selected constant. Therefore, it follows that  
\[V[x(t)]=\int_{t}^{t+T}[f^{2}_{max}(x)+x^{T}x+u^{T}u]dt+V[x(t+T)] \tag{29}\]  
Based on the integral reinforcement relationship (29) and optimal control (9), the PI algorithm of robust control for matched uncertain nonlinear systems is given below. The convergence of Algorithm 1 is illustrated as follows. The following conclusion gives an equivalent form of the Bellman Equation (30). \begin{tabular}{l l} \hline \hline
**Algorithm 1** PI algorithm of robust control for matched uncertain nonlinear systems \\ \hline (1) & Select supremum \(f_{max}(x)\) to satisfy \(\|h(x)\|\leq f_{max}(x)\); \\ (2) & Initialization: for the nominal nonlinear system (4), select an initial stabilization control \\ \(u_{0}(x)\); \\ (3) & Policy evaluation: for control input \(u_{i}(x)\), calculate cost \(V_{i}(x)\) from the Bellman equation \\ \[V_{i}[x(t)]=\int_{t}^{t+T}[f_{max}^{2}(x)+x^{T}x+u_{i}^{T}(x)u_{i}(x)]dt+V_{i}[ x(t+T)]\] (30) \\ (4) & Policy improvement: compute the control law \(u_{i+1}(x)\) using \\ \[u_{i+1}(x)=-\frac{1}{2}\delta^{T}(x)\
abla V_{i}.\] (31) \\ \end{tabular}   
By repeatedly iterating between (30) and (31), until the control input is convergent. **Proposition 1**. : _Suppose that \(u_{i}(x)\) is a stabilization controller of nominal system (4). Then the optimal cost \(V_{i}(x)\) solved from (30) is equivalent to solving the following equation_  
\[f_{max}^{2}(x)+x^{T}x+u_{i}^{T}(x)u_{i}(x)+\
abla V_{i}[f(x)+g(x)u_{i}(x)]=0. \tag{32}\]  
Proof. : Dividing both sides of (30) by \(T\) and finding the limit yields  
\[\lim_{T\to 0}\frac{V_{i}[x(t+T)]-V_{i}[x(t)]}{T}+\lim_{T\to 0}\frac{\int_{t}^{t+T}[f_{max}^{2}(x)+x^{T}x+u_{i}^{T}( x)u_{i}(x)]dt}{T}=0\]  
From the definition of function limit and [PERSON] rule, we can get  
\[\frac{dV_{i}[x(t)]}{dt}+\lim_{T\to 0}\frac{d}{dT}\int_{t}^{t+T}[f_{max}^{2}(x)+x^{ T}x+u_{i}^{T}(x)u_{i}(x)]dt=0 \tag{33}\]  
It follows that  
\[f_{max}^{2}(x)+x^{T}x+u_{i}^{T}(x)u_{i}(x)+\
abla V_{i}[f(x)+g(x)u_{i}(x)]=0\]  
Thus, we can deduce (32) from (30). On the other hand, along the stable system \(\dot{x}=f(x)+g(x)u_{i}(x)\), finding the time derivative of \(V_{i}(x)\) yields  
\[\frac{dV_{i}[x(t)]}{dt}=\
abla V_{i}[f(x)+g(x)u_{i}(x)]\]  
Integrating both sides from \(t\) to \(t+T\), yields  
\[V_{i}[x(t+T)]-V_{i}[x(t)]=\int_{t}^{t+T}\
abla V_{i}[f(x)+g(x)u_{i}(x)]dt\]  
Therefore, we can get the following result from (32)  
\[V_{i}[x(t)]=\int_{t}^{t+T}[f_{max}^{2}(x)+x^{T}x+u_{i}^{T}(x)u_{i}(x)]dt+V_{i }[x(t+T)]\]  
This proves that (32) can deduce (30). According to [32; 33; 34], if the initial stabilization control policy is given \(u_{0}(x)\), then the follow-up control policy calculated by the iterative relations of (30) and (31) is also a stabilizing control policy, and cost sequence \(V_{i}[x(t)]\) calculated by iteration converges to the optimal cost. By Proposition 1, it is known that (30) and (32) are equivalent, so the iterative relations (30) and (31) in Algorithm 1 converge to the optimal control and optimal cost. Similarly, we give a PI algorithm of robust control for nonlinear systems with mismatched uncertainties. The steps of policy evaluation (34) and policy improvement (35) are iteratively calculated until the policy improvement step does not change the current policy. The optimal cost function is calculated as \(V^{*}(x)\), then \(u^{*}(x)=-\frac{1}{2}g^{T}(x)\
abla V^{*}(x)\) is the robust control law. The convergence proof of Algorithm 2 is similar to Algorithm 1, which will not be repeated here.