## 3 Optimization-Based Strategies
### Data-Driven Strategies
#### 3.3.3 Adaptive Dynamic Programming (ADP)  
With the increased dimensions of the state variables \(x\) and the control variables \(u\), the computation and storage capacity of the traditional dynamic programming increase significantly, often leading to the phenomenon of \"dimensional disasters\". In addition, the nonlinear optimal control needs to solve the nonlinear Hamilton-Jacobi-Bellman (HJB) equation; however, this involves solving nonlinear partial difference equations, which are difficult to work out. With the development of artificial neural networks characterized by their strong abilities for self-learning and adaptivity, adaptive dynamic programming (ADP) as an approximate solution to DP has been applied in effectively solving the dynamic programming problem of nonlinear discrete-time systems with strong coupling, strong nonlinearity and high complexity. This method has been put into practical application in transportation, power systems, industrial production, and other aspects. The core idea of adaptive dynamic programming is to approximate the performance index function and control strategy in the dynamic programming equation by using the function approximation structure to satisfy the optimality principle and obtain the optimal control sequence. The performance index function can be expressed as:  
\[J(x_{t})=\sum_{t=0}^{N-1}\beta^{t}U(x_{t},u_{t})=U(x_{t},u_{t})+\beta J(x_{t+1}) \tag{22}\]  
where \(U\) is the utility function, \(J\) is the performance index function (cost function), and \(\beta\) is the discount factor. ADP consists of three parts, each of which can be replaced by a neural network with the simplest neural network-BP, as shown in Figure 11. The model network (MNN) is used to approximate the dynamic system to estimate the state at the next moment, the action network (ANN) is used to approximate the optimal control law by mapping the relationship between state and control variables, and the critic network (CNN) is used to estimate the optimal performance index function. According to the different input and output of the critic network (CNN), the adaptive dynamic programming (ADP) can be roughly subdivided into six categories. Among them, heuristic dynamic programming (HDP) [103] is the most basic and widely used structure among ADP algorithms, CNN output of which is the cost function. When the output of CNN is the partial derivative of the cost function, the structure turns into dual heuristic programming (DHP) [104]. To improve calculation accuracy and efficiency, researchers have developed two improved structures: action-dependent heuristic dynamic programming (ADHDP) and action-dependent dual heuristic programming (ADDHP) [105], whose inputs of CNN are the state and  
* Determine the utility function. Generally speaking, the utility function \(U(x,u,k)\) can be chosen as the quadratic form of the states and the control variables: \[U(x,u,k)=x_{k}\cdot A\cdot x_{k}^{T}+u_{k}\cdot B\cdot u_{k}^{T}\] (23) where \(A\) and \(B\) are symmetric positive definite matrices with appropriate dimensions. If there is a particular constraint on SOC value, the square of the error between SOC and the standard value can be added to the utility function. * Design the ADP algorithm. The error definition and weight update are the focal points in it. * Design the algorithm procedure of ADP. The key procedure is to train ANN and CNN until the stopping rule is satisfied;
* Analyze the convergence and stability of the algorithm, which includes theoretical proof and analysis of the simulation results. The uniform ultimate boundedness (UUB) of the associated weight estimation errors is proved using a (positive) Lyapunov approach [108]. In the implementation of HDP, the ADP structure consists of an action neural network (ANN), a critic neural network (CNN) and a model neural network (MNN), which are established by a three-layer neural network (BP-NN) and all contain forward calculation and backward error back propagation to update weight.