## 3 Optimization-Based Strategies
### Data-Driven Strategies
#### 3.3.3 Adaptive Dynamic Programming (ADP)
To identify the system dynamics based on the input-output data, the model neural network is trained in advance to obtain the next state, which takes the current state and control inputs as the inputs of MNN. The forward calculation of can be expressed as:  
\[\left\{ \begin{array}{c}inputM=\left[u_{t}^{T},x_{t}^{T}\right]^{T}\\ \hat{x}_{t+1}=W_{m2}^{T}(t)\cdot\varphi_{m}\left(W_{m1}^{T}(t)\cdot inputM(t)+ \xi_{m1}\right)+\xi_{m2}\end{array} \right. \tag{24}\]  
where \(W_{m1}\) is the input-to-hidden weight vector of MNN, \(W_{m2}\) is hidden-to-output weight vector of MNN, \(\xi_{m1}\) is the threshold of the hidden layer of MNN, \(\xi_{m2}\) is the threshold of the output layer of MNN, \(\hat{x}_{t+1}\) is the next system state vector, and \(\varphi_{m}(\cdot)\) is the activation function, which can be selected as \(\varphi_{m}(\cdot)=\frac{1-e^{-\alpha}}{1+e^{-\alpha}}\). The identification error of MNN is defined as \(e_{m}=x_{t+1}-\hat{x}_{t+1}\), where \(x_{t+1}\) is the expected value. The quadratic form of the identification error is denoted as the model network error: \(E_{m}=\frac{1}{2}e_{m}^{T}e_{m}\). The gradient-based descent approach is used for weight updating during the back-propagation process, which is:  
\[W_{m}(t+1)=W_{m}(t)-\eta_{m}\frac{\partial E_{m}}{\partial e_{m}}\frac{ \partial e_{m}}{\partial\hat{x}_{t+1}}\frac{\partial\hat{x}_{t+1}}{\partial W _{m}} \tag{25}\]  
Figure 12: Depiction of the main steps in applying ADP to achieve energy management control. The critic network is used to approximate the iterative cost function, and there are two critic networks in each iteration, which have the same input-to-hidden vector \(V_{c1}\) and hidden-to-output weight vector \(V_{c2}\). In the forward calculation of CNN, the first critic network (CNN1) takes current state \(x_{t}\) as the input and outputs the cost function of the current moment \(\hat{J}(x_{t})\), while the second critic network (CNN2) outputs the cost function of the next moment \(\hat{J}(x_{t+1})\) and takes the next state \(\hat{x}_{t+1}\) as the input, which is the output of MNN. The cost function is formulated as:  
\[\hat{J}(x_{t})=V_{c2}^{T}\varphi_{c}\big{(}V_{c1}^{T}x_{t}+\xi_{c1}\big{)}+\xi_ {c2} \tag{26}\]  
where \(\xi_{c1}\) is the threshold of the hidden layer of CNN1, \(\xi_{c2}\) is the threshold of the output layer, and \(\varphi_{c}(\cdot)\) is the activation function, which can be selected as \(\varphi_{c}(\cdot)=\frac{1-e^{-ax}}{1+e^{-ax}}\). The quadratic form of the error between the estimated cost function and the expected value is defined as the critic network error. The expected cost function in each iteration is denoted as:  
\[J(x_{t})=\big{(}x_{t}\cdot A\cdot x_{t}^{T}+u_{t}\cdot A\cdot u_{t}^{T}+\beta \big{)}(x_{t+1})\big{)} \tag{27}\]  
Similarly, the weight vectors are regulated based on the gradient-based adaptation algorithm [109]. \[W_{c}(t+1)=W_{c}(t)-\eta_{c}\cdot\frac{\partial E_{c}}{\partial e_{c}}\frac{ \partial e_{c}}{\partial\hat{J}_{t}}\frac{\partial\hat{J}_{t}}{\partial W_{c}} \tag{28}\]  
The action network with three-layer neural network is constructed to approach the iterative control law. In the forward calculation, ANN takes the current state as the input and outputs the estimated control law \(\hat{u}_{t}\):  
\[\hat{u}_{t}=W_{a2}^{T}(t)\cdot\varphi_{a}\big{(}W_{a1}^{T}(t)\cdot x_{t}+\xi_ {a1}\big{)}+\xi_{a2} \tag{29}\]  
where \(W_{a1}\) is the input-to-hidden weight vector of ANN, \(W_{a2}\) is the hidden-to-output weight vector of ANN, \(\xi_{a1}\) is the threshold of the hidden layer, \(\xi_{a2}\) is the threshold of the output layer of MNN, and \(\varphi_{a}(\cdot)\) is the activation function, which can be selected as \(\varphi_{a}(\cdot)=\frac{1-e^{-ax}}{1+e^{-ax}}\).