## 3 Optimization-Based Strategies
### Global Information-Driven Optimization
#### 3.2.3 Stochastic Dynamic Programming (SDP)  
The stochastic dynamic programming algorithm uses the existing standard driving cycles or historical driving data as the sample to establish the statistical model of driver demand, and then uses dynamic programming to solve the energy management problem represented by the statistical model. This optimal strategy is the strategy with the lowest expected cost in the general sense, and it can be applied to energy management of various hybrid vehicles with different drive configurations. Considering a stochastic dynamic programming problem with a finite state space \(\mathrm{S}=\left[s^{1},s^{2},\ldots,s^{N}\right]\) and a control space \(A\), the cost function is \(L:S\times A\to L\), and the transition probability matrix is \(T(s,a,s^{\prime})\). The cost function means that the cost that the state \(s\) moves to the state \(s^{\prime}\) under the control action \(a\), while the \(T(s,a,s^{\prime})\) represents the probability that the state \(s\) moves to the state \(s^{\prime}\) under the control action \(a\). As for a certain state \(s\) in the state space of the system, it has a finite set \(A_{s}=\left\{a^{1},a^{2},\ldots,a^{M}\right\}\) that the system can take action to move to any other state. That is, the system performs based on the state transition matrix \(T\). When applying SDP to solve the energy management for HEVs, the objective is to find the optimal control variables under some constraints to minimize fuel consumption. Therefore, the battery SOC, the vehicle speed and the required power can be chosen as state variables, and the engine speed and torque distribution ratio can be chosen as control variables. In addition, the cost function can be expressed as:  
\[J=\lim_{N\rightarrow\infty}E\Bigg{\{}\sum_{k=0}^{N-1}\beta^{k}L(x(k),u(k)) \Bigg{\}} \tag{17}\]  
\[J(s^{\prime})=min\sum T(s,s^{\prime})(L(s,s^{\prime})+J(s)) \tag{18}\]  
\[fuel_{k+1}(s^{\prime})=min\sum T(s,u,s^{\prime})\Big{(}fuel_{step\,k}(s,u,s^{ \prime})+fuel_{k}(s)\Big{)} \tag{19}\]  
where \(\beta\in[0,1]\) is the discount to ensure the convergence of the total cost function at the expected iteration step, and the value reflects the importance of the future costs compared with the current costs. The closer the value is to 1, the greater the proportion of future long-term expectations in the total cost function. \(T\) is the state transition probability matrix, \(L(x(k),u(k))\) is the one step reward function in the \(k\) moment, which consists of the fuel consumption of the engine \(fuel_{step\,k}\); sometimes, it also includes the electricity consumption of the motors or the penalty term for the deviation of SOC from the expected final value, which can be defined as \(w_{s}(SOC(t)-SOC_{r})^{2}\). The demand power can be modeled using a Markov chain model, which is established by the transition probability matrix. The state transition probability matrix \(T\) can be obtained by the maximum likelihood estimation method as follows:  
\[T(i,j)=\frac{m_{ij}}{m_{i}} \tag{20}\]  
where \(m_{ij}\) is the frequency of the requested power, \(m_{i}\) is the total frequency. As for the solution to SDP method, there are three typical algorithms: value iteration, policy iteration and modified policy iteration. The value iteration method calculates the optimal cost function \(J^{*}\), and then acquires the optimal strategy based on the optimality principle. Unlike the value iteration, the policy iteration contains policy evaluation and policy improvement, which calculates the value function of all states by giving an initial strategy \(\pi_{0}\) in advance and calculates a series of new strategies based on the greedy strategy, and then repeats the above steps. The modified policy iteration combining the value iteration and the policy iteration is more effective than the above strategies.