# The Development of an Optimal Control Strategy for a Series Hydraulic Hybrid Vehicle
## 3 Control System Development
### Dynamic Programming (DP) Optimal Control Strategy with DP Application  
#### 3.3.1 The Fundamental Formulation of DP  
Many textbooks and papers have been published on the subject of dynamic programming (DP) theory and its applications. DP is a powerful numerical method for solving optimal control problems. DP is a computational technique that extends the decision-making concept to sequences of decisions, which together define an optimal policy and trajectory. An optimal solution for the original problem can be found by using optimal solutions of sub-problems for each decision. The sub-problems are solved recursively in the same fashion. This method uses [PERSON]'s Principle of Optimality [18]: \"An optimal  
Figure 5: Schematic of Thermostatic Control Scheme. Figure 6: Schematic of Modulated-Pressure Control Strategy. policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.\"  
For a given system, DP is used to find the optimal control input that minimizes a certain cost function. The main advantage of DP is that the global optimal solution is guaranteed for any type of problem. However, the computational cost for DP grows exponentially with the number of state variables and inputs for the underlying dynamic system. [PERSON] called this difficulty the \"curse of dimensionality.\" The main drawback of this approach is that all disturbances must be known _a priori_. Therefore, DP is often not a useful method for the design of real-time control systems. However, DP is a very useful tool for providing an optimal performance benchmark. In the design of a causal real-time controller, this benchmark is then used to assess the quality of a controller. In some cases, the optimal non-realizable solution provides an insight into how the suboptimal but realizable control system should be designed. In order to apply DP, the dynamic system is considered as a discrete-time system, which is expressed as:  
\[x_{k+1}=f_{x}\left(x_{k},u_{k},w_{k}\right),k=0,1,\ldots,N-1 \tag{1}\]  
the dynamic states, \(x_{k}\in X_{k}\in IR_{\delta}^{n}\), the control inputs, \(u_{k}\in U_{k}\in IR_{\delta}^{m}\), and the disturbances, \(w_{k}\in D_{k}\in IR_{\delta}^{d}\), are discrete variables both in time (index \(k\)) and value. The control inputs, \(u_{k}\), are limited to the subset, \(U_{k}\), which depends on the values of the state variables, \(x_{k}\); _i.e.,_\(u_{k}\in U_{k}\left(x_{k}\right)\). As mentioned previously, the disturbance, \(w_{k}\), must be known in advance for all \(k\in[0,N-1]\). To evaluate the fuel economy of a vehicle, the vehicle's speed profile is considered as a known disturbance. The control variable, \(u\), and the state variable, \(x\), must be discretized in time and space, as shown in Figure 7. Because the computation time and the amount of memory required for DP grows exponentially with the number of states and control inputs, this number must be as small as possible. The selection of grid sizes for the states and the control inputs are a compromise between the representation of the system's dynamics and computation time. If the grid size is too large, the state does not change as the control changes. However, if the grid size is too small, DP requires a large number of computations. Based on the principle of optimality, the calculation of the DP can be done in reverse following algorithm:  
At \(k=N\), if there is no constraint on the final state, \(g_{N}\left(x_{N}\right)=\ 0\). If there is constraint on the final state, the final cost is defined as:  
\[g_{N}\left(x_{N}\right)=\left\{ \begin{array}{ll}\infty&\text{if}x\left(N \right)\
eq x_{N}\\ 0&\text{if}\ x\left(N\right)=x_{N}\end{array} \right. \tag{2}\]  
Step \(N-1\):  
\[J_{N-1}^{*}\left(x_{N-1}\right)=\underset{u_{N-1}}{\text{min}}g_{N-1}\left(x _{N-1},u_{N-1}\right)+g_{N} \tag{3}\]  
Figure 7: State and Control Input Discretization in Time and Space.