# Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning
## 4 Experimental Results and Discussion
### Online Application
#### 4.2.2 Experimental Results  
In the first case, we trained the neural network 50 times under the NEDC driving cycle with the same initial condition. Unlike the offline learning, this process is online and simulates the vehicle running under the NEDC driving cycle. The track of loss is depicted in Figure 8. The loss also decreases quickly along the training process in the online application. Figure 9 depicts the track of total reward and the fuel consumption of one driving cycle along the training process, and the overall trend of the total reward is the same as the offline application. This reveals the proposed DRL-based online learning architecture is effective. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Control Strategy** & **Fuel Consumption (L/100 km)** & **Equivalent Fuel Consumption (L/100 km)** \\ \hline Rule-Based & 3.857 & 3.861 \\ DRL-based & 3.468 & 3.550 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of the results under UDDS. Figure 7: Simulation results of the trained EMS under UDDS. As we can see from Figure 9, the trend of the total reward and the fuel consumption is nearly opposite. This reflects that the definition of the reward is suitable. Simulation results of the online trained DRL-based EMS for the NEDC driving cycle are shown in Figure 10. The comparison of the results are listed in Table 4. Fuel consumption is also improved compared to the rule-based control strategy, as fuel consumption is decreased by 10.29%, while the equivalent fuel consumption is decreased by 2.57%. Figure 8: Track of loss in the online application. Figure 10: Simulation results of the trained EMS under NEDC. Figure 9: Track of the total reward and fuel consumption in the online application. In the second case, the neural network was pre-trained offline under the UDDS driving cycle, such that the DRL-based EMS can adapt to the UDDS driving cycle but have no a priori knowledge about the NEDC driving cycle. The comparison of the results between the offline trained DRL-based EMS under the UDDS driving cycle for the NEDC driving cycle and other control strategies are listed in Table 5. It is obvious that the offline-trained EMS under the UDDS driving cycle does not adapt well to the NEDC driving cycle. Based on the offline pre-trained EMS under the UDDS driving cycle, we can apply the online learning process under the NEDC driving cycle. We trained the pre-trained neural network 20 times under the NEDC driving cycle with the same initial conditions. After the training process, we tested the trained EMS under the NEDC driving cycle. The simulation results are shown in Figure 11. The comparison of the results are listed in Table 6. The results show that the pre-training process can contribute to effectively decrease the online training time. This is because the DRL-based EMS learns some of the same features between different driving conditions. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Control Strategy** & **Fuel Consumption (L/100 km)** & **Equivalent Fuel Consumption (L/100 km)** \\ \hline DRL-based EMS trained under NEDC online & 3.478 & 3.792 \\ DRL-based EMS only pre-trained under UDDS offline & 3.690 & 3.872 \\ DRL-based EMS which pre-trained offline and trained & 3.440 & 3.795 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparison of the results under NEDC. Figure 11: Simulation results under NEDC of the online trained EMS which was pre-trained under UDDS. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Control Strategy** & **Fuel Consumption (L/100 km)** & **Equivalent Fuel Consumption (L/100 km)** \\ \hline Rule-Based & 3.877 & 3.892 \\ DRL-based EMS trained under NEDC online & 3.478 & 3.792 \\ DRL-based EMS only pre-trained under UDDS offline & 3.690 & 3.872 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparison of the results under NEDC. Figure 12 depicts the track of the total reward and the fuel consumption of one driving cycle along the training process, the curves are smoother than the curves without pre-training.