## 5 Discussion  
The present paper aims at providing the reader with a comprehensive view of a thorough experimental test for a Q-learning algorithm to be implemented as HEV real-time capable EMS. The experiments conducted in the research activity are hence aimed at providing the reader with a complete procedure for a reliable assessment of the performance obtained by a RL algorithm for HEVs. Two elements have been introduced to evaluate the agent performances. First, the distinction between training and testing episodes. Training episodes are featured by law about the agent exploration rate. On the contrary, the agent exploration strategy is turned off and the greedy action is always selected in testing episodes. Second, the identification of the discounted return during training and testing episodes as the most meaningful signal about the learning process of the RL agent. Contrarily, the sum of the rewards obtained by the agent in the episodes is typically used in the literature. Thanks to the present research, the trend of the discounted return computed in different time steps of training or testing episodes has proved to be a consistent index for the assessment of the RL agent performances. In addition, the evaluation of the response of a Q-learning agent has been discussed considering a massive testing of the algorithm on different real-world driving conditions. Specifically, four different analyses have been conducted and thoroughly presented in this paper which are commonly neglected in the literature. First, the effect of the discount factor  
Figure 19: Comparison of the discounted return evaluated at 2/3 of the testing episodes for L1s and L3s. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**DM**} & \multicolumn{2}{c}{\(SoC_{f}\) (\(\star\))} & \multicolumn{2}{c}{**FC (g)**} & \multicolumn{2}{c}{**rFC (g)**} \\ \cline{2-7}  & \(L1s\) & \(L3s\) & \(L1s\) & \(L3s\) & \(L1s\) & \(L3s\) \\ \hline CLUST1 & 0.6014 & 0.6010 & 267.74 & 286.08 & 267.74 & 286.08 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Results by L1s and L3s exploration strategies on CLUST1. on the decisional process of the agent has been analyzed considering very different discount factors. Starting from the case of a null discount factor, an improved performance of the Q-learning has been appreciated in most of the cases when the discount factor has been set to values toward the unity. Such an outcome has been proved by means of the discounted return trajectory evaluated at three different stages of the episode. Second, a reward shaping analysis has been carried out considering an optimization of the cumulative fuel consumed at the end of the driving mission in presence of a boundary condition about the battery _SoC_ charge sustaining. Such an analysis has been conducted considering different discount factors so as to highlight the non-trivial response of the agent when the training parameters are mixed and tested over different environments. As an outcome, the FC-oriented reward function has proved to be more efficient when higher discount factors are considered with respect to a reward focusing more on the battery _SoC_ CS. This can be addressed to the nature of the HEV control problem. A FC-oriented control strategy can, in fact, fall into unfeasible policies with a high frequency due to the difficulty of maintaining the battery _SoC_ under the admitted _SoC_ window when any information (or information with small relevance) is provided to the agent about the influence of the battery _SoC_ variations on the system. Third, the influence of the learning rate on the performance of the Q-learning agent. Recalling the learning rate to be responsible of the Q-table rate of update, the algorithm has proved to be very sensitive to the variation of its internal learning parameter. In fact, a worsened response of the agent can be obtained when a reduction in the learning rate value is applied.