## 4 Results  
The design of a RL-based agent for the control of HEVs requires the analysis of a set of meaningful experiments aimed at assessing the change in the performance of the algorithm when consistent changes are applied to its configuration. Therefore, the identification of an exhaustive test plan of the agent has to be carried out. In the present section, the results obtained by a thorough analysis of the Q-learning response when used as control agent for a HEV are analyzed relating to the variation of the most influencing parameters defined in Section 2. The main characteristics of the considered parallel HEV (Figure 3) are listed in Table 1. The conventional driveline has been integrated with a 70 kW MG and a 74 kW-6.1 kWh battery. The peak power of the MG has been considered both for traction and regenerative braking phases since a symmetrical operating map has been involved in the analyses. About the energy storage system, a sizing operation has been performed so that the peak power achieved by the battery could fulfill the power requirements of the MG, both during traction and regenerative braking conditions. The HEV performances have been evaluated on four different experimentally derived DMs [37], identified as CLUST1, CLUST2, CLUST3 and CLUST4. In Figures 4-7, some of the main characteristics of each DM are compared, specifically the vehicle velocity as a function of time, the vehicle as a function of distance traveled, maximum vehicle acceleration and velocity and average vehicle acceleration and velocity. The selection of the DMs has been based upon the necessity of testing the RL control agent for HEVs in a driving environment characterized by a miscellaneous set of driving conditions. As far as the vehicle velocity profiles are concerned, the longest DM among the four is CLUST1, whereas CLUST2, CLUST3 and CLUST4 are progressively shorter. Moreover, CLUST1 is featured by the highest maximum and average velocities. On the contrary, the shortest DM (CLUST4) involves the most demanding average acceleration during traction, as it can be seen from the very spiky velocity trajectory charted in Figure 5. The maximum acceleration during traction is achieved on CLUST3, whereas the less demanding operations are required on CLUST2. So as to provide a fair comparison between the performances of the different tested Q-learning configurations, the results obtained by the DP on the same environment have been considered as benchmark in the case of \(SoC^{*}=0.6\). Moreover, the state and control variables of the DP have been identified based on the observations and the actions of the Q-learning agent (i.e., battery \(SoC\) as observation, power flow and gear number as actions), respectively. As for the battery \(SoC\) discretization, the same mesh has been applied to the DP computational grid as well as to the Q-table of the Q-learning algorithm according to [29]. Specifically for the present case of study, 501 levels have been considered in a battery \(SoC\) window of [0.55-0.65]. As far as the control space is concerned, the set of Q-learning actions has been paired to the set of DP control variables. Specifically, two indexes have been considered for the gear number (GN) and the power flow (PF), resulting in 42 possible actions (6 gears and 7 possible power-splits between ICE and MG). Finally, the objective function of the DP has been set according to the problem formulation of (1), for which the FC has to be minimized while a complete battery charge sustaining (CS) is attained. For the sake of clarity, no optimization of the Q-learning agent for the specific vehicular application is targeted in this study. The results of the Q-learning presented in the following section have hence not been considered as representative of the best performances that could be obtained by an optimized Q-learning version.