## 3 An Integrated Modular Software Framework for Hybrid Electric Vehicles
### Simulator  
The pre-transmission parallel HEV architecture reported in Figure 3 has been modeled and employed as a Simulator of the software framework. The layout refers to the driveline of a hybridized front-wheels driven passenger car equipped with a conventional down-sized ICE and a 6-gears transmission. A kinematic backward approach has been used to compute the power requested to the different stages of the powertrain (from wheels to the power components) according to [34; 35; 36]. The final drive (FD) and the torque coupling device (TCD) have been modeled as fixed speed ratios; clutches have been included in the powertrain to allow the HEV for exploiting its macro-operating modes during traction conditions. Specifically, a pure thermal mode can be realized by engaging the clutch C1 while disengaging the clutch C2 so that the ICE can be used as a stand-alone component. A pure electric mode can be realized by engaging clutch C2 while disengaging clutch C2 so that the MG (and the battery) can be used as a stand-alone component. A power-split mode can be realized by engaging both clutches C1 and C2 and demanding power to both the propellers. Finally, a battery charging mode can be obtained by engaging both clutches C1 and C2 while using the MG as generator with the capability of transferring a given share of power to the battery even during traction. Evidently, the ICE is demanded to produce a higher power with respect to the power requested by the road. Efficiencies have been considered for each powertrain component, including FD, multi-gears transmission (TR), torque-coupling device (TCD) and AC/DC converter. Each of these, except TR, has been simply modeled as a fixed efficiency to be paid both during traction and braking phases. About TR, a different efficiency has been considered for each gear. Moreover, experimentally derived two-dimensional look-up tables have been used to estimate the fuel consumption and the motor efficiency of the ICE and the MG, respectively. Finally, the battery performances have been accounted for by a fast-running low-throughput Rint model in which the effect of environmental temperature, battery temperature and aging process have been ignored [29]. For the sake of clarity, moving to a more detailed battery model would have led to a relevant increase in the computational time needed by the software calculations while not changing the focus of the present study. In fact, even with a change in the numerical results obtained by the RL control algorithm, any step would have changed in the approach. Extending the study of the RL performance when coupled with detailed physical models will be considered a future step.  
Figure 2: Integrated modular software framework based on Reinforcement Learning for hybrid electric vehicles.  
The Simulator receives the Agent's physical action \(\widetilde{a_{t}}\) for the current time step (please refer to next section for the definition of physical action) and it generates the new state \(s_{t+1}\). During this process, it also produces the action feasibility mask \(m_{t+1}\), which represents the set of actions allowed to the Agent at each time step \(t\). Assuming a discrete action space, the action feasibility mask can be represented as a boolean vector that excludes from the action space the actions physically impossible from the state just generated. The feasible actions have been represented by 1 whereas unfeasible actions have been represented by 0. The latter could be obtained based on two specific conditions: the power requested by the road is not realizable or the battery _SoC_ window has been exceeded.