SidraBhatti/sasac-shev-engine-control

SA-SAC: Sequence-Aware Soft Actor-Critic for Series Hybrid Electric Vehicle Engine Control

A reinforcement learning framework that extends Soft Actor-Critic (SAC) with sequence-aware architectures — Gated Recurrent Units (GRU) and Decision Transformers (DT) — for engine control in Class-8 series hybrid electric trucks (SHEV), reformulating engine control as a sequential decision-making problem to capture temporal dependencies in driving patterns.

Role & Attribution

Sidra Ghayour Bhatti — supervisory/co-advising role, alongside Qadeer Ahmed (PI, OSU Center for Automotive Research). Lead author and primary implementation: Wafeeq Jaleel, with Md Ragib Rownak and Athar Hanif as co-developers/collaborators.

Fig. 2: SA-SAC architecture — the SAC engine controller (actor/critic networks, swappable between DNN/GRU/DT) interacts with the SHEV powertrain model (engine, generator, electric machine, battery, rear diff), storing transitions in a sequential replay buffer and training via normalized reward, soft target updates, and auto-tuned entropy.

Motivation

Adaptive energy management is critical for heavy-duty hybrid trucks to reduce fuel consumption while maintaining battery charge over long operating durations. Existing RL-based HEV controllers typically use feed-forward networks that ignore the temporal dependencies inherent in driving patterns. This work is among the first to incorporate sequence-aware architectures (GRU, DT) into both the actor and critic of SAC for HEV energy management.

Method

State, action, reward. State: battery SOC, distance traveled, and electric-machine power demand. Action: engine speed and torque (the engine is mechanically decoupled from the wheels in a series HEV, so SAC is free to learn the best operating point independent of wheel speed). Reward: penalizes fuel consumption scaled by (initial SOC)², combined with a piecewise SOC-shaping term that heavily penalizes the final SOC falling outside a 15–18% target band and moderately penalizes it outside 15–85% overall.

Three actor-critic variants compared:

• FFN (baseline) — standard feed-forward memoryless networks
• SAC-GRU — GRU-based actor and critic, hidden state reset each episode, capturing short-to-mid-term temporal patterns
• SAC-DT — Decision Transformer actor (return-conditioned trajectory modeling with causal attention) paired with a GRU critic (found to perform best as critic; the DT critic underperformed in the ablation)

Training. A sequential replay buffer stores trajectories (not independent transitions) for the recurrent/transformer variants; sequence length k is sampled per batch. 10 HFET training cycles (130 minutes total). Built on the CleanRL continuous-SAC implementation, with the DT implementation adapted from the original Decision Transformer codebase.

Results

Ablation study — six studies isolating the effect of sampling strategy, architecture choice, input sequence length, and robustness to varying initial SOC / cycle duration / power demand:

Fig. 3: Across all six studies, GRU- and DT-based agents converge faster and generalize better than FFN under varying operating conditions, though FFN converges fastest on a single fixed condition. Longer input sequences (k=100) help DT; shorter sequences (k=10) work better for GRU.

Validation against Dynamic Programming, using the best-performing agent from each architecture family, tested on the HFET training cycle plus two unseen cycles (US06 aggressive-driving, HHDDT heavy-truck cruise):

Cycle	Agent	SoC_f (%)	MPG	Δ MPG vs. DP
HFET	DP (reference)	15.55	23.71	—
HFET	FFN	15.81	20.73	−12.57%
HFET	GRU	15.10	21.07	−11.14%
HFET	DT	15.38	21.68	−8.54%
US06	DP (reference)	16.44	4.63	—
US06	FFN	14.67	4.27	−7.72%
US06	GRU	15.63	4.43	−4.24%
US06	DT	17.58	4.042	−12.69%
HHDDT	DP (reference)	17.29	18.82	—
HHDDT	FFN	5.11	13.81	−13.81%
HHDDT	GRU	15.59	15.8	−12.8%
HHDDT	DT	15.23	20.75	−4.93%

On the HFET training cycle, DT-GRU (DT actor, GRU critic) came within 1.8% of Dynamic Programming in fuel savings, vs. 3.16% for GRU-GRU and 3.43% for the FFN baseline. On unseen cycles (US06, HHDDT), sequence-aware agents (GRU, DT) consistently outperformed the FFN baseline in generalization, though DT's engine speed/torque outputs showed more fluctuation — noise the paper flags as needing further tuning before real-world deployment.

Fig. 4: SOC trajectories across DP, FFN (DNN), GRU, and DT agents on the HFET training cycle — all four track the DP reference closely, with GRU showing a slight high-SOC bias late in the cycle.

All architectures achieve inference times under 5 ms, meeting real-time control requirements. All three sequence-aware/baseline agents were validated using a high-fidelity MATLAB/Simulink SHEV forward simulator, not simulation-only.

Vehicle specifications

Class-8 series HEV: 36,287 kg curb weight; engine 270 kW / 2300 rpm max, 1500 Nm / 1120–1480 rpm max torque; generator 240 kW max; electric machine 400 kW / 3500 Nm max; NMC battery, 323.94 kWh / 4.85 Ah, 160S/115P.

Data availability

The GitHub repo includes drive-cycle data and validation results. Proprietary engine/generator maps and vehicle parameters are excluded — equivalent data is required to train new agents from scratch.

Published: Jaleel, W., Rownak, M.R., Hanif, A., Bhatti, S.G., & Ahmed, Q. (2026). "Sequence Aware SAC Control for Engine Fuel Consumption Optimization in Electrified Powertrain." 2026 American Control Conference (ACC). Preprint: arXiv:2508.04874 Code: github.com/wafeeqaj/sasac-shev-v1

Citation

@inproceedings{jaleel2026sasac,
  author    = {Jaleel, Wafeeq and Rownak, Md Ragib and Hanif, Athar and Bhatti, Sidra Ghayour and Ahmed, Qadeer},
  title     = {Sequence Aware SAC Control for Engine Fuel Consumption Optimization in Electrified Powertrain},
  booktitle = {2026 American Control Conference (ACC)},
  year      = {2026},
  eprint    = {2508.04874},
  archivePrefix = {arXiv}
}