A Robust Fuel Optimization Strategy For Hybrid Electric Vehicles: A Deep Reinforcement Learning Based Continuous Time Design Approach
Authors: Nilanjan Mukherjee
Sudeshna Sarkar
Venue:  (2021)
Keywords: Deep Reinforcement Learning
Optimal Control
Fuel Management System
Hybrid Electric vehicles
H∞ Performance Index

Abstract: This paper deals with the fuel optimization problem for hybrid electric vehicles in reinforcement learning framework. Firstly, considering the hybrid electric vehicle as a completely observable non-linear system with uncertain dynamics, we solve an open-loop  deterministic optimization problem. This is followed by the design of a  deep reinforcement learning based optimal controller for the non-linear system using concurrent learning based system identifier such that the actual states and the control policy are able to track the optimal trajectory and optimal policy, autonomously even in the presence of external disturbances, modeling errors, uncertainties and noise and signigicantly reducing the computational complexity at the same time, which is in sharp contrast to the conventional methods like PID and Model Predictive Control (MPC) as well as traditional RL approaches like ADP, DDP and DQN that mostly depend on a set of pre-defined rules and provide sub-optimal solutions under similar conditions. The low value of the H-infinity ($H_{\infty})$ performance index of the proposed optimization algorithm addresses the robustness issue. The optimization technique thus proposed is compared with the traditional fuel optimization strategies for hybrid electric vehicles to illustate the efficacy of the proposed method.

Under review as a conference paper at ICLR 2021
A ROBUST FUEL OPTIMIZATION STRATEGY FOR HY- BRID ELECTRIC VEHICLES: A DEEP REINFORCEMENT LEARNING BASED CONTINUOUS TIME DESIGN AP- PROACH
Anonymous authors Paper under double-blind review
ABSTRACT
This paper deals with the fuel optimization problem for hybrid electric vehicles in reinforcement learning framework. Firstly, considering the hybrid electric vehicle completely observable non-linear system with uncertain dynamics, an open-loop deterministic optimization problem to determine a nominal optimal state. This is followed by the design of a deep reinforcement learning based op- timal controller for the non-linear system using concurrent learning based system identifier such that the actual states and the control policy are able to track the optimal state and optimal policy, autonomously even in the presence of external disturbances, modeling errors, uncertainties and noise and signigicantly reducing the computational complexity at the same time, which is in sharp contrast to the conventional methods like PID and Model Predictive Control (MPC) as well as set
solve
traditional RL approaches like ADP, DDP and DQN that mostly depend pre-defined rules and provide sub-optimal solutions under similar conditions. The low value of the H-infinity (H²) performance index of the proposed opti- mization algorithm addresses the robustness issue. The optimization technique thus proposed is compared with the traditional fuel optimization strategies for hy- brid electric vehicles to illustate the efficacy of the proposed method.
INTRODUCTION
Hybrid electric vehicles powered by fuel cells and batteries have attracted great enthusiasm in mod- em days as they have the potential to eliminate emissions from the transport sector. Now, both the fuel cells and batteries have got several operational challenges which make the separate use of each of them in automotive systems quite impractical. HEV's and PHEVs powered by conventional diesel engines and batteries merely reduce the emissions, but cannot eliminate completely. Some of the drawbacks include carbon emission causing environmental pollution from fuel cells and long charging times, limited driving distance per charge, non-availability of charging stations along the driving distance for the batteries. Fuel Cell powered Hybrid Electric Vehicles (FCHEVs) powered by fuel cells and batteries offer emission-free operation while overcoming the limitations of driv- ing distance per charge and long charging times. So, FCHEVs have gained significant attention in recent years. As we find, most of the existing research which studied and developed several types of Fuel and Energy Management Systems (FEMS) for transport applications include Sulaiman al. (2018) who has presented a critical review of different energy and fuel management strategies for FCHEVs. Li et al. (2017) has presented an extensive review of FMS objectives and strategies for FCHEVs. These strategies, however can be divided into two groups, i.c., model-based and model- free. The model-based methods mostly depend on the discretization of the state space and therefore suffers from the inherent curse of dimensionality. The coumputational complexity increases in an exponential fashion with the increase in the dimension of the state space. This is quite evident the methods like state-based EMS (Jan et al., 2014; Zadeh et al., 2014; 2016), rule-based fuzzy logic strategy (Motapon et al., 2014), classical Pl and PID strategies (Segura et al., 2012), Potryagin's 's minimum principle (PMP) (Zheng et al., 2013; 2014), model predictive control (MPC) (Kim et al., 2007: Torreglosa et al., 2014) and differential dynamic programming (DDP) (Kim et al., 2007). Out of all these methods, differential dynamic programming is considered to be computationally quite

---

Under review as a conference paper at ICLR 2021
efficient which rely the linearization of the non-linear system equations about nominal state trajectory followed by a policy iteration to improve the policy. In this approach, the control policy for fuel optimization is used to compute the optimal trajectory and the policy is updated until the convergence is achieved.
The model-free methods mostly deal with the Adaptive Dynamic Programming (Bithmead et al., 1991; Zhong et al., 2014) and Reinforcement Learning (RL) based strategies (Mitrovic et al., 2010; Khan al., 2012) icluding DDP (Mayne al., 1970). Here, they tend to compute the control pol- icy for fuel optimization by continous engagement with the environment and measuring the system response thus enabling it to achieve at a solution of the DP equation recursively in an online fash- ion. In deep reinforcement learning, multi-layer neural networks are used to represent the learning function using a non-linear parameterized approximation form. Although a compact paremeterized form do exist for the learning function, the inability to know it apriori renders the method suffer from the curse of dimensionality (O(d)) where, d is the dimension of the state space), thus making it infeasible to apply to high-dimemsional fuel managememt system.
The problem of computational complexity of the traditional RL methods like policy iteration (PI) and value iteration (VI) (Bellman et al., 1954; 2003; Barto et al., 1983; Bartsekas, 2007) can be overcome by a simulation based approach (Sutton et al., 1998) where the policy or the value function can be parameterized with sufficient accuracy using small number of parameters. Thus, we will be able to transform the optimal control problem to an approximation problem in the parameter space (Bartesekas al., 1996; Tsitsiklis et al., 2003; Konda et al., 2004) side stepping the need for model knowledge and excessive computations, However, the convergence requires sufficient exploration of the state-action space and the optimality of the obtained policy depends primarily on the accuracy of the parameterization scheme.
As a result, a good approximation of the value function is of utmost importance the stability of the closed-loop system and it requires convergence of the unknown parameters to their optimal values. Hence, this sufficient exploration condition manifests itself as a persistence of excitation (PE) condition when RI. is implemented online (Mehta et al., 2009; Bhasin et al., 2013; Vrabie, 2010) which impossible to be guaranteed a priori.
Most of the traditional approaches for fuel optimization are unable to adrress the robustness issue. The methods described the literature including those of PID (Segura et 20,, Model Predic- tive Control (MPC) (Kim et 21.:007:: et al., 2014) and Adaptive Dynamic Programming (Bithmead et et al., 1996; Tsitsiklis et al., 2003; Konda et al., 2004) suffer from the drawback of providing sub- optimal solution in the presence of external disturbances and noise. As a result, application of these methods for fuel optimization for hybrid electric vehicles that are plagued by various disturbances in the form of sudden charge and fuel depletion, change in the environment and in the values of the parameters like remaining useful life, internal resistance, voltage and temperature of the battery, are quite impractical.
et al., 2014) as well as the simulation based RL strategies (Bartesekas
The fuel optimization problem for the hybrid electric vehicle therefore have been formulated fully observed stochastic Markov Decision Process (MDP). Instead of using Trajectory-optimizze LQG (T-LQG) or Model Predictive Control (MPC) provide a sub-optimal solution in the presence of disturbances and noice, we propose a deep reinforcement learning- baaning-based optimization strategy seerrivaivv-aatio--eeeeee
tuples to present a robust optimal solution. The convergence of the weight estimates of the policy and the value function to their optimal values justifies our claim. The two major contributions of the proposed approch can be therefore be summarized as follows:
using concurrent learning (CL) that uses the
1) The popular methods RL literature including policy iteration and value iteration suffers from the curse of dimensionality owing to the use of simulation based technique which requires sufficient exploration of the state space (PE condition). Therefore, the proposed model-based RL scheme aims relax the PE condition by using a concurrent learning (CL)-based system identifier to reduce the computational complexity. Generally, an estimate of the true controller designed using the CL- based method introduces an approximate estimation error which makes the stability analysis of the system quite intractable. The proposed method, however, has been able to establish the stability of the closed-loop system by introducing the estimation error and analyzing the augmented system trajectory obtained under the influnece of the control signal.
2

---

Under review as a conference paper at ICLR 2021
2) The proposed optimization algorithm implemented for fuel management in hybrid electric vehi- cles will nullify the limitations of the conventional fuel management approaches (PID, Model Pre- dictive Control, ECMS, PMP) and traditional RL approaches (Adaptive Dynamic Proagramming, DDP, DQN), all of which suffers from the problem sub-optimal behaviour in the presence of ex- ternal disturbances, model-uncertainties, frequent charging and discharging. change of enviroment and other noises. The H-infinity (o) performance index defined as the ratio of the disturbance to the control energy has been established for the RL based optimization technique and compared with the traditional strategies to address the robustness issue of the proposed design scheme.
The rest of the paper is organised as follows: Section 2 presents the problem formulation including the open-loop optimization and reinforcement learning-based optimal controller design which have been described in subsections 2.1 and 2.2 respectively. The parametric system identification and value function approximation have been detailed in subsections 2.2.1 and 2.2.2. This is followed by the stability and robustness analysis (using the H-infinity (Hoo) performance index ) of the closed loop system in subsection 2.2.4. Section 3 provides the simulation results and discussion followed by the conclusion in Section 4.
2 PROBLEM FORMULATION
Considering the fuel management system of a hybrid electric vehicle as continous time affine non-linear dynamical system:
i = f(x,w) + g(x)u, y = h(r,v)
where, R"= YE U R" the state, output and the control vectors respectively, f(.)
denotes the drift dynamics and g(-) denotes the control effectivenss matrix. The functions f and assumed to be locally Lipschitz continuous functions such that (()== 0 and Vf(x) is continous for every bounded.x x E TR==, The process noise w and measurement noise are assumed to be zero-mean, uncorrelated Gausssian white noise with covariances W and V, respectively.
Assumption 1: We consider the system to be fully observed:
h(x, v) = x
This assumption is considered provide a tractable formulation of the fuel management problem to side step the need for a complex treatment which is required when a stochastic control problem is treated as partially observed MDP (POMDP).
Remark 1:
Optimal Control Problem: For continous time system with unknown nonlinear dynamics f(.). we need to find an optimal control policy ne in a finite time horizon [0, t] where n. is the control policy at
time such that me u(t) to minimize the cost function given by J f (x" (x"Qr= Qx - uRu" dt + x= Fr
where, Q.F > 0 and R 2 0.
2.1 OPEN LOOP OPTIMIZATION
Considering a noise-free non-linear stochastic dynamical system with unknown dynamies:
+ f(x, 0) g(x)),, h(x, v)
(3)
where, xo E "= E RR", R" are the initial state, output and the control vectors respec- tively, J(.) have their usual meanings and the corresponding cost function is given by Ja (xo, u,)
'' (wTQr+ +Qr+uulhh uRu" dt =f
Remark: We have used piecewise convex function to approximate the non-convex fuel function globally which has been used to formulate the cost function for the fuel optimization.
3

---

Under review as a conference paper at ICLR 2021
The open loop optimization problem is to find the control sequence u, such that for given initial state Xg,
d arg min Ja( ur),
subject to å f(x 0) g(z) hxx u) = x.
(4)
The problem is solved using the gradient descent approach (Bryson et al., 1962; Gosavi et al., 2003), and the procedure is illustrated as follows: Starting from a random initial value of the control sequence ( = [I, the control policy updated iteratively
U(++1) - Ja U()),,
until the convergence is achieved upto certain degree of accuracy where U(w) denotes the control value at the nh iteration and is the step size parameter. The gradient vector is given by:
Ja Und))= OJa aJa ŌJa
OJd
Oun' Oun Ouz"
Ou.
The Gradient Descent Algorithm showing the approach has been detailed in the Appendix A.1.
Remark 2:
The open loop optimization problem is thus solved using the gradient descent approach considering a black-box model of the underlying system dynamics using a sequence of input-output tests without having the perfect knowlegde about the non non-lincarities in the model at the time of the design. This method proves to be very simple and useful strategy for implementation in case of complex dynamical systems with complicated cost-to-go functions and suitable for parallelization.
2.2 REINFOR LEARNING BASED OPTIMAL CONTROLLER DESIGN
Considering the affine non-linear dynamical system given by equation (1), our objective is to design
control law to track the optimal time-varying trajectory x(t) R A novel cost function is formulated in terms of the tracking error defined by e = z(t) =(t) and the control error defined by the difference between the actual control signal and the desired optimal control signal. This formulation helps to overcome the challenge of the infinte cost posed by the cost function when defined in terms of the tarcking error e(t) and the actual control signal signal u(t) only (Zhang et al., 2011; I; Kamalapurkar et al., 2015). The following assumptions is made to determine the desired steady state control.
Assumption 2: (Kamalapurkar et al., 2015) The function g(x) in equation (1) is bounded, the matrix g(x) has full column rank for all (t) E "R and the function g+ R" + mm which is defined as g = (9g g) is bounded and locally Lipschitz.
A Assumption 3: (Kamalapurkar et al., 2015) The optimal trajectory is bounded by a known positive constant b IR such that ||क||| < b and there exists a locally Lipschitz function ha such that á hag (i) and g()) ()) g (z)(ha(z) f(ø)) = hā( f()).
Using the Assumption 2 and Assumption 3, the control signal ua required to track the desired trajec- tory I(t) is given as ua(x) = g (ha(z) - fa) where fa = f()) and g g (Ø). The control error given by μ = u(t) u (.). The system dynamics can now be expressed
(=F(C) + G(())
where, the merged state <(t) E R is given by ((t) = jeT, .TTTT and the functions F({) and G(C) "
are defined as F(C) = If (e + r) - h + u (3)g" (e (e +), and G(C) = [g" (e + I). xmx where, 0mx, denotes a matrix of zeroes. The control error j is treated hereafter as the design variable. The control objective is to solve finite-horizon optimal tracking problem online, i.e.,
to design a control signal a that will minimize the cost-to-go function, while tracking the desired
trajectory, given by J(C, = r(((T), where, the local cost r IR2n I.m + R given as r(G.7) = Q(e)+ + RRu, RJ,, R c mm is a positive definite symmetric matrix and Q R + R is continous positive definite function.
Based on the assumption of the existence of an optimal policy, it can be character- ized in terms of the value function 4 22n
R which is defined V*(
4

---

Under review as a conference paper at ICLR 2021
min ((+)))) I
where U E R is the action space and o" (t; to: So) the trajectory of the system defined by equation (10) with the control effort μt Ry + R with the initial condition So E IR" and the initial time to E Ryo- Taking into consideration that an op- timal policy exists and that V* is continously differentiable everywhere, the closed-form solution
a()) This satisfies dr
(Kirk, 2004) is given as p² (() -1/2 R. (C)V where, Ve(.) the Commilton-Jacbble Jacbbi-Belblaaam (HJB) equation (Kirk, 2004) given as
(CV + ((C))((()) + Q(() + n² (C) (S) =0
(8)
where, the initial condition V* = 0, and the funtion Q I2 T IR is defined as Q(["", (lle",""'') Q(e) where, (e(t), +()) E I=
Since, a closed-form solution of the HJB equation is generally infeasible to obtain, we sought an approximate solution. Therefore, an actor-critic based method is used obtain the parametric estimates of the optimal value function and the optimal policy which are given as VG, Wa) and A(S Wa) where, We E IR' and Wa E IR' define the vector paramater estimates. The task of the actor and critic is to learn the corresponding parameters. Replacing the estimates V andd jú for V and A in the HJB equation, we obtain the residual error. also known as the Bell Error (BE) as 8(S, W W.) = Q() + ATT Rii(s W.) + V. (S. (F)) (Ç,) W.) where, 5: R22 X R. X XR' R. The solution of the problem requires the actor and the critic to find set of parameters W. and W. respectively such that ō(C. W.. Wa) = O and (Ç, Wa 1/2 ( where, VÇ E IR". As the exact basis fucntion for the approximation is not known apriori, we seek to find set of approximate parameters that minimizes the BE. However, an uniform approximation of the value function and the optimal control policy over the entire operating domain requires to find parameters that will able to minimize the error E. : R² X R + E defined as E. Wa) supeelo (16, We, W) thus, making it necessary to have an exact knowledge of the system model. Two of the most popular methods used to render the design of the control strategy robust to system uncertainties in this context are integral RL (Lewis et al., 2012: Modares al., 2014) and state derivative estimation (Bhasin et al., 2013; Kamalapurkar et al., 2014). Both of these methods suf- fer from the persistence of exitation(PE) condition that requires the state trajectory (t;;; lo, Co) to cover the entire operating domain for the convergence of the parameters to their optimal values. We have relaxed this condition where the integral technique is used in augmentation with the replay of the experience where every evaluation of the BE is intuitively formalized as a gained experience, and these experiences are kept in history stack so that they can be iteratively used by the learning algorithm to improve data efficiency.
Therefore, to relax the PE condition, the we have developed a CL-based system identifier which is used to model the parametric estimate of the system drift dynamics and is used to simulate the experience by extrapolating the Bell Error (BE) over the unexplored territory in the operating domain thereby, prompting an exponential convergence of the parameters to their optimal values.
2.2.1 PARAMETRIC SYSTEM IDENTIFICATION
Defined by any compact set C C R, the function f can be defined using a neural network (NN) as f(x) = 8T a 21) + cz(z) where, x1 = [1, x 2 E R=+1, 0 E m and Y E R=+1Xp indicates the constant unknown output-layer and hidden-layer NN weight, oy Rp + RR++1 denotes bounded NN activation function, <g: RR. + is the function reconstruction error, p N denotes the number of NN neurons. Using the universal functionional approximation property of single layer NNs, given constant matrix Y such that the rows of or o x1) form proper basis, there exist constant ideal weights 0 and known constants 0, Eg, e E I such that ||0|| < 0 < oo, supart [|<g())]| < &g. supzeC ||
< éø where, |I-I| denotes the Euclidean norm for vectors and the Frobenius norm for matrix (Lewis et al., 1998).
Taking into consideration approximated by the function ff 2 X R - R which is defined as f(ç.0) 0T aC((), where ag 2n - [R++1 can be defined as (C) = [Y + ITJT). An estimator for online identification of the drift dynamics is developed
estimate 0 E IX of the weight matrix 0 the function f can be
à 0T (o({) + g(x))())))) ) + kö
5

---

Under review as a conference paper at ICLR 2021
where, E = x 2 and k c R R is positive constant learning gain.
Assumption 4: A history stack containing recorded state-action pairs fay, uz / along with uz)
numerically computed state derivatives { Al that satisfies Amin (23=) [130 = da >
0, ||ă - اارl < d, Vj is available a priori, where 0s os (Y" [1, [1, "ff" E R is a known positive constant, 3 = f (1zz) + g (mrz My and Amiu (.) denotes the minimum eigenvalue.
The weight estimates are updated using the following CL based update law:
M
= T (1 He + KoTo Σ (ā) - gj - 031))) T
(10)
where ka E IR is constant positive CL gain, and I' E [pttlxp1 is a constant, diagonal, and positive definite adaptation gain matrix. Using the identifier, the BE in (3) can approximated as
(5. 0, We, W.) =Q(() A²T (s.W)) Rí (s. + +V.V (s, (Fo(S.0) + Fi(() - G(S) (s.
(11)
The BE is now approximated as
Wo. +VG (c. (.) (Fo(S,0) 0) + +F((() + G(C) (W
8 (5. W.. W.) = Q(()) + 1²" (s. Rp (5. W.
(12)
0T (Too(() g(x)g* (xa) θT 00 ([
Onx1 Id
1) and Fi(()
In equation (12), Fo(S.
0nxl
(-ha + g (e + ma) g+ (ma) ha) nz]".
2.2.2 VALUE FUNCTION APPROXIMATION
As V and _r' are functions of the state Ç, the optimization problem as defined in Section 2.2 is quite an intractable one, so the optimal value function is now represented as C C R2n using NN as V²(()= = WT (())+((( <()), where W E RR denotes vector of unknown NN weights, indicates bounded NN activation function, c 2n -> IK deñines the function reconstruction error, and L E N denotes the number of NN neurons. Considering the universal function approximation property of single layer NNs, for any compact set C 2R there exist constant ideal weights W and known positive constants W, ट, and e' E R such that |[ < W co supcec [|<(C)] ε, and supçee /<<(()) é' (Lewis al., 1998).
1R - R_
A NN representation of the optimal policy is obtained as
(c)W+ 11' (²(() = R (c) (VeoF(C)W + VCeF(C)
(13)
Taking the estimates W. and W. for the ideal weights W, the optimal value function and the optimal policy are approximated as V (c. = WT (C), á (c, W.)
- (c) (C)W.. The optimal control problem is therefore recast to find set of weights W. and W. online to minimize the error Éd [ W.) = supçex 8 @5 W., W.) for a given 6, while simultaneously improving i using the Cl.-based update law and ensuring stability of the system using the control law
W.)
u u= á (c, W.)) + (a((s
(14)
where, āa(C.0) = gā (ha (ha-0" 0t dea) 00ooa)
and aod = oo (1 01xn 2x 17) (I 0 (1
of 1").
The error between Ud and ås is included in the stability analysis based the fact that the error trajectories generated by the system é = f(x) (((z)u- ta under the controller in (14) are identical to the error trajectories generated by the system Ç F(C) + G(C) under the control law = a C, W.) + -t 0" and + g ead, where ead 2 (xa).

---

Under review as a conference paper at ICLR 2021
2.2.3 EXPERIENCE SIMULATION
The simulation of experience is implemented by minimizing squared sum of BEs over finitely many points in the state space domain as the calculation of the extremum (supremum) in Eg is not tractable. The details of the analysis has been explained in Appendix A.2 which facilitates the aforementioned approximation.
2.2.4 STABILITY AND ROBUSTNESS ANALYSIS
To perform the stability analysis, we take the non-autonomous form of the value function (Kamalapurkar et al., 2015) defined by Vi* :R" X + R which is defined as (e, t) =
V* (le". x xf]( Ve RR, E R, is positive definite and decrescent. Now, V (0,t) = 0, Vt
R and there exist class K. functions v : IR I and ú R R such that e(Ilell) < (e,t) S ([[e]]), for all e E R" and for all t E R. We take augemented state given as Z E R2n+2l=n(0+1))
is defined as
Z = eT. WT. WT, S7, (vec(0))))
(15)
and a candidate Lyapunov function defined as
(2, = (e W +
W a 2
x x+ + 2 (0"r²'0)
(16)
2
where, vec (-) denotes the vectorization operator. From the weight update in Appendix A.2 we get positive constants 7. 7 E I such that 7 S
|t) S 7, Vt E R. Taking the bounds on I'
and V and the fact that tr ( 0) (vc (T @ Ip+z) (vec(()) the candidate Lyapunov
function be bounded as
uy (I) < (2(21)) S 6: Z1)
(17)
for all Z E R2u+2l+a((tt) and for all t E R. where यी R - R and गे : R are class IC functions. Now. Using (1) and the fact that Vú (e(t),t) (C(tt), Vt R, the time-derivative of the candidate Lyapunov function is given by
Vr. = VeV* (F + Gpd) - WTT W. -w. 2 W TT
2
(18)
W.W. 1/ + Vp Gp
Under sufficient gain conditions (Kamalapurkar et al., 2014), using (9), (10)-(13), and the update laws given by W., I' and Wr, the time-derivative of the candidate Lyapunov function can be bounded as
(19) where is a positive constant, and x C +2=++++((+++ is a compact set. Considering (13) and (15), the theorem 4.18 in (Khalil., 2002) can be used to establish that every trajectory Z(t) satisfying
V - oz (|) 2 of (L), VZ E
I|Z (to)l I शा (p)( where p is a positive constant, is bounded for all t E R and satisfies lim s || [t) S Di (v, (e, ())).. This aforementioned analysis addresses the stability issue of the closed loop system.
The robustness criterion requires the algorithm to satisfy the following inequality (Gao et al., 2014) in the presence of external disturbances with a pre-specified performance index y known as the H-infinity (Ho)) performance index, given by
S. (1(t)))P < 7² S: [Iu(t)]dtt
(20)
where, y(r) is the output of the system, w(1) is the factor that accounts for the modeling errors, parameter uncertainties and external disturbances and y is the ratio of the output energy to the disturbance in the system.
Using (1) and the fact that V₁* = (C(()) Vt E R, the time-derivative of the candidate Lyapunov function is given by
Vz. (F+* - W.
-
c
(21)
-W. + i0 + Vp -VV-
7

---

Under review as a conference paper at ICLR 2021
Gao et al. (2014) has shown if (22) and (23) is satisfied, then it can written that
0 < VL(TT) = J. VL()) S I. V (()((())) + +2² S/" w (t)w((t)dt
(22)
Thus, the performance inequality constraint given by So (Iv(()) <² So [[ in terms of satisfied.
3 SIMULATION RESULTS AND DISCUSSION
Here, we are going to present the simulation results to demonstrate the performance of the proposed method with the fuel management system of the hybrid electric vehicle. The proposed concurrent learning based RL optimization architecture has been shown in the Figure 1.
Action
Dalasrrmm identifier
History Stack
State
Simulated
Environment
triplets
Reward
Stare
Action
fI
Actor
Critis
Action
Figure 1: Reinforcement Learning- based Optimization Architecture
In this architecture, the simulated cene--actioo-dervvvni triplets performs the action of concurrent learning to approximate the value function weight estimates to minimize the bell eтror (BE). The history stack used to store the evaluation of the bell error which carried out by dynamic system identifier as gained experience so that it can iteratively used to reduce the computational burden.
A simple two dimensional model of the fuel management system is being considered for the simula- tion purpose to provide genralized solution that can be extended in other cases of high dimensional system.
We consider a two dimensional non-linear model given by
21 0
0
0
b
cos(220 + 2) ] w(t) sin
0
=
x22 - (cos(2x1 + 2)²))
E1
d
(23)
where a, c, d R are unknown positive parameters whose values selected 1- 1, b b=
1, c = -0.5, -0.5, Iı and 22 are the two states of the hybrid electric vehicle given by the charge present in the battery and the amount of fuel in the car respectively and w(t) sin(t) is sinusoidal disturbance that is used to model the external disturbance function. The control
objective is to minimize the cost function given by J(S, f r(((T), where, the local
cost r 2 R - IR is given as r(C.7) Q(e) + 1== RH,, mm is positive definite symmetric matrix and Q I + IR is a continous positive definite function, while following the desired trajectory We chhose Q 1222 and R 1. The optimal value function and optimal
control for the system (15) are V*(x)
2 x x2 and u² (x) = ccos(2(x1) + 2) The basis
8

---

Under review as a conference paper at ICLR 2021
function 0 IR² RR3 for value function approximation is = [2î, zfa22 22). The ideal weights W =0.55, 0, 1]. The initial value of the policy and the value function weight estimates are W. W, = [1. 11,1,1|" least square gain is [(0) 100/3x3 and that of the system states are z(0) [-1,- -1]². The state estimates å and f are initialized to O and respectively while the history stack for the CL is updated online. Here, Figure 2 and Figure 3 shows the state trajectories obtained by the
022
x7
x2
X9
-04
M0I 0-
20
-7.5
-0.5
Enstttional bucc Coabol
-12
Optirn centri Tegeetary
CL-Joued Contool Lapel
10
20
30
40
0
2
10
20
30
40
Thane (9)
Zume (5)
Figure State Trajectories
Figure 3: State Trajectories
Figure 4: Control Input
0.4
Ry-7
We.2
iFa2
0.35
Yolur Farction Heiglts
Putty Fametton Frgghtrr
Magnitude
0.3
Performance Indest by RI-based method (ADP)
0.2
0.1
Purformance Index based method
12
16
10
0.00
Flace (3)
Time (S)
Figure 5: Value Function
Figure 6: Policy Function
Figure 7: Performance Index
traditional RL methods and that obtained by the CL-based RI. optimization technique respectively in the presence of disturbances. It can be stated that settling time of trajectories obtained by the proposed method is significantly less (almost 40 percent) as compared with that of the conventional RL strategies thus justifying the uniqueness the method and causing saving in fuel consumption by about 40-45 percent. Figure 4 shows the corresponding control inputs whereas Figure 5 and Figure 6 indicates the convergence of the NN weight functions to their optimal values. The Hoc performance index in Figure 7 shows value of 0.3 for the CL-based RL method in comparison 0.45 for the traditional RL-based control design which clearly establishes the robustness of our proposed design.
4 CONCLUSION
In this paper, we have proposed robust concurrent learning based deep optimization strategy for hybrid electric vehicles. The uniqueness of this method lies in use of a concurrent learning based RL optimization strategy that reduces the computational complexity significanty comparison to the traditional RL. approaches used for the fuel management system mentioned in the literature. Also, the use the the H-infinity (Hoo) performance index in case of RL optimization for the first time takes care of the robustness problems that most the fuel optimization nethods suffer from. The simulation results validate the efficacy of the method over the conventional PID, MPC as well as traditional RL based optimization techniques. Future work will generalize the approach for large- scale partially observed uncertain systems and it will also incorporate the movement of neighbouring RI. agents.

---

Under review as a conference paper at ICLR 2021
REFERENCES
R. Akrour, A. Abdolmalcki, H. Abdulsamad, and G. Neumann. Model Free Trajectory Optimization for Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML), 2016.
A. Barto, R. Sutton, and C. Anderson. Neuron-like adaptive elements that can solve difficult learning control problems. IEEE Transaction on Systems, Man, and Cybernetics, 13: 334-846, 1983.
R. Bellman. The theory of dynamic programming. DTIC Document, Technical Representations. 1954.
D. Bertsekas and J. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996.
D. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, 2007.
S. Bhasin, R. Kamalapurkar, M. Johnson, K. Vamvoudakis, F. L. Lewis, and W. Dixon. A novel octor-critie-ideniffee architecture for approximate optimal control of uncertain nonlinear systems. Automatica, 49: 89-92, 2013.
R. Bithmead, V. Wertz, and M. Gerers. Adaptive Optimal Control: The Thinking Man's G.P.C. Prentice Hall Professional Technical Reference, 1991.
A. Bryson and H. Y.-C. Applied Optimal Control: Optimization, Estimation and Control. Washing- ton: Hemisphere Publication Corporation, 1975.
G. V. Chowdhary and F. N. Johnson. Theory and flight-test validation of a concurrent-learning adaptive controller. Journal of Guidance Control and Dynamics,34: 592-607, 2011.
G. Chowdhary, T. Yucelen, M. Mühlegg, and N. Johnson. Concurrent learning adaptive control of linear systems with exponentially convergent bounds. International Journal of Adaptive Control and Signal Processing, 27: 280-301, 2013
Gareía, J. P. Torreglosa, L. M. Femández, and F. Jurado. Viability study of FC-batterySC tramway controlled by equivalent consumption minimization strategy. International Journal of Hy- drogen Energy, 37: 9368-9382, 2012.
A. Gosavi. Simulation-based optimization: Parametric optimization techniques and reinforcement learning. Kluwer Academic Publishers, Norwell, MA, USA, 2003.
J. Han, and Y. Park. A novel updating method of equivalent factor in ECMS for prolonging the lifetime of battery in fuel cell hybrid electric vehicle. In IFAC Proceedings. volume 45(4), pp- 227-232, 2012.
J. Han, J.F. Charpentier, and T. Tang. An Energy Management System of a Fuel Cell/Battery Hybrid Boat. Energies, 7: 2799-2820, 2014.
J. Han, Y. Park, and D. Kum. Optimal adaptation of equivalent factor of equivalent consumption minimization strategy for fuel cell hybrid electric vehicles under active state inequality constraints. Journal of Power Sources, 267: 491-502 2014.
D. Jacobsen and D. Mayne. Differential Dynamic Programming. Elsevier, 1970.
R. Kamalapurkar, L. Andrews, Walters, and E. Dixon. Model-basee based reinforcement learning for mainite-horizon approximate optimal tracking. In Proceedings of the IEEE Conference on Decision and Control (CDC), pp. 5083-5088, 2014.
R. Kamalapurkar, H. Dinh, S. Bhasin, and W. F. Dixon. Approximate optimal trajectory tracking for continuous-time nonlinear systems. Automatica, 51: 40-48, 2015.
S. G. Khan et al. Reinforcement learning and optimal adaptive control: An overview and implemen- tation examples. Annual Reviews in Control, 36: 42-59, 2012.
M.J. Kim and H. Peng. Power management and design optimization of fuel cell/battery hybrid vehicles. Journal of Power Sources, 165: 819-832, 2007.
D. Kirk. Optimal Control Theory: An Introduction. Mineola, NY, Dover, 2004.
10

---

Under review as a conference paper at ICLR 2021
V. Konda and J. Tsitsiklis. On actor-critic algorithms. SIAM Journal Control and Optinization, 42: 1143-1166, 2004.
S. Levine and P. Abbeel. Learning Neural Network Policies with Guided Search under Unknown Dynamics. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
F. L. Lewis, S. Jagannathan, and A. Yesildirak. Neural network control of robot manipulators and nonlinear systems. CRC Press, Philadelphia, PA, 1998.
F. L. Lewis, D. Vrabie, and V. L. Syrmos. Optimal Control, 3rd edition. Wiley, NJ, 2012.
H. Li, A. Ravey, A. DiDiay, and A. Djerdir. A Review of Energy Management Strategy for Fuel Cell Hybrid Electric Vehicle. In IEEE Vehicle Power and Propulsion Conference (VPPC), pp. 1-6, 2017.
W.S. Lin and C.H. Zheng. Energy management of fuel cell/ultracapacitor hybrid power system using an adaptive optimal control method. Journal of Power Sources, 196(6): 3280- 3289, 2011.
P. Mehta and S. Meyn, "Q-learning and pontryagin's minimum principle. In Proceedings of IEEE Conference on Decision and Control, pp. 3598-3605, 2009.
D. Mitrovic, S. Klanke, and S. Vijayakumar. Adaptive Optimal Feedback Control with Learned Internal Dynamics Models. Springer, pp. 65-84, Berlin, 2010.
H. Modares and F. L. Lewis. Optimal tracking control of nonlinear partially-unknown input systems using integral reinforcement learning. Automatica, 50(7): pp. 1780-1792, 2014. unknown constrained-
S. J. Moura, D. S. Callaway, H. K. Fathy, and J. L. Stein. Tradeoff's between battery energy capacity and stochastic optimal power management in plug-in hybrid electric vehicles. Journal of Power Sources, 1959(9): pp. 2979-2988, 2010.
S. N. Motapon, L. Dessaint, and K. Al-Haddad. A Comparative Study of Energy Management Schemes for Fuel-Cell Hybrid Emergency Power System of More-Electric Aircraft. IEEE Trans- actions on Industrial Electronics, 61: pp. 1320-1334 1334, 2014.
G. Paganelli, S. Delprat, M. Guerra, J. Rimaux, and Santin. Equivalent consumption mini- mization strategy for parallel hybrid powertrains. In IEEE 55th Vehicular Technology Conference, VTC Spring 2002 (Cat. No. No.02CH37367), pp. 2076-2081, 2002.
F. Segura and J. M. Andújar. Power management based on sliding control applied to fuel cell sys- tems: further step towards the hybrid control concept. Applied Energy, 99: 213-225, 2012.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 1998.
E. Theoddorou, Y. Tassa, and E. Todorov. Stochastic Differential Dynamic Programming. In Pro- ceedings of American Control Conference, 2010.
E. Todorov and Y. Tassa. Iterative Local Dynamic Programming. In Proceedings of the IEEE International Symposium on ADP and RI. 2009.
J. P. Torreglosa, P. García, L. M. Fernández, and F. Jurado. Predictive Control for the Energy Man- agement of a Bell-Cel-Battery- SuutreaaSSuurraaaciitoo Tramway. IEEE Transactions on Industrial Infor- matics, 10(1): 276-285, 2014.
D. Vrabie. Online adaptive optimal control for continuous-time systems. Ph.D. dissertation, Uni- versity of Texas at Arlington, 2010.
Dan Yu, Mohammadhussein Rafieisakhaei, and Suman Chakravorty. Stochastic Feedback Control of Systems with Unknown Nonlinear Dynamics. In IEEE Conference Decision and Control, 2017.
M. K. Zadeh. Stability Analysis Methods and Tools for Power Electronics-Based DC Distribution Systems: Applicable to On-Board Electric Power Systems and Smart Microgrids. NTNU, 2016.
M. K. Zadeh, L. Saublet, R. Ghoagsaz B. Nahid Mobarakeh, S. Pierfederici, and M. Molinas. Energy management and stabilization a hybrid DC microgrid for transportation appli-
11

---

Under review as a conference paper at ICLR 2021
cations. In IEEE Applied Power Electronics Conference and Exposition (APEC), pp. 3397-3402, 2016.
X. Zhang, C. Mi, A. Masrur, and D. Daniszewski. Wavelet caansformmbssee power management of hybrid vehicles with multiple on-board energy sources including fuel cell, battery and ultracapac- itor. Journal of Power Sources, 185(2): 1533-1543 2008.
C. Zheng, S. W. Cha, Y. Park, W. S. Lim, and G. Xu. PMP-based power management strategy of fuel cell hybrid vehicles considering multi-objective optimization. International Journal Preciston Engineering and Manufacturing, 14(5): 845-853, 2013.
C. H. Zheng, G. Q. Xu, Y. I. Park, W. S. Lim, and S. W. Cha. Prolonging fuel cell stack lifetime based on Pontryagin's Minimum Principle in fuel cell hybrid vehicles and its economic influence evaluation. Journal Power Sources, 248: 533-544, 2014.
X. Zhong, H. He, H. Zhang, and Z. Wang. Optimal Control for Unknown Diiscrete-Tinee Nonlin- Markov Jump Systems Using Adaptive Dynamic Programming. IEEE Transactions on Neural networks and learning systems, 25(12): pp. 2141-2155, 2014.
A APPENDIX
A.1 THE GRADIENT DESCENT ALGORITHM
The Gradient Descent Algorithm has been explained as follows:
Algorithm; Gradient Descent
Input: Design Parameters ( = Ug, a, R
Output: Optimal control sequence {ū.}
1. + 0, VuJa (zo, ( t
2. while Ju (ro, U(n))= 2 c do
3. Evaluate the cost. function with control (7())
(n) by h 0.... ,t, and calculate the gradient vector
4. Perturb each control variable u,
VuJa (zo, (n)) using (7) and (8)
5. Update the control policy: (n+1) U(n) avvJa (zo, U(1)
6. n + n n
7. end
8. {ūc} < (m))
A.2 EXPERIENCE SIMULATION
Assumption (Kamalapurkar et al., 2014) There exists a finite set of points
{G E i 1, 1.... NN wow! pi
and constant IR such that 0 c 1/4 (infreR (Zman {
})) where Pi =1 +
To, R. and w; Vçar (G) Fa (5..0) (G) G (G.) á (c.WW))
Using Assumption 5, simulation of experience is implemented by the weight update laws given by
W. Mea
162
(24)
D
N
Pi
r [T ne1 wwo wwo?
r)
(1rles (IT (40) ST.
(25)
W. = na1 (Wo - W.) "azWaa
Mo1
N
Go mi
W.
(26)
4p
4Np;
12

---

Under review as a conference paper at ICLR 2021
where, w. = Vcr((S) (Fo(s.0) + Fi()) + (s. T E R.XL is the least-squares gain
matrix, E R denotes a positive saturation constant, E IR indicates a constant forgetting factor, Mc,, Mo2, fa1, Mal; Ma2 E I defines constant positive adaptation gains, (.) denotes the indicator function of the set (-}.G.
(= V (C), and p + vw"Tw, where E I is a positive normalization constant. In the above weight update laws, for any function E(S, :) the notation Ea, is defined as & (Gi, +), and the instantancous BEs ò, and d are given as d, (c, We. Wa. 0) and d2s = (c.. We, W..0)
13


Reviews:
[{"review_id": "ha-1m_BbWhP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a reinforcement learning framework for the fuel optimization problem in hybrid electric vehicles. \n\nStrong points:\n+ The design of the deep RL-based controller has been discussed in detail.\n+ The simulation results show both the value and policy functions can quickly converge to their optimal values.\n+ The $H_\\infty$ performance index shows the RL-based method is more robust than the classical PID/MPC-based methods.\n\nWeak points:\n- The HEV fuel/energy functions are generally nonlinear non-convex functions that have been widely modeled in the past literature. It’s not clear how the authors consider the cost function as a quadratic formulation (Page 4 Section 2.1) and relate it to optimizing the HEV fuel consumption. \n- The quality and completeness of Figure 1 needs to be improved.  \n- More details should be provided on how the data-driven identifiers work and interact with the actor-critic networks in Figure 1. What are the benefits of the proposed method, when compared with directly applying a commonly known actor-critic RL method such as DDPG or PPO to solving the problem?\n- The estimated percentage of fuel savings with the proposed RL-based method over the benchmark studies (PID/MPC-based methods and optimal control-based method) is not discussed.\n- For the two-dimensional nonlinear model given by equation 11, can this model be extended to consider other surrounding agents’ movement?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S9BLTPi0YWT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n### Summary\nThe paper proposed to use RL methods for control for hybrid vehicles fuel\nstrategy. The organisation of the paper is difficult to follow and I may have\nmissed some of the arguments the authors are making. The experimental section is\nvery thin with only a 2 d system simulation.\n\n### To improve\n\n1. How is this even possible ? \"We now solve the open loop optimization problem\n   using a general non-linear programming solver without actually knowing the\n   exact form of the underlying dynamics\" The cost doesn't have the dynamical system\n   equation but you need to enforce it either as a constraint or solve is via\n   typical single shooting methods. Constraint is mentioned in the section 2.1 but it\n   is still incorrect to says that gradient descent is without access to dynamics.\n   Do you mean to say that we learn dynamics from the data ? and model is\n   assumed to be unknown at the design time ?\n2. MPC and PID are not the same and I am not sure why they are clubbed together\n   as a baseline. Especially in the experiment shown the system is simple\n   enough to be learned by wide enough NN. Then we can use non-linear MPC with\n   out of the box optimisers as SOTA baseline  for MPC ? \n3. Authors learn a NN based model as part of the proposed method, MPC would be\n   an algorithm that can use this model to optimise the control.\n4. If the hybrid system model is as described in the experimental section I am\n   struggling to see how this scholarship helps compared to standard optimal\n   control methods ?\n5. I found the overall paper very challenging to follow. There are large logical\n   jumps. I am open to changing my review, if authors can show the clear benefits of their proposal.\n\n### citations\n\n1. correct source for DDP is Mayne [1]\n\n### Language\n\n1. This has no impact on technical rating of the paper and it does not directly\n   contribute to the review score.\n2. I had started to write all the typos and grammatical errors in the paper and\n   I stopped as there are far too many of them. Please review this for language\n   errors as this breaks flow of reading.\n\n### Ref\n\n1. Jacobson, D. H., & Mayne, D. Q. (1970). Differential Dynamic Programming. American Elsevier Publishing Company. https://books.google.co.uk/books?id=tA-oAAAAIAAJ", "extra_scores": {}, "extra_text": {}}, {"review_id": "JdHasA-NSac", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a deep reinforcement learning-based optimization strategy to the fuel optimization problem for the hybrid electric vehicle. The problem has been formulated as a fully observed stochastic Markov Decision Process (MDP). A deep neural network is used to parameterize the policy and value function. A continuous time representation of the problem  is also used compared to conventional techniques which mostly use a discrete time formulation. \n\nThe paper is very convoluted, with lots of different novelties, making it hard to understand the main contributions and how much each of the contributions truly affect the final results. Due to this, experiments are also lacking, testing the different aspects of the proposed approach. I.e., is the proposed reward model truly necessary? Does the overall RL model have to be so complex for this to work well? Perhaps a simpler model would work just as well or even better? What motivates the use of each of the different aspects of the model and each of the assumptions that are made? For example, the paper mentions \"the convergence of the traditional RL requires sufficient exploration of the state-action space\" but fails to explain how the proposed RL approach addresses this issue. In fact the proposed approach seems to be a straightforward application of actor-critic methods. The experiments seems the core contribution but there is only one page and isn't able to be replicated and thus, difficult to really evaluate. As an applied work, the solution is very specialized such that I am unsure that the observations would transfer to another domain. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0uMioygkhH2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper purports to be about Hybrid vehicle fuel optimization using RL. In reality, most of the content seems to be a long derivation of a continuous-space generic RL based controller for a trajectory optimization problem. There does not seem to be much that is  novel in this entire presentation and if there is, the authors have not explicated it in the introduction or other sections. In the last section a very brief simulation-based experiment is described supporting the hybrid vehicle part. The bulk of the paper derives a controller using what seems to me to be standard ideas or minor variations thereof. Authors or other reviewers can correct me if I'm wrong but if so, the sheer density of  sec 2, esp 2.2 esp. made it hard to assess this (and I lacked motivation given what I perceived about the overall paper structure), so I would still argue for reject on clarity grounds.\n\nThe introduction has a comprehensive literature review of methods that have been used for open loop trajectory control, fuel optimization and so on.  This should be appreciated.\n\nComing to the claims in the introduction: First we solve an open loop trajectory optimization problem, then design an RL controller to track the nominal trajectory. There is nothing new in this idea. The second point brings up the use of something called Concurrent Learning as a contribution which is mentioned exactly once again in sec 2.1 in passing and never defined or referred to again. Perhaps 2.2 onwards describes a use of it but I can't tell.  Contribution #3 seems to be that using H-\\infinity as a performance measure for an RL control algorithm is proposed for the first time, but again no description or justification is given.\n\nSec 2: up to 2.1 seems pretty straightforward. \n\n2.2: Assumption 2: is g^+ the usual psuedo-inverse? It is usually defined with an extra A^T at the end.  is there a typo in assumption 3?  from that condition it seems that gg^+ should be identity, i dont think this is intended. Where is novelty in the rest of this section? Using HJB etc is standard. The description of various approximation schemes is ok but doesnt lead anywhere, unless 2.2.1-2 represents Concurrent learning.  sec 2.2.1 just seems to say we can approximate the dynamics using an NN. sec 2.2.2 approximates the value function using a NN as well. The universal function approximation theorem etc is invoked, but what new thing is really being said about VF approximation? I couldn't tell. What is the CL-based update law?\n\n2.2.4: I dont remember enough about lyapunov conditions etc to dive deeply into this.  I know that lyapunov conditions are used to show stability of controllers. Is this a straightforward application of this method then?\n\nsec 3: Since the main thrust of the paper is fuel optimization, absolutely no explanation or citation is given for why these particular system dynamics are relevant. Indeed, they seem far simpler than I would expect such a complex system to have, but I am definitely no expert. The results seem fine but not enough to justify this paper for ICLR as a deep and original contribution on learning representations.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to solve the fuel optimization problem in hybrid electric vehicles using reinforcement learning. The work is interesting, but the reviewers consider it lacks novelty and it has different concerns on the assumptions of the modeling. The paper is quite difficult to follow. 
Decision: Reject