Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: This paper purports to be about Hybrid vehicle fuel optimization using RL. In reality, most of the content seems to be a long derivation of a continuous-space generic RL based controller for a trajectory optimization problem. There does not seem to be much that is  novel in this entire presentation and if there is, the authors have not explicated it in the introduction or other sections. In the last section a very brief simulation-based experiment is described supporting the hybrid vehicle part. The bulk of the paper derives a controller using what seems to me to be standard ideas or minor variations thereof. Authors or other reviewers can correct me if I'm wrong but if so, the sheer density of  sec 2, esp 2.2 esp. made it hard to assess this (and I lacked motivation given what I perceived about the overall paper structure), so I would still argue for reject on clarity grounds.

The introduction has a comprehensive literature review of methods that have been used for open loop trajectory control, fuel optimization and so on.  This should be appreciated.

Coming to the claims in the introduction: First we solve an open loop trajectory optimization problem, then design an RL controller to track the nominal trajectory. There is nothing new in this idea. The second point brings up the use of something called Concurrent Learning as a contribution which is mentioned exactly once again in sec 2.1 in passing and never defined or referred to again. Perhaps 2.2 onwards describes a use of it but I can't tell.  Contribution #3 seems to be that using H-\infinity as a performance measure for an RL control algorithm is proposed for the first time, but again no description or justification is given.

Sec 2: up to 2.1 seems pretty straightforward. 

2.2: Assumption 2: is g^+ the usual psuedo-inverse? It is usually defined with an extra A^T at the end.  is there a typo in assumption 3?  from that condition it seems that gg^+ should be identity, i dont think this is intended. Where is novelty in the rest of this section? Using HJB etc is standard. The description of various approximation schemes is ok but doesnt lead anywhere, unless 2.2.1-2 represents Concurrent learning.  sec 2.2.1 just seems to say we can approximate the dynamics using an NN. sec 2.2.2 approximates the value function using a NN as well. The universal function approximation theorem etc is invoked, but what new thing is really being said about VF approximation? I couldn't tell. What is the CL-based update law?

2.2.4: I dont remember enough about lyapunov conditions etc to dive deeply into this.  I know that lyapunov conditions are used to show stability of controllers. Is this a straightforward application of this method then?

sec 3: Since the main thrust of the paper is fuel optimization, absolutely no explanation or citation is given for why these particular system dynamics are relevant. Indeed, they seem far simpler than I would expect such a complex system to have, but I am definitely no expert. The results seem fine but not enough to justify this paper for ICLR as a deep and original contribution on learning representations.

review_title: Little original contribution and not really about fuel optimization

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work proposes a deep reinforcement learning-based optimization strategy to the fuel optimization problem for the hybrid electric vehicle. The problem has been formulated as a fully observed stochastic Markov Decision Process (MDP). A deep neural network is used to parameterize the policy and value function. A continuous time representation of the problem  is also used compared to conventional techniques which mostly use a discrete time formulation. 

The paper is very convoluted, with lots of different novelties, making it hard to understand the main contributions and how much each of the contributions truly affect the final results. Due to this, experiments are also lacking, testing the different aspects of the proposed approach. I.e., is the proposed reward model truly necessary? Does the overall RL model have to be so complex for this to work well? Perhaps a simpler model would work just as well or even better? What motivates the use of each of the different aspects of the model and each of the assumptions that are made? For example, the paper mentions "the convergence of the traditional RL requires sufficient exploration of the state-action space" but fails to explain how the proposed RL approach addresses this issue. In fact the proposed approach seems to be a straightforward application of actor-critic methods. The experiments seems the core contribution but there is only one page and isn't able to be replicated and thus, difficult to really evaluate. As an applied work, the solution is very specialized such that I am unsure that the observations would transfer to another domain. 
review_title: Official Blind Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a reinforcement learning framework for the fuel optimization problem in hybrid electric vehicles. 

Strong points:
+ The design of the deep RL-based controller has been discussed in detail.
+ The simulation results show both the value and policy functions can quickly converge to their optimal values.
+ The $H_\infty$ performance index shows the RL-based method is more robust than the classical PID/MPC-based methods.

Weak points:
- The HEV fuel/energy functions are generally nonlinear non-convex functions that have been widely modeled in the past literature. It’s not clear how the authors consider the cost function as a quadratic formulation (Page 4 Section 2.1) and relate it to optimizing the HEV fuel consumption. 
- The quality and completeness of Figure 1 needs to be improved.  
- More details should be provided on how the data-driven identifiers work and interact with the actor-critic networks in Figure 1. What are the benefits of the proposed method, when compared with directly applying a commonly known actor-critic RL method such as DDPG or PPO to solving the problem?
- The estimated percentage of fuel savings with the proposed RL-based method over the benchmark studies (PID/MPC-based methods and optimal control-based method) is not discussed.
- For the two-dimensional nonlinear model given by equation 11, can this model be extended to consider other surrounding agents’ movement?

review_title: Official Blind Review #4 

final_score: rating: 3: Clear rejection
review_content: 

### Summary
The paper proposed to use RL methods for control for hybrid vehicles fuel
strategy. The organisation of the paper is difficult to follow and I may have
missed some of the arguments the authors are making. The experimental section is
very thin with only a 2 d system simulation.

### To improve

1. How is this even possible ? "We now solve the open loop optimization problem
   using a general non-linear programming solver without actually knowing the
   exact form of the underlying dynamics" The cost doesn't have the dynamical system
   equation but you need to enforce it either as a constraint or solve is via
   typical single shooting methods. Constraint is mentioned in the section 2.1 but it
   is still incorrect to says that gradient descent is without access to dynamics.
   Do you mean to say that we learn dynamics from the data ? and model is
   assumed to be unknown at the design time ?
2. MPC and PID are not the same and I am not sure why they are clubbed together
   as a baseline. Especially in the experiment shown the system is simple
   enough to be learned by wide enough NN. Then we can use non-linear MPC with
   out of the box optimisers as SOTA baseline  for MPC ? 
3. Authors learn a NN based model as part of the proposed method, MPC would be
   an algorithm that can use this model to optimise the control.
4. If the hybrid system model is as described in the experimental section I am
   struggling to see how this scholarship helps compared to standard optimal
   control methods ?
5. I found the overall paper very challenging to follow. There are large logical
   jumps. I am open to changing my review, if authors can show the clear benefits of their proposal.

### citations

1. correct source for DDP is Mayne [1]

### Language

1. This has no impact on technical rating of the paper and it does not directly
   contribute to the review score.
2. I had started to write all the typos and grammatical errors in the paper and
   I stopped as there are far too many of them. Please review this for language
   errors as this breaks flow of reading.

### Ref

1. Jacobson, D. H., & Mayne, D. Q. (1970). Differential Dynamic Programming. American Elsevier Publishing Company. https://books.google.co.uk/books?id=tA-oAAAAIAAJ
review_title: Lacks clarity to judge the paper objectively 

Meta-review: This paper proposes to solve the fuel optimization problem in hybrid electric vehicles using reinforcement learning. The work is interesting, but the reviewers consider it lacks novelty and it has different concerns on the assumptions of the modeling. The paper is quite difficult to follow. 
Decision: Reject