Long-term planning, short-term adjustments
Authors: Hamed Khorasgani
Chi Zhang
Chetan Gupta
Susumu Serita
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Control

Abstract: Deep Reinforcement Learning (RL) algorithms can learn complex policies to optimize
agent operation over time. RL algorithms have shown promising results
in solving complicated problems in recent years. However, their application on
real-world physical systems remains limited. Despite the advancements in RL
algorithms, the industries often prefer traditional control strategies. Traditional
methods are simple, computationally efficient and easy to adjust. In this paper,
we propose a new Q-learning algorithm for continuous action space, which can
bridge the control and RL algorithms and bring us the best of both worlds. Our
method can learn complex policies to achieve long-term goals and at the same time
it can be easily adjusted to address short-term requirements without retraining.
We achieve this by modeling both short-term and long-term prediction models.
The short-term prediction model represents the estimation of the system dynamic
while the long-term prediction model represents the Q-value. The case studies
demonstrate that our proposed method can achieve short-term and long-term goals
without complex reward functions.

Under review as a conference paper at ICLR 2020
LONG-TERM PLANNING, SHORT-TERM ADJUSTMENTS
Anonymous authors
Paper under double- suuble-blind review
ABSTRACT
Deep reinforcement learning (RL) algorithms can learn complex policies to op- timize agent operation over time. RL algorithms have shown promising results in solving complicated problems in recent years. However, their application on real-world physical systems remains limited. Despite the advancements in RL algorithms, the industries often prefer traditional control strategies. Traditional methods are simple, computationally efficient and easy to adjust. In this paper, we propose new Q-Jearning algorithm for continuous action space, which bridge the control and RI. algorithms and bring us the best of both worlds. Our method can learn complex policies to achieve long-term goals and the same time it can be easily adjusted to address short-term requirements without retraining. We achieve this by modeling both short-term and long-term prediction models. The short-term prediction model represents the estimation of the system dynamic while the long-term prediction model represents the Q-value. The case studies demonstrate that our proposed method can achieve short-term and long-term goals without complex reward functions.
INTRODUCTION
Optimal control methodologies use system dynamic equations to design actions that minimize de- sired cost functions. A cost function can be designed to track a trajectory, reach a goal, or avoid obstacles. It is also possible to design cost function to achieve combination of goals. Model Predictive Control (MPC) is a common optimal control technique and has been applied to many in- dustrial applications such as pressure control and temperature control in chemical processes (Garcia et al., 1989). The traditional control solutions are not adequate to address the challenges raised with the evolution of industrial systems. Recently, deep reinforcement learning (RL) has shown promis- ing results in solving complex problems. For example, it has generated superhuman performance in chess and shogi (Silver et al., 2017). The following advantages make deep RL strong candidate to overcome traditional control limitations. First, deep RL has an advantage in solving complex problems, especially when the consequences of an action are not immediately obvious. Moreover, it can learn an optimal solution without requiring detailed knowledge of the systems or their engi- necring designs. Finally, deep RL not limited to time-series sensors and can use new sensors such as vision for a better control.
Inaley
Paplod sogle
Taget %c)
Plyload
Soytedd (sx%)))
Figure 1: Crane system. The long-term goal is move the payload to the target location, (ara, ya) as soon as possible. The short-term goal is to have zero sway at the destination, wp = 0.
However, deep RL has not been applied to address industrial problems in a meaningful way. There several key issues that limit the application of deep RI. real-world problems. Deep RL al-

---

Under review as a conference paper at ICLR 2020
gorithms typically require many samples during training (sample complexity). Sample complexity leads to high computational costs. A high computational cost can be justified for industries as a one- time charge. However, oftentimes small changes in the system goal, such as changing the desired temperature in a chemical reactor, or a new constraint such as a maximum allowable temperature in the reactor, require retraining the model. Moreover, industrial systems often have several short-term and long-term objectives. For example, consider the crane system shown in Figure 1. The long-term goal is to convey the payload to the target location soon as possible. However, when the payload gets close to the target, it must have minimum sway for the safety of the operators. Designing re- ward function that can capture these short-term and long-term goals concurrently can be challenging or even infeasible.
class of short-term objectives related to safe exploration during RI. training have been studied recently. Gu et al. (2017) presented an application of deep RI. for robotic manipulation control. To ensure safe exploration, they set maximum limits for the joint positions, and joint velocities. Moreover, they set sphere boundary for the end-effector position and when the boundaries were about to be violated, they used correction velocity to force the end-effector position back to the center of the sphere. Dalal et al. (2018) formulated the safe exploration They proposed to add a safety layer that modifies the action at each time step. Toward this end, they learn the constraint function using linear model and use this model find the minimal change to the action such that the safety constraints are met at each time step. To the best of our knowledge, there is no study addressing short-term objectives during application.
optimization problem.
In this paper, we present a Locally Linear Q-Learning (LLQL) algorithm for continuous action space. The LLQL includes a short-term prediction model, long-term prediction model, and controller. The short-term prediction model represents a locally linear model of the dynamic system, while the long-term prediction model represents the value function + a locally linear advantage function. The controller uses the short-term prediction model and the long-term term-termmmm prediction model to generate actions that achieve short-term and long-term goals simultaneously. adopts policy that maximizes Q-value while achieving short-term goals. The LIQQI. algorithm has the following advantages:
It does not require designing sensitive reward functions for achieving short-term and long- term goals concurrently.
It shows better performance in achieving short-term and long-term goals compared to the traditional reward modification methods.
It is possible to modify the short-term goals without time-consuming retraining.
The remainder of this paper organized as follows. Section represents the background in dynamic systems and RL. Section 3 represents the LLQL algorithm. Section 4 presents our methodology to achieve short-term and long-term goals using LLQL. Section presents our experimental results. Section 6 presents the conclusions of the paper. Section A discusses the related work. Section B presents additional experiments for those interested.
2 BACKGROUND AND DEFINITIONS
In this section, we review the backgrounds in dynamic systems and reinforcement learning.
2.1 DYNAMIC SYSTEMS
A continuous-time dynamic system can be represented as:
dx(t) = f f(x(t), u(t), t:p),
dt
where given the system parameters, p, maps the state variables, x E X, and actions, u U. to the state derivative, dz Tcontrol (xt)) that generates proper actions trajectory, (t)... is challenging to design a control policy for a nonlinear complex system repre- sented in equation (1).
de at time t. In state space control, the goal is to design a control policy,
the state variables follow the given desired
2

---

Under review as a conference paper at ICLR 2020
The control problem becomes much easier address when this system is linear with respect to the input (Chen et al., 2003). We can present these systems as:
dx(tt) (=(()))
dt
Since measurements are typically sampled in discrete times, we derive discrete time version of linear system (2). Using a first-order approximation:
dx((x) s((l+1)) xx(12)
dt
1x+1 ix
where t represents time at sample point k. In this paper, we assume the sampling rate is constant; A 1++1 la- Using (2) and (3), have:
a(tx+1) - x(tx) = A(f(x(ta)) + ((a(tt))u((t))
(4)
For brevity, we present l by k. f(x((x)) by (xxx,, and g(x((x)) by g(zx) in the remainder of the paper. Therefore, we can represent dynamic system as:
1k++1 Tx + ((((x)) + g(xx)ux)
(5)
2.2 REINFORCEMENT LEARNING
The goal of RL is to learn a policy, TRL (u//=*), that generates a set of actions, U, that maximize the expected sum of rewards in the environment, En. Consider:
T
R
Σ "r(x, u4), 1=k
(6)
where 7 is the discount factor, the reward function and T' represents the end time and can be set to T' = co. The goal is to learn TRL for environment, E,, such that:
max R "a [R]]).
Unlike control algorithms, model-free reinforcement learning algorithms assume the system dy- namic is unknown. Q-function, Q (Ix, up) is defined as the expected return state Iy when we take action un and adopt policy afterward:
Q" (Iz,, ux) Er
RR RR ugl).
Qelerrning algorithms (Watkins & Dayan, 1992) are among the most common model-free RL meth- ods for discrete action space problems. These algorithms use the Bellman recursive equation to model D-function:
(I un) =
E Er.z [r(*k, ux) + 7Q" ((zx+1))))
where y represents greedy deterministic policy that selects the action which maximizes Q-value at each step:
(( argmax,, Q(Tk, Uk).
(10)
Q-learning algorithms leamn the parameters of the function approximator, 09, by minimizing the Bellman error:
min(L =
ux|09) -xx)
(11)
V/ ( ux) + ((
where yv is the fixed target fufnction, and 3 represents the exploration policy.
For continuous action domain problems, it is not trivial to solve equation (10) at each time step. Finding an action to maximize Q which can be complex nonlinear function is computationally expensive or even infeasible. To address this problem, Lillicrap et al. (2015) proposed the Deep Deterministic Policy Gradient (DDPG) algorithm, which learns two networks simultaneously. The critic network learns Q-function by minimizing the Bellman error, and the actor network learss parameters of the policy to maximize the estimated value of Q-function. Gu et al. (2016) proposed
3

---

Under review as a conference paper at ICLR 2020
Normalized Advantage Function (NAF) Q-learning which formulates the Q-function the sum of the value function, Vz)), and the advantage function, A(r,u) u).
(z ul62) V(x10") + A(r, ul0A),
(12)
where
1 ((al" Pua v(x]0)))
(13)
P(x|0P)= L(x L(x) (L(x))))((())))) where [xx a lower-triangular matrix. The value function not function of action, Therefore, the action which maximizes advantage function, A, maxi- mizes the Q function. P(x10") is positive-definite matrix, and therefore, the action that maximizes the advantage function and the Q-function is given by v)z)0").
3 LOCALLY LINEAR Q-LEARNING
Short-derm Network
Grezn boses: Inputs
3 Networis
fx
Blue becent short-amm nelwork Orange bokess longtterm term network
Ets 195)
Xx+1
Networks
8k
Loss lunction
X
Controller
Mi
Longi serm Netwoakk
V Network
fundtion
lgg
Advantage function
ii Networkk
dy
Figure 2: Learning the LLQL Network Parameters.
In this section, propose the LLQL algorithm, which like (Lillicrap et al., 2015) and (Gu al., 2016) can handle continuous action space. Our approach learns short-term and long-term prediction models. Using the long-term and short-term models, controller generates actions that guide the system toward its short-term and long-term goals. Figure 2 shows our proposed structure to learn the parameters of the short-term and long-term prediction models.
Short-term prediction: consider the nonlinear system presented in equation (5). In this work, we use deep neural networks to estimate system functions, (*xx,, and g(æx) at each operating point. Substituting the network estimations for these functions in equation (5), we can predict the next state as:
サ 3rk + ((f(x))) + g(xx|09))us
(14)
where 1 represents our estimation of the next step. and 0f and @ are the network parameters. A is a constant hyper parameter. In dynamic systems, the difference between two consecutive states, rk+1 Tk. typically very small. Considering small A leads to reasonable f and g values and, therefore, improves learning time and accuracy.
We call this dynamic system model short-term prediction model. The controller uses this model to generate actions, which lead the system toward its short-term goals. Note that previous work have used the system short-term dynamic model for generating additional samples imagination rollout (for example, see (Gu et al.. 2016), and (Racanière al., 2017)). In this paper, show that this model can also be used to design actions to achieve short-term goals. To learn the parameters of short-term prediction model, gf and 6 minimize the short-term loss function, L1, as it is presented in Algorithm 1.
Long Long-term -erm prediction: Q-function represents the maximum cumulative reward that can be achieved from current state, Fg. taking an action Ug. Therefore, by learning Q-function, we learn the long- term prediction model for the system. Like NAF (Gu al., 2016) (see equation (12)), present
4

---

Under review as a conference paper at ICLR 2020
Q-function sum of value function and advantage function. However, we present the advantage function, A(r, u ul@*) using a locally linear function of I& and uy as:
Q(2, u/ u000) )- - (x10") A(x, 2 (0*), A(r, u 00) (1xx + d( (0*))=))l)
(15)
where hxxx(0)) and d(æ 10") networks model the locally linear vantage function. Note the NAF advantage function is special case of the LLQL advantage function when d(xx)00*) I, where represents the identity matrix.
To maximize Q-function, we have design ug which minimizes h( 6h)) d(xx For simplicity, we present h(zu/0"), and d(zx (0") with h2 and dą respectively in the remainder of the paper. To maximize Q-function and achieve the long-term goal, we can use simple pseudo-inverse matrix multiplication and derive solution with the least squares enror as:
ugg. = -df d h4.
(16)
When ||d&]| - O, it means the network predicts that our action has impact the advantage function. Therefore, we choose random action. Random exploration is an important part of any deep RL algorithm. Therefore, in addition this unlikely case, add noise, N&g, to the action, ug. during the training. We reduce the amount of noise injected to the action as the algorithm converges.
Algorithm Locally Linear Q-Learning Training
I: Initialize network (equation (15)) with random weights.
2: Initialize target network, Q' parameters: g² 0
3: Create the reply buffer Rt = Ø.
4: for episode 1:M do Initialize random process A for action exploration. Receive the initial observation, 20.
7: for k = 1:T do
8: if |dall then
9:
Set uA - (df d (hx + N
10:
else
Sct uu Na
Execute ua and observe Te+1 Tk- Store transition (æx,, uk, 1xk, rT)) in
13:
for iteration I:T, do Randomly select mini-batch of N. transition from R.
14:
15:
Update 8² and 8a by minimizing the loss: L1 g(2,|00 )+))l for itcration 1:71 do Randomly select a mini-batch of N₁ transition from R.
NN
1|2141 - 24, A(f(x.100) +
N,
16:
17:
18:
Set y, = + QQ (#:41) (00
19:
Update d by minimizing the loss: La 1. Σm 111. Q(x 18 19)1].
N:
Update the target network: 0º 70 (1 -1-
20:
In the application, the controller solves Ug with additional constraints to achieve the desired short- term trajectories. We will discuss our short-term adjustment algorithms in the next section. learn -ufunction, in addition to the state estimation eπrоr, minimize the long-term loss function, L2. as it is presented in Algorithm 1. Note that having the short-term model, it is straightforward to add imagination rollout to algorithm increase sample efficiency. However, improving sample efficiency in RL is not the focus of this work.
CONTROL. STRATEGY
By separating action design from prediction models, LLQL gives us the freedom to design different control strategies for achieving short-term and long-term goals. Moreover, the linear structure of short-term and long-term models simplifies the control design. Consider the case where LLQL has learned perfect long-term model for environment using Algorithm 1. In this case, the optimum
5

---

Under review as a conference paper at ICLR 2020
solution to achieve the long-term goal is given by equation (16). When we have more short- term goals as well, we can formulate the control design as an optimization problem to satisfy both short-term and long-term goals much as possible.
In this paper, we consider two types of short-term goals: 1) desired trajectory, and 2) constraint. In the first scenario, the agent has short-term desired trajectory. For example, may be required to travel with specific speed during certain periods. In the second scenario, the agent has some limitation for a specific period of time. For example, a is required keep its speed below certain thresholds at some periods during the trip. To address the first problem, we add an additional term to the cost function for the short-term goal and solve for the action. We deal with the second problem as constraint optimization.
4.1 SHORT-TERM TRAJECTORY
Let 2d represent our desired short-term trajectory. We develop control strategy to track zy while pursuing the long-term goals.
Using system dynamic functions f& and Ok. we can define our control optimization problem as:
min (%1 (hx + duux)² 72(=a(++1) Ik A(fk + g)²)))²),
(17)
find My
where 11(k++1 represents the desired trajectory time k + 7/1 and 7/2 are positive coefficients and can be adjusted to give higher weights to the short-term or long-term goals. Note that in this work, we assume the short-term goals are temporary and when their time expires the system goes to the long-term optimum policy given by (16). For example, we may require car to have a specific speed at some specific locations.
We can apply similar pseudo-inverse matrix multiplication, and derive solution with the least squares error for (17) as:
7i 722gg
di 1 72 gg
de 171 gg
1
-7 (2( ( + Ix Afis)
(18)
u1 =
4.2 SHORT-TERM CONSTRAINT
The LLQL algorithm provides a framework to design the actions considering different constraints. For safe operation, the agent may have to avoid specific states for period of time (for example, high speed or locations close to an obstacle). For simplicity, we assume at each moment we only have maximum in physical systems the agent is close to one of the boundaries at any moment in time. When this not the case, we can define new constraints constraint the state variable, xf, at time k. We can define the constraint optimization problem for LLQL as:
constraint
state variable, x². This reasonable assumption, because
combination of constraints. Consider oj, as the
min 1 (h+ + d
findu,
(19)
such that:
Th S cl41-
is coefficient added simplify the mathematical operation. Using our estimation of the next step, 2i41 1r + A(fi + glue). we can derive the optimum action which satisfies the constraint
as:
u = ddf (h2 + xaa));
(20)
where ai Ag (d 02
Aggi (dff - and X* of fAfi (affff--++--0 anhy The deriva-
0102
tion details for short-term constraints are presented in Section C.

---

Under review as a conference paper at ICLR 2020
Variables
Poution
DCPC Postion
11.5 Feirnded Fostice
URC Fattnnates Ronibon
MPC DDPG
1.9
100
LLqU
Valocty
LUX Vilcolty
Total Reward
50
DCW bnlecity
MBC velcity
Stop
(b) State variables and estimated state variables. Us- ing LLQQI model, the reached the top in 78 steps. Using DDPG model, the reached the top in 110 steps. The car never reached the top when we used the MPC model. The mean absolute error for LLQL po- (a) Average and standard deviation of cumulative re- sition estimation is 0.00078, the mean absolute enror wards during the learning process. LLQL and DDPG for LLQL velocity estimation is 0.000084, the mean algorithms converge in less than 40 episodes. Model- absolute error for MPC position estimation is 0.00039 based reinforcement learning algorithm (MPC) cannot and the mean absolute error for MPC velocity estima- learn policy that guides the car to the top.
-50
Episode
tion is 0.00014.
Figure 3: LLQL for MountainCarrCnnimuuu The network's parameters are presented in Section
5 EXPERIMENTAL RESULTS
In this section, we demonstrate the performance of LLQL using Mountain Car with Continuous Action (MountainCarConniinnn from OpenAl Gym', In Section B, we apply LLQL to control the crane system shown in Figure 1.
5.1 MOUNTAINCAA
has two state variables: 1) car's position -1.2 2g S. 0.5 and 2) car's velocity - -0.07 S U < 0.07. ui is the continuous action at time k. A negative action pushes the car to the left and positive action pushes the car to the right. The experience stops after 1, 000 steps or when the car reaches the goal on top of the mountain, 24 = 0.5, whichever occurs first. In the beginning of each episode, the car randomly positioned -0.6 S 2 S -0.4. The reward for each episode is 100 for reaching the goal on top of the mountain minus the squared sum of actions from start the goal. Figure 3a shows the cumulative rewards during the training for the LLQL, the DDPG, and a model-based reinforcement learning based on MPC presented by (Nagabandi al., 2018). The details of the MPC based solution is presented in Section E. For each approach we performed the training 20 times, and selected the top 5 models with the maximum cumulative rewards to calculate mean and standard deviation of each episode in Figure 3a.
The MountainCarConntnnnn
The MPC based solution uses the learned short-term predictive model (system dynamic model) to generate a sequence of actions that maximize the reward over a finite horizon. Figure 3b shows that the short-term predictive model estimates future states with high precision. However, optimizing for a finite horizon is a disadvantage for the model- based solution in achieving long-term goals. Increasing the horizon may improve the long-term performance, but it also increases the compu- tational costs in the application phase. In our experiments, the car never reached the top of the mountain using the model-based method. Figure 3b presents the first 110 steps of sample experi- ment. Unlike the MPC based solution, the LLQL and the DDPG algorithms converged in less than 40 episodes (see Figure 3a) and reach the top of the mountain in all experiments (see Figure 3b). Note that it is possible to improve the LLQL convergence time by applying imagination rollout. In fact, our short-term prediction model can be used to generate imaginary scenarios. However, sample efficiency is beyond the focus of this work. Figure shows by applying the long-term predictive model, the LLQL algorithm outperforms the model-based reinforcement algorithm in achieving the long-term goals. In the next two subsections, will show the LLQL algorithm can outperform the DDPG algorithm in achieving the short-term goals by using the short-term predictive model.
'htp::////e openai opwnai.eep........oo............ www/enos homm/enss/Mcunt.aa nCa rCont nuous-vo/

---

Under review as a conference paper at ICLR 2020
State Variables
CDC Hoathed Raward NPC Mod Roward
Act
0.5
Edtrated
Lompaarr
UGL
EStrated
0132
CDN Hoothed Ranard
Velocty
Mod Boward
norm
6325
MIC. Mill Soouel
0200
(10) 110 Forreed
May
Step 200 350
(a) With LLQL the car reached the top of the mountain (b) Comparing the actions in normal case versus in 82 steps with a short-term goal error of 0.00008. With with the short-term trajectory. The hybrid strat- DDPG + modified reward (3t reward in Table 1) the egy (sec equation (21)) leads to quick adjustment in reached the top in 138 steps with short-term goal error LLQL action and therefore, smaller short-term goal 0.0023. The MPC + modified reward cannot guide the error compared to the DDPG reward modification car to the top.
solution.
Figure 4: MountainCarCominnuus with short-term and long-term goals. 71 1, 72 2000.
5.2 SHORT-TERM TRAJECTORY
Figure 3b shows that the policy presented in equation (16) can lead the car to the top of the mountain using the LLQL algorithm. We can see that the car's velocity is above 0.025 (the red line) when reaches the top. Now consider the case where we want the car to reach the top of the mountain with our desired velocity, 2/ 0.025. Using equation (18), we can design control strategy to reach this goal without requiring retraining the LLQL model. We apply the following hybrid control strategy to reach the top of the mountain with our desired speed.
use equation (16) a < 0 use equation (18), otherwise.
21
(21)
Figure 4a shows that the car can reach the top of the mountain with our desired velocity. When we did not impose our desired speed the system, the car reached the top of the mountain in 78 steps (see Figure 3b). Demanding lower speed. slowed down the car and increased the number of steps 82 (see Figure 4a). Figure 4b shows the actions with and without the short-term trajectory. We can see that the action temporarily becomes negative to reduce the velocity to the desired level and then goes back to positive to push the car to the top of the mountain.
To solve this problem in the traditional way, we had to modify the reward function to achieve both short-term and long-term goals. For comparison, we perform the following experience. We apply DDPG networks and MPC based networks with the modified reward functions shown in Table I to solve the MountainCarConttmuuur with the short-term and long-tew short-term trajectory, have considered different functions of the absolute error between the car's velocity and the desired velocity, [2 2/. in the final stages as an additional penalty. We have tried various architectures and hyper parameters to design the DDPG network. The final parameters presented in Section D. We train a model for each modified reward function up to 300 episodes and save the model with maximum cumulative reward for our experiment. We run each model 10 times and measure the average number of steps it takes to reach the top of the mountain, and the average error between the car's final speed and the desired speed, V/ = 0.025. Table shows that the MPC based solution cannot guide the car to the top. The DDPG with all the modified reward functions can achieve the long-term goal in all the experiences, reaching the top of the mountain in 10 out of 10 experiments. However, the DDPG based solutions do not perform very well with regard to the short- term goal. On the other hand, I.QQI. does not require additional training or reward modification, achieves the long-term goal and has the least velocity error, 0.4%.
goal problem. To achieve the
5.3 SHORT-TERM CONSTRAINT
Now consider the case where it is unsafe to drive the car above a specific speed, for example, we plan to keep the speed under Uy 0.035. We can use the following hybrid control strategy to achieve the
8

---

Under review as a conference paper at ICLR 2020
Table 1: Short-tern term trajectory performance
Average number of steps
Long-term
Average velocity error
Modified Reward function
goal
Method
success rate
DDPG
Twew
Tk
5000
done 0k Uu
0.0232
109.1
10/10
DDPG
Unew
100
1//
0.45
0.0193
183.6
10/10
DDPG
Tnew
100
Ukk
Vil
It. > 0.45
0.0088
173.8
10/10
Tnew Tk
5000
|ex val if done
DDPG
Tnow Tk
25000
(Vk
=M done
0.0193
103.3
10/10
MPC
Tnew = Tk
5000
[7 Ma done
1000
0/10
MPC
Tnew Th
100
Ukk ti
if > 0.445
1000
0/10
MPC
Tnew 7k
100
U/ Val IT; >> 0.45
1000
0/10
Tne
Tk
5000
[uk Val done
MPC
Tnew
25000
(Ue val
if done
1000
0/10
LLQL
0.0001
89.5
10/10
State Variables
DDFG Modileed Revard
Postboo
8.-
IBC Mocifled Roward
NPC Estinsted
-0.5
15
20
1/5
.00
Velocity
0.2
Mecifie
0.9
MPC Eximated
LLOL
Lacee trem
Sh
MDC Mott Fomard
-2.02
LOOL Extinated
000
135 15.0
175
200
(a) With LLQL the car reached the top in 105 steps without (b) Comparing the actions in the normal case ver- violating the constraints. With DDPG modified reward with the short-term constraint. The hybrid (3rd reward in Table 2) the car reached and violated the constraints 13 timesteps. The MPC + mod- ment in LLQL action when the car gets close to ified reward cannot guide the car to the top.
top 199 steps strategy (see equation (22)) leads sharp adjust-
the hazardous areas.
Figure 5: State variables for MountainCarContinun with long-term goal and short-term constraint. The horizontal red lines, |ual 0.035, represent the boundaries. The car reaches the goal in 97 steps.
long-term goal while keeping the speed safe:
use equation (16) if [lpl S 0.033
WA use equation (20), otherwise.
(22)
We selected the boundary slightly less than the hazardous threshold (0.033 instead of 0,035) to be safe. Figure 5a shows that with the LLQL policy the reaches its goal while staying outside of hazardous areas. The MPC based solution keeps the car outside of hazardous areas but cannot deliver the long-term goal (reaching the top of the mountain). The DDPG modified reward reaches the top but fails to deliver the short-term goal (keeping the car out of hazardous areas). Like the previous section, we apply DDPG network + modified reward function and MPC modified reward function to compare LLQL with the traditional model-free and model-based reward engineering approaches. We select the model with maximum cumulative rewards during 300 episodes training. Table 2 shows that unlike LLQL, the modified rewards fail to keep the car below the allowed speed while reaching the top of the mountain. The model-base reinforcement learning algorithm baseline presented in Section E uses the same short-term network as the LLQL network. Table I and Table show that even though the short-term part of our solution is useful achieving short-term goals, not enough to solve the entire problem and achieve the long-term goal.

---

Under review as a conference paper at ICLR 2020
Table 2: Short-tens thort-term constraint performance
Average number of steps out of boundary
Average number of steps
Long-term
RL
Modified Reward function
goal suecess rate
Method
DDPG
Fnow
10
if 04
0.033
21.5
100
I0/10
DDPG
Tnew Fk
100( [4]
0.033) Uy > 0.033
25.2
106.8
10/10
DDPG
Tnew Ph
(100 Uk
0.033)) " if [ex] 0,033
20.4
147.5
10/10
DDPG
Fnew 10 if
ex 0.033
25.1
104.1
I0/10
MPC
Fnew = rkk
lex. > 0.033
0
1000
MPC
Trew
v%. 0.033) if v& > 0.033
0
1000
0/10
MPC
Tnew Fk (100( [/4 0.033))* |ox| > 0.033
1000
0/10
MPC
Fnew 10 if vi 0.033
0
1000
0/10
LLQU
0
98.9
10/10
Short-term Trajectory
Short tert-ttemm Constraint
Number of Steps to the Top
Number of Stops to the Top
320
100
310
110
300
300
0.000
0015
0,20
0,225
08300
0635
D042
2.02
0034
0005 0038 0.40 0012 0044 D046
Desired Velaciity
Velocity Limit
(a) Number of steps to the top desired fi- nal velacity. 71 1, 72 2000.
(b) Number of steps to the top vs velocity constraint.
Figure 6: Long- term performance vs short-term goals. We run the model with each short-term goal 10 times and present the average and standard deviation of the long-term goal.
5.4 EFFECT OF SHORT-TERM GOALS ON LONG-TERM PERFORMANCE
Using equations (18) or (20) for deriving a set actions is equivalent to solving sub-optimum solution for the long-term goal in order to satisfy the short-term desired trajectories constraints. When the short-term goals are far from the global optimum solution, the long-term performance de- grades. Figure 6a shows that lower desired velocities lead to longer traveling time for the Mountain- Car. Similarly, Figure 6b shows that further limiting the maximum velocity degrades the long-term performance.
CONCLUSIONS
In this work, we presented LLQL as a new model-based RL algorithm with the capability of achiev- ing both short-term and long-term goals without requiring complex reward functions. By presenting the advantage function with a locally linear model and separating designing actions from the learn- ing process, our method is capable of adopting control strategies achieve different short-term goals without retraining the model. This can be very significant for industrial applications where the RI. algorithms have not been used due to the necessity of different short-term adjustments.
The LLQL algorithm deviates from the optimal policy temporarily to address local short-term goals (trajectories or constraints). The agent would return to the optimum policy if the deviation small enough that the agent is still in the environment explored during the training. In the future work, we will investigate conditions where short-term goals are feasible and develop a more analytical approach to set the meta parameters for the controller to guarantee short-term and long-term goals. Moreover, we will model uncertainties in short-term prediction model and apply robust control theory design robust control solutions.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Giorgio Bartolini, Alessandro Pisano, and Elio Usai. Second-order sliding mlide cranes. Automatica, 18(10):1833-1790, 2002.
mode control of container
Boustany and Brigitte Andrea Novel. Adaptive control of an overhead crane using dynamic feedback linearization and estimation design. In Proceedings 1992 IEEE International Confer- ence on Robotics and Automation, pp. 1963-1968. IEEE, 1992.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances Neural Information Processing Systems, pp. 9224-8234, 2018.
Wen- Chen, Donald J Ballance, and Peter J Gawthrop. Optimal control of nonlinear systems: predictive control approach. Automatica, 39(4):633 641, 2003.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint rrXiv:/801.0875/ 2018.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value expansion for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018.
Carlos Garcia, David M Prett, and Manfred Morari. Model predictive control: theory and practice- survey, Automatica, 25(3):335 --33,,33,,,1 1989.
Shixiang Gu, Timothy Lillicrap. Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning, pp. 2829- 2838, 2016.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA). pp. 3389-3396. IEEE, 2017.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122 2018.
Harold W Kuhn and Albert Tucker. Nonlinear programming. In Traces and emergence of non-
linear programming, pp. 247-258. Springer, 2014.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv 2015.
preprint
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing. and Sergey Levine, Neural network dynamics for model-basee deep reinforcement learning with model-free fine-tuning In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. JEEE, 2018
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination- -aggmene-- agents for deep reinforcement learning. In Advances in neural information processing systems, pp. 5690-5701, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 1rXiv:171..01815. 2017.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-22)) 1992.
11

---

Under review as a conference paper at ICLR 2020
A RELATED WORK
Our work can be categorized as a new model-based RL approach. Model-based RL algorithms use the environment model which represents the state transition function to plan ahead and select actions that lead to higher rewards. Several model-based algorithms assume the environment model known. Among them, AlphaZero (Silver al.. 2017) one of the most famous. AlphaZero uses the game's s rules (Chess, Shogi and Go) as the environment model to generate series of self- play simulated games. During the simulations, the actions for both players are selected using a Monte-Carke Tree Search (MCTS) algorithm. The MCTS performs as a planning algorithm by generating candidate actions which are superior to the current policy. The neural network parameters are updated at the end of each game to minimize the game prediction error (loss, draw or win) and maximize the similarity of policy vector to the planning algorithm. AlphaZero is limited to the discrete action space problems. The environment model is typically unknown in real-world applications. Therefore, many model-based RL. algorithms learn the state transition model from the data. NAF (Gu et al., 2016) learns a linear model for state transition each operating point and uses this model to generate additional samples through imagination rollout. World Models (Ha & Schmidhuber, 2018) uses Variational Auto Encoder (VAE) to map a state variable, E X to lower dimensional variable z in a latent space Z. It then uses recurrent neural network (RNN) to learn the state transition model in the latent space. Finally, it applies a simple linear controller and the hidden state in the RNN, h, to control the system.
Imagination- -Aggmmennd Agents (/2As) (Racanière al., 2017) introduces two paths: 1) model-free path and 2) imagination path. The imagination path learns transition model and uses this model to generate imagination rollouts. These rollouts are aggregated with the samples in the model-free path. To generate actions in the imagination path, 12As uses the model-free path policy. Therefore, the rollouts in the imagination path improve as the 12As policy improves. Using the imagination rollouts, 12As converge faster than model-free network with the same number of parameters. Nagabandi et al. (2018) showed that a two-step control policy based on 1) learning the dynamic model and 2) applying MPC the learned model is significantly more sample efficient than model-free RL. However, this approach cannot achieve high rewards. To achieve higher rewards and preserve sample efficiency, they proposed hybrid model-based and model-free (MBMF) algorithm which runs the model-based approach to achieve the initial result in a sample efficient way, it then trains a model- free policy mimic the learned model-based controller, and uses the resulting imitation policy as the initialization for the final model-free RL algorithm.
Feinberg et al. (2018) proposed Model-based Value Expansion (MVE) algorithm, which limits the uncertainty in the model by only allowing imagination up to a fixed number of steps, H. MVE uses the learned system dynamic model to generate simulation data up to H steps into the future, and applies these sample points to estimate Q-function. Instead of saving simulated samples in imag- ination buffer, MVE retrains the dynamic model and generates a new imagination rollout at each step. Buckman et al. (2018) expanded MVE algorithm by proposing Stochastic Ensemble Value Expansion (STEVE), to generate a solution more robust to model uncertainty. Dalal et al. (2018) proposed safe exploration by modeling constraints using linear model and applied Lagrangian op- timization to modify the action in order to guarantee safety. In this work, we also used Lagrangian optimization for short-term constraints. However, our approach is different in two ways: 1) our method does not modify the RL action to achieve the goals. Instead, it derives an action by consid- ering both long-term goals and short-term constraints. This is possible because our algorithm uses a locally linear model to represent the advantage function. 2) Unlike safe exploration, the focus of this paper is in handling new constraints in the application phase without retraining the model.
12

---

Under review as a conference paper at ICLR 2020
200
100
Cumulative Reward
-200
-300
-400
-500
-600
100
125
150
175
Episod
(a) Cumulative rewards for LLQL.
10.00
1000
8042
0.0
4.00
-280
17
Time
Time
(b) State variables for the crane system. The crane reaches the goal in 12s.
Figure LLQL for the crane system.
B CRANE CONTROL SYSTEM
Gantry cranes widely used in industrial production lines and construction projects for transferring heavy and hazardous materials. The objective of the crane system is to convey the payload from the start position the destination position as soon as possible while keeping the payload sway the destination minimum. Higher traveling speed improves the efficiency and reduces costs. However, excessive movements at the destination wastes time and energy and can lead to accidents. To move the payload as fast as possible and stop the sway at the destination, skillful operators are required.
Labor shortage industries, and risk of human error, have motivated us to develop an automated solution for crane control. The crane dynamic system is highly nonlinear. Traditional nonlinear control techniques such as sliding control (Bartolini et al., 2002) and adaptive control (Boustany & d' Andrea Novel, 1992) have been applied to these systems. These methods require detailed mathematical model of the system and its environment, which can be complicated and expensive to derive. When simulator is available for alternative to traditional control methodologies. This is the case in many industries, where for intellectual property concerns the companies are willing to provide simulators the costumers but refuse to reveal mathematical models of their products.
system, RL algorithms
provide compelling
Our crane simulator provides six state variables: 1) trolley location, Ttrolley 2) trolley velocity, Utrolley. 3) payload angle. Opayload, 4) payload angular velocity, Wpaylood- 5) payload horizontal location, Zpayloads and 6) payload vertical location, Speyload- The only action the force applied to the trolley. Utrolley- The overall goal is to reach the final destination Tpd and ypd in the shortest time possible. We choose the following reward function to learn a policy do so.
500 if (Tpaylooa(kk) Fpdl & [3payylond ynal < otherwise,
Fk
(23)
13

---

Under review as a conference paper at ICLR 2020
where is small constant. Figure 7b shows that learned policy pushes the trolley with maximum force, Utrollyy eqaual for the entire episode, till the payload reaches the goal End 6, 3pd = 10.
In additional to the long-term goal, our short-term goal to minimize the object's sway when it reaches the final destination. Instead of designing complicated reward functions to achieve minimum travel time and minimum sway, we consider 4poylood = 0 at the final destination as a short-term desired trajectory. We consider the following hybrid strategy reach the final destination with close to zero sway.
use equation (16) if payloon < 5.5
Utrolley
use equation (18), otherwise
(24)
Figure shows that our strategy can reach the destination with close zero swing.
12.00
Xalay/owd
Vplay/oad
3:55
05
Uiyoltey
-2.0
-25
-006
Time
Time
Figure 8: State variables for the crane system with required short-term trajectory, Wpayload = 0. The final error is 0.0001. The crane reaches the goal in 12.5s. 71 = 1, 72 = 1000.
14

---

Under review as a conference paper at ICLR 2020
C DERIVATION DETAILS FOR SHORT-TERM CONSTRAINTS
Consider equation (19). Substituting our estimation the next step from equation (14) in (19), we have
1
min (hu dx
lindia 2
(25)
such that:
# A gkus) S cè+1
Using Lagrangian method at each time step k, we have
1
L(uk- A)
2 (hx + 2x )²
(26)
A(x + Afh + gi,ux) ck+1)
Taking the gradient of L with respect to uk, we can write the Karush-Kult Kuhn-Tucker (KKT) (Kuhn & Tucker, 2014) conditions for optimal solution of equation (25), {uzz )"} as:
(he + de + +1* (0 Ash gf Ck+1)
(27)
With this assumption, show
u (d. d+) d (he (hk + X" A' (dxc%))
(28)
Note that when there no constraint: input we computed in equation (16). When X & O, we have
O, we have ux = -dp dx) dl hg. This exactly the
1 + Afk + g Cb+1 =0
(29)
We define a = Ag di (dad))- and as Ag (df (df dz) -'dff an and ag scalar. Substituting
un from equation (28) in equation (29) we have
# + Afk a24 + a1 C++1 = 0
(30)
Therefore,
1 + Dfj C++1 azh4
A
0100
(31)
(df dx) di (h+ + x²a)).
15

---

Under review as a conference paper at ICLR 2020
D NETWORKS PARAMETERS
We used the following network structures and parameters in the experimental studies.
LLQLfor MountainCarContiiinn
h, d, V. and g networks each has two hidden layers with 200 neurons in each layer. All the activation functions are Rectified Linear Units (ReLUs). Each episode maximum 1000 steps. The number of iterations for short-term and long-term prediction model: I. = 11 5. The learning rate for the long reggtterm prediction model is 0.001. The batch size for this model 10. The discount rate y 0.999. The target model update rate, T 0.001. The learning rate for the short-term prediction model is 0.001 for the first 20000 steps and then reduces to 0.0001. A 0.001. The batch size for this model is 100.
LLQ the crane system: h. d, V. and networks each has two hidden layers with 200 neurons each layer. All the activation functions are ReLUs. Each episode includes maximum 1000 actions. The number of iterations for short-term and long-term prediction model: I, I 5. The learning rate for the long-term prediction model is 0.001. The batch size for this model rate 0.999. The target model update rate, 0.001. The learning rate for the short-term prediction model is 0.01 for the first 20000 steps and then reduces to 0.001. size for this model is 200.
The discount
0.001. The batch
DDPG networks with modified reward functions: The Q-network, and the deterministic policy net- work each has two hidden layers with 200 neurons. All the activation functions are ReLUs. The learning rate for the Q-network is 0.00001, and the learning rate for the deterministic policy network 0.000001. The discount rate 7 0.99. The batch size is 8. The target model update rate for both networks is 0.1.
In all the networks we shift and scale the state variables zero mean and unit standard deviation for a better learning. For exploration, we add an additive normal noise to the action:
m(k) = (1 - aN (kk))* aNN,,
(32)
where ukk) represents the optimum action generated by LLQL or DDPG, and V U(-1,1) is continuous uniform random variable. We consider = 0.05 in the beginning, and a.N .99x x9x0 after each episode with positive cumulative rewards.
16

---

Under review as a conference paper at ICLR 2020
E MODEL-BASED REINFORCEMENT LEARNING
Nagabandi et al. (2018) proposed a model-based RL based on MPC. Their approach uses the system dynamic model reward function to generate sequence of actions that maximize the cumulative reward. To represent the effect of short-term and long-term predictive models in our algorithm, we present modified version of (Nagabandi et al., 2018) algorithm using our short-term predictive model, f and in equation (14). The model based RL uses the leamned dynamic model, f and networks, and generates a sequence of actions U% (Uk:.. Uk+ H--) to maximize reward over finite horizon, H.
k+
U = max Σ k=
r(ā;, ui),
(33)
such that: fg zy and 2:41 2, +
Solving equation (33) for the exact solution is computationally expensive. Therefore, Nagabandi et al. (2018) applied a simple random sampling method and select the candidate action sequence with the highest expected cumulative reward. In this work, apply the method to solve equation (33). We set H = 25 and select 10000 samples at each step. Following the MPC closed- loop control framework, the algorithm only executes the first action the sequence and calculate new sequence in the next step. Algorithm presents the model-based RL.
Algorithm Model-based Reinforcement Learning (Nagabandi et al., 2018)
1:
Initialize ang networks (equation (14)) with random weights. Create dataset random trajectories Brand-
3: Create the reply buffer R, =
4: for episode = 1:M do
5: Randomly select mini-batch of N. transition from Ryand and Rys.
6: Update g and 8 by minimizing the loss: Li = N, Ei ||2111 - 2: (xf(x:)))) + (((0)))))(00)))))))))))))))))))) [x:;)10))))))
=
Solve equation (33) for the optimum sequence U at state Th.
9: Exccutes the first action in the sequence, Wk.
10:
Store transition (It, ug 1 r')) in Ryi.
17


Reviews:
[{"review_id": "rJxHfKlVqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Locally Linear Q-Learning (LLQL) method for continuous action control. It uses a short-term prediction model and a long-term prediction model to generate actions that achieve short-term and long-term goals simultaneously. The problem the paper seeks to solve is important. However, this paper has several issues.\n\nFirst, there seems to be an over-claim of the contribution. The proposed method is more like hybrid of model-based and model-free RL method. Specifically, the short-term prediction model is in fact the linearized dynamic system with system parameters modeled by deep neural networks, while the long-term prediction model is in fact different (state- and action-) value functions. For this reason, it is probably unnecessary to name them as “short-term network” or “long-term network”, since they are simply the system model (or the model-based part) and the value functions (the model-free part).\n\nSecond, the proposed method is not sufficiently evaluated. It is only evaluated on the toy Mountain Car task (and the Crane system in supplementary material). In order to justify the performance of an RL algorithm for continuous action space, it should be at least evaluated on the set of MuJoCo tasks. \n\nThird, the proposed method is not sufficiently compared with different baselines. In Figures 3-5, the proposed LLQL algorithm is never compared to any baseline method, leaving it open whether it is actually better than earlier methods like DDPG. In Table 1, LLQL is compared to DDPG (a model-free method), and is shown to achieve better performance. However, this seems to be unfair because the proposed method is in fact a model-based RL algorithm. Therefore, it should at least compare to other model-based algorithms (and also other riche set of safe-exploration RL methods).\n\nOther comments:\n•\tIn eqn. (6), \\gamma should be \\gamma^{i-k}?\n•\tIn the paragraph after (8), “Q-learning algorithms (16)…” is referring to a wrong equation (16) for Q-learning. Or probably the authors are not using the correct format to cite the reference. (This seems to happen repeatedly in later part of the paper such as as in the paragraph between (14) and (15).) It confuses the equation number and the reference number.\n•\tMore explanation should be given about d(x_k|\\theta^d) and h(x_k|\\theta^h) after (15). The meaning of them has never to defined before.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1x5kUA3FB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for combining continuous Q learning and linear MPC. They propose learning a continuous control policy for long term behavior trained with standard model free reinforcement learning and a short term linear control model implemented as a linear dynamical system. They propose learning them separately each with its own loss function. At test time a combined controller can be used to infer actions by solving small optimization problem. The linearity makes inference very efficient and even when additional constraints are added only at test time. The model should therefore me more flexible and allows for adapting 0 shot when the constraints are known. The paper is well written and mostly clear. \n\nThe equations seem correct but I am not an expert in continuous control. The examples considered seem a bit too simple to be insightful. \n\n* the algorithm seems to be be setup to with the dynamics model being learnt on the transitions coming from an epsilon greedy with respect to the optimal policy. Does this mean that generalization may be affected if the model strays off the optimal trajectories ?\n* There would be some benefit in trying the model in some other domain maybe reacher with some constraints ?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlA08HWcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thank the authors for the response. I will keep my score.\n----------------------------------------\nSummary\nIn this paper, the author proposes LLQL (Locally Linear Q-Learning), which separates the action design from prediction models, such that the model can have short-term and long-term goals. The short-term network takes the current state and the action, and predicts the next state (to be specific, the status change) under the assumption that the output is linear to the action. The long-term network takes the current state as input, controls the controller to take action, and also optimizes a Q-function. I lean to vote for accepting this paper, though the experiment part can be further improved.\nStrengths\n- The proposed model is novel, which can take a trajectory or a constraint as a short-term goal. As a result, we can control the behavior of the model easily. Otherwise, we have to design a new reward function to guide our model indirectly. As the authors showed in Tables 1 & 2, LLQL outperforms this approach.\n- The paper is clearly written. The author provides clean formulas throughout the paper, which makes the paper easy to understand.\nWeaknesses\nMore experiments can be conducted to demonstrate the effectiveness of LLQL.\n- Only tested in one scenario. In the main text, though it is already very long, there are still only experiments in one scenario (i.e., Mountain Car).\n- Only compared to one baseline. In order to demonstrate the effectiveness of LLQL taking a short-term trajectory or constraint, the authors compared it to only one method (i.e., DDPG) with a few manually designed reward function.\nMinor\nEquation 6 looks incorrect to me.\nPossible Improvements\nAs mentioned before, it would be great to include more experiments in the main text, comparing LLQL to more baselines, testing LLQL in more scenarios, or doing some ablation studies (removing one part of LLQL to demonstrates the effectiveness of that part).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a reinforcement learning algorithm for continuous action domains that combines a short-horizon model-based objective and a long-horizon value estimate.  The stated benefits to this approach are the ability to modify the model-based objective without extensive retraining.  This model-based objective can also capture custom constraints and implements a linear dynamics model that is used in conventional control theory.  The proposed method was evaluated on two domains (the mountain car domain and a custom crane domain) and compared to a continuous action space method (DDPG).  

This discussion of this paper highlighted both strengths and weaknesses.  The reviewers said the presentation was clear.  The reviewers also appreciated the relevance of the problem.  The primary weakness was the evaluation of the method. One repeated concern from the reviewers was having only one standard domain for evaluation (mountain car).  Another concern was the absence of other model-based algorithms, which was addressed by the author response.   

This paper is not yet ready to be published, despite its possible benefits,  due to the lack of evidence for this method on more continuous action problems.   
Decision: Reject