Title: LONG-TERM PLANNING, SHORT-TERM ADJUSTMENTS

Abstract: Deep reinforcement learning (RL) algorithms can learn complex policies to optimize agent operation over time. RL algorithms have shown promising results in solving complicated problems in recent years. However, their application on real-world physical systems remains limited. Despite the advancements in RL algorithms, the industries often prefer traditional control strategies. Traditional methods are simple, computationally efficient and easy to adjust. In this paper, we propose a new Q-learning algorithm for continuous action space, which can bridge the control and RL algorithms and bring us the best of both worlds. Our method can learn complex policies to achieve long-term goals and at the same time it can be easily adjusted to address short-term requirements without retraining. We achieve this by modeling both short-term and long-term prediction models. The short-term prediction model represents the estimation of the system dynamic while the long-term prediction model represents the Q-value. The case studies demonstrate that our proposed method can achieve short-term and long-term goals without complex reward functions.

1. INTRODUCTION:

Optimal control methodologies use system dynamic equations to design actions that minimize desired cost functions. A cost function can be designed to track a trajectory, reach a goal, or avoid obstacles. It is also possible to design a cost function to achieve a combination of goals. Model Predictive Control (MPC) is a common optimal control technique and has been applied to many industrial applications such as pressure control and temperature control in chemical processes (Garcia et al., 1989) . The traditional control solutions are not adequate to address the challenges raised with the evolution of industrial systems. Recently, deep reinforcement learning (RL) has shown promising results in solving complex problems. For example, it has generated superhuman performance in chess and shogi (Silver et al., 2017) . The following advantages make deep RL a strong candidate to overcome traditional control limitations. First, deep RL has an advantage in solving complex problems, especially when the consequences of an action are not immediately obvious. Moreover, it can learn an optimal solution without requiring detailed knowledge of the systems or their engineering designs. Finally, deep RL is not limited to time-series sensors and can use new sensors such as vision for a better control. However, deep RL has not been applied to address industrial problems in a meaningful way. There are several key issues that limit the application of deep RL to real-world problems. Deep RL al-gorithms typically require many samples during training (sample complexity). Sample complexity leads to high computational costs. A high computational cost can be justified for industries as a onetime charge. However, oftentimes small changes in the system goal, such as changing the desired temperature in a chemical reactor, or a new constraint such as a maximum allowable temperature in the reactor, require retraining the model. Moreover, industrial systems often have several short-term and long-term objectives. For example, consider the crane system shown in Figure 1 . The long-term goal is to convey the payload to the target location as soon as possible. However, when the payload gets close to the target, it must have minimum sway for the safety of the operators. Designing a reward function that can capture these short-term and long-term goals concurrently can be challenging or even infeasible.

A class of short-term objectives related to safe exploration during RL training have been studied recently. Gu et al. (2017) presented an application of deep RL for robotic manipulation control.

To ensure safe exploration, they set maximum limits for the joint positions, and joint velocities. Moreover, they set a sphere boundary for the end-effector position and when the boundaries were about to be violated, they used correction velocity to force the end-effector position back to the center of the sphere. Dalal et al. (2018) formulated the safe exploration as an optimization problem.

They proposed to add a safety layer that modifies the action at each time step. Toward this end, they learn the constraint function using a linear model and use this model to find the minimal change to the action such that the safety constraints are met at each time step. To the best of our knowledge, there is no study addressing short-term objectives during application.

In this paper, we present a Locally Linear Q-Learning (LLQL) algorithm for continuous action space. The LLQL includes a short-term prediction model, a long-term prediction model, and a controller. The short-term prediction model represents a locally linear model of the dynamic system, while the long-term prediction model represents the value function + a locally linear advantage function. The controller uses the short-term prediction model and the long-term prediction model to generate actions that achieve short-term and long-term goals simultaneously. It adopts a policy that maximizes Q-value while achieving short-term goals. The LLQL algorithm has the following advantages:

• It does not require designing sensitive reward functions for achieving short-term and longterm goals concurrently.

• It shows better performance in achieving short-term and long-term goals compared to the traditional reward modification methods.

• It is possible to modify the short-term goals without time-consuming retraining.

The remainder of this paper is organized as follows. Section 2 represents the background in dynamic systems and RL. Section 3 represents the LLQL algorithm. Section 4 presents our methodology to achieve short-term and long-term goals using LLQL. Section 5 presents our experimental results. Section 6 presents the conclusions of the paper. Section A discusses the related work. Section B presents additional experiments for those interested.

2. BACKGROUND AND DEFINITIONS:

In this section, we review the backgrounds in dynamic systems and reinforcement learning.

2.1. DYNAMIC SYSTEMS:

A continuous-time dynamic system can be represented as:

EQUATION

where given the system parameters, p, f maps the state variables, x 2 X, and actions, u 2 U , to the state derivative, dx dt at time t. In state space control, the goal is to design a control policy, ⇡ control (u(t)|x(t)), that generates proper actions so as the state variables follow the given desired trajectory, x d (t). It is challenging to design a control policy for a nonlinear complex system represented in equation (1).

The control problem becomes much easier to address when this system is linear with respect to the input (Chen et al., 2003) . We can present these systems as:

EQUATION

Since measurements are typically sampled in discrete times, we derive a discrete time version of linear system (2). Using a first-order approximation:

EQUATION

where t k represents time at sample point k. In this paper, we assume the sampling rate is constant; = t k+1 t k . Using ( 2) and (3), we have:

EQUATION

For brevity, we present t k by k, f (x(t k )) by f (x k ), and g(x(t k )) by g(x k ) in the remainder of the paper. Therefore, we can represent a dynamic system as:

x k+1 = x k + (f (x k ) + g(x k )u k ).

(5)

2.2. REINFORCEMENT LEARNING:

The goal of RL is to learn a policy, ⇡ RL (u k |x k ), that generates a set of actions, u 2 U , that maximize the expected sum of rewards in the environment, E n . Consider:

EQUATION

where < 1 is the discount factor, r is the reward function and T represents the end time and can be set to T = 1. The goal is to learn ⇡ RL for environment, E n , such that:

EQUATION

Unlike control algorithms, model-free reinforcement learning algorithms assume the system dynamic is unknown. Q-function, Q ⇡ (x k , u k ) is defined as the expected return at state x k when we take action u k and adopt policy ⇡ afterward:

EQUATION

Q-learning algorithms (Watkins & Dayan, 1992) are among the most common model-free RL methods for discrete action space problems. These algorithms use the Bellman recursive equation to model Q-function:

EQUATION

where µ represents a greedy deterministic policy that selects the action which maximizes Q-value at each step:

EQUATION

Q-learning algorithms learn the parameters of the function approximator, ✓ Q , by minimizing the Bellman error:

EQUATION

where y k is the fixed target Q-function, and represents the exploration policy.

For continuous action domain problems, it is not trivial to solve equation (10) at each time step. Finding an action to maximize Q which can be a complex nonlinear function is computationally expensive or even infeasible. To address this problem, Lillicrap et al. (2015) proposed the Deep Deterministic Policy Gradient (DDPG) algorithm, which learns two networks simultaneously. The critic network learns Q-function by minimizing the Bellman error, and the actor network learns parameters of the policy to maximize the estimated value of Q-function. Gu et al. (2016) proposed Normalized Advantage Function (NAF) Q-learning which formulates the Q-function as the sum of the value function, V (x), and the advantage function, A(x, u).

EQUATION

where

EQUATION

P (x|✓ P ) = L(x|✓ P )L(x|✓ P ) T , where L(x|✓ P ) is a lower-triangular matrix. The value function is not a function of action, u. Therefore, the action which maximizes advantage function, A, maximizes the Q function. P (x|✓ P ) is a positive-definite matrix, and therefore, the action that maximizes the advantage function and the Q-function is given by µ(x|✓ u ).

3 LOCALLY LINEAR Q-LEARNING In this section, we propose the LLQL algorithm, which like (Lillicrap et al., 2015) and (Gu et al., 2016) can handle continuous action space. Our approach learns short-term and long-term prediction models. Using the long-term and short-term models, a controller generates actions that guide the system toward its short-term and long-term goals. Figure 2 shows our proposed structure to learn the parameters of the short-term and long-term prediction models.

Short-term prediction: consider the nonlinear system presented in equation ( 5). In this work, we use deep neural networks to estimate system functions, f (x k ), and g(x k ) at each operating point. Substituting the network estimations for these functions in equation ( 5), we can predict the next state as:

EQUATION

where xk+1 represents our estimation of the next step, and ✓ f and ✓ g are the network parameters. is a constant hyper parameter. In dynamic systems, the difference between two consecutive states, x k+1 x k , is typically very small. Considering a small leads to reasonable f and g values and, therefore, improves learning time and accuracy.

We call this dynamic system model short-term prediction model. The controller uses this model to generate actions, which lead the system toward its short-term goals. Note that previous work have used the system short-term dynamic model for generating additional samples in imagination rollout (for example, see (Gu et al., 2016) , and (Racanière et al., 2017) ). In this paper, we show that this model can also be used to design actions to achieve short-term goals. To learn the parameters of our short-term prediction model, ✓ f and ✓ g , we minimize the short-term loss function, L 1 , as it is presented in Algorithm 1.

Long-term prediction: Q-function represents the maximum cumulative reward that can be achieved from current state, x k , taking an action u k . Therefore, by learning Q-function, we learn the longterm prediction model for the system. Like NAF (Gu et al., 2016) (see equation ( 12)), we present Q-function as a sum of value function and advantage function. However, we present the advantage function, A(x, u|✓ A ) using a locally linear function of x k and u k as: To maximize Q-function, we have to design u k which minimizes h(

EQUATION

x k |✓ h ) + d(x k |✓ d )u k .

For simplicity, we present h(x k |✓ h ), and d(x k |✓ d ) with h k and d k respectively in the remainder of the paper. To maximize Q-function and achieve the long-term goal, we can use simple pseudo-inverse matrix multiplication and derive a solution with the least squares error as:

EQUATION

)

When ||d k || = 0, it means the network predicts that our action has no impact on the advantage function. Therefore, we choose a random action. Random exploration is an important part of any deep RL algorithm. Therefore, in addition to this unlikely case, we add noise, N k , to the action, u k , during the training. We reduce the amount of noise injected to the action as the algorithm converges.

Algorithm 1 Locally Linear Q-Learning Training 1: Initialize Q network (equation ( 15)) with random weights.

2: Initialize target network, Q 0 , parameters: ✓ Q 0 = ✓ Q . 3: Create the reply buffer R = ;. 4: for episode = 1:M do 5:

Initialize a random process N for action exploration. 6:

Receive the initial observation, x0.

7::

for k = 1:T do 8:

if

||d k || 6 = 0 then 9: Set u k = (d T k d k ) 1 (d T k )h k + N k 10: else Set u k = N k 11:

Execute u k and observe x k+1 and r k . 12:

Store transition (x k , u k , x k+1 , r k ) in R.

13::

for iteration = 1:Is do 14:

Randomly select a mini-batch of Ns transition from R.

15::

Update ✓ f and ✓ g by minimizing the loss:

L1 = 1 Ns P Ns i=1 ||xi+1 xi (f (xi|✓ f ) + g(xi|✓ g )ui)||.

16::

for iteration = 1:I l do 17:

Randomly select a mini-batch of N l transition from R.

18::

Set yi = ri + Q 0 (xi+1|✓ Q 0 ). 19:

Update ✓ Q by minimizing the loss: L2 = 1

N l P N l i=1 ||yi Q(xi, ui|✓ Q )||. 20:

Update the target network:

✓ Q 0 = ⌧ ✓ Q + (1 ⌧ )✓ Q 0

In the application, the controller solves u k with additional constraints to achieve the desired shortterm trajectories. We will discuss our short-term adjustment algorithms in the next section. To learn Q-function, in addition to the state estimation error, we minimize the long-term loss function, L 2 , as it is presented in Algorithm 1. Note that having the short-term model, it is straightforward to add imagination rollout to our algorithm to increase sample efficiency. However, improving sample efficiency in RL is not the focus of this work.

4. CONTROL STRATEGY:

By separating action design from prediction models, LLQL gives us the freedom to design different control strategies for achieving short-term and long-term goals. Moreover, the linear structure of short-term and long-term models simplifies the control design. Consider the case where LLQL has learned a perfect long-term model for an environment using Algorithm 1. In this case, the optimum solution to achieve the long-term goal is given by equation ( 16). When we have one or more shortterm goals as well, we can formulate the control design as an optimization problem to satisfy both short-term and long-term goals as much as possible.

In this paper, we consider two types of short-term goals: 1) desired trajectory, and 2) constraint. In the first scenario, the agent has a short-term desired trajectory. For example, a car may be required to travel with specific speed during certain periods. In the second scenario, the agent has some limitation for a specific period of time. For example, a car is required to keep its speed below certain thresholds at some periods during the trip. To address the first problem, we add an additional term to the cost function for the short-term goal and solve for the action. We deal with the second problem as a constraint optimization.

4.1. SHORT-TERM TRAJECTORY:

Let x d represent our desired short-term trajectory. We develop a control strategy to track x d while pursuing the long-term goals.

Using system dynamic functions f k and g k , we can define our control optimization problem as:

EQUATION

where x d(k+1) represents the desired trajectory at time k + 1. 1 and 2 are positive coefficients and can be adjusted to give higher weights to the short-term or long-term goals. Note that in this work, we assume the short-term goals are temporary and when their time expires the system goes to the long-term optimum policy given by ( 16). For example, we may require a car to have a specific speed at some specific locations.

We can apply a similar pseudo-inverse matrix multiplication, and derive a solution with the least squares error for (17) as:

EQUATION

)

4.2. SHORT-TERM CONSTRAINT:

The LLQL algorithm provides a framework to design the actions considering different constraints.

For safe operation, the agent may have to avoid specific states for a period of time (for example, high speed or locations close to an obstacle). For simplicity, we assume at each moment we only have maximum one constraint on one state variable, x i . This is a reasonable assumption, because in physical systems the agent is close to one of the boundaries at any moment in time. When this is not the case, we can define new constraints as a combination of constraints. Consider c i k as the constraint on the state variable, x i , at time k. We can define the constraint optimization problem for LLQL as:

min findu k 1 2 (h k + d k u k ) 2

such that:

EQUATION

)

1 2 is a coefficient added to simplify the mathematical operation. Using our estimation of the next step,

x i k+1 = x i k + (f i k + g i k u k ),

we can derive the optimum action which satisfies the constraint as:

EQUATION

where

↵ 1 = g i k d T k (d k d T k ) 1 , ↵ 2 = g i k (d T k d k ) 1 d T k , and ⇤ = x i k + f i k c k+1 ↵2h k ↵1↵2

. The derivation details for short-term constraints are presented in Section C. 

5. EXPERIMENTAL RESULTS:

In this section, we demonstrate the performance of LLQL using Mountain Car with Continuous Action (MountainCarContinuous) from OpenAI Gym1 . In Section B, we apply LLQL to control the crane system shown in Figure 1 .

5.1. MOUNTAINCARCONTINUOUS:

The MountainCarContinuous has two state variables: 1) car's position 1.2  x k  0.5 and 2) car's velocity 0.07  v k  0.07. u k is the continuous action at time k. A negative action pushes the car to the left and a positive action pushes the car to the right. The experience stops after 1, 000 steps or when the car reaches the goal on top of the mountain, x k = 0.5, whichever occurs first. In the beginning of each episode, the car is randomly positioned at 0.6  x 0  0.4. The reward for each episode is 100 for reaching the goal on top of the mountain minus the squared sum of actions from start to the goal. Figure 3a shows the cumulative rewards during the training for the LLQL, the DDPG, and a model-based reinforcement learning based on MPC presented by (Nagabandi et al., 2018) . The details of the MPC based solution is presented in Section E. For each approach we performed the training 20 times, and selected the top 5 models with the maximum cumulative rewards to calculate mean and standard deviation of each episode in Figure 3a .

The MPC based solution uses the learned short-term predictive model (system dynamic model) to generate a sequence of actions that maximize the reward over a finite horizon. Figure 3b shows that the short-term predictive model estimates future states with high precision. However, optimizing for a finite horizon is a disadvantage for the model-based solution in achieving long-term goals. Increasing the horizon may improve the long-term performance, but it also increases the computational costs in the application phase. In our experiments, the car never reached the top of the mountain using the model-based method. Figure 3b presents the first 110 steps of a sample experiment. Unlike the MPC based solution, the LLQL and the DDPG algorithms converged in less than 40 episodes (see Figure 3a ) and reach the top of the mountain in all experiments (see Figure 3b ). Note that it is possible to improve the LLQL convergence time by applying imagination rollout. In fact, our short-term prediction model can be used to generate imaginary scenarios. However, sample efficiency is beyond the focus of this work. Figure 3 shows by applying the long-term predictive model, the LLQL algorithm outperforms the model-based reinforcement algorithm in achieving the long-term goals. In the next two subsections, we will show the LLQL algorithm can outperform the DDPG algorithm in achieving the short-term goals by using the short-term predictive model. 

Figure 3b shows that the policy presented in equation ( 16) can lead the car to the top of the mountain using the LLQL algorithm. We can see that the car's velocity is above 0.025 (the red line) when it reaches the top. Now consider the case where we want the car to reach the top of the mountain with our desired velocity, v d = 0.025. Using equation ( 18), we can design a control strategy to reach this goal without requiring retraining the LLQL model. We apply the following hybrid control strategy to reach the top of the mountain with our desired speed.

EQUATION

Figure 4a shows that the car can reach the top of the mountain with our desired velocity. When we did not impose our desired speed to the system, the car reached the top of the mountain in 78 steps (see Figure 3b ). Demanding a lower speed, slowed down the car and increased the number of steps to 82 (see Figure 4a ). Figure 4b shows the actions with and without the short-term trajectory. We can see that the action temporarily becomes negative to reduce the velocity to the desired level and then goes back to positive to push the car to the top of the mountain.

To solve this problem in the traditional way, we had to modify the reward function to achieve both short-term and long-term goals. For comparison, we perform the following experience. We apply DDPG networks and MPC based networks with the modified reward functions shown in Table 1 to solve the MountainCarContinuous with the short-term and long-term goal problem. To achieve the short-term trajectory, we have considered different functions of the absolute error between the car's velocity and the desired velocity, |v d v k | in the final stages as an additional penalty. We have tried various architectures and hyper parameters to design the DDPG network. The final parameters are presented in Section D. We train a model for each modified reward function up to 300 episodes and save the model with maximum cumulative reward for our experiment. We run each model 10 times and measure the average number of steps it takes to reach the top of the mountain, and the average error between the car's final speed and the desired speed, v d = 0.025. Table 1 shows that the MPC based solution cannot guide the car to the top. The DDPG with all the modified reward functions can achieve the long-term goal in all the experiences, reaching the top of the mountain in 10 out of 10 experiments. However, the DDPG based solutions do not perform very well with regard to the shortterm goal. On the other hand, LLQL does not require additional training or reward modification, achieves the long-term goal and has the least velocity error, 0.4%.

Now consider the case where it is unsafe to drive the car above a specific speed, for example, we plan to keep the speed under v k  0.035. We can use the following hybrid control strategy to achieve the long-term goal while keeping the speed safe:

DDPG rnew = r k 100 ⇤ |v k v d | if x k > 0.45 rnew = r k 5000 ⇤ |v k v d | if done 0.0088 173.8 10/10 DDPG rnew = r k 25000 ⇤ (v k v d ) 2 if done 0.0193 103.3 10/10 MPC rnew = r k 5000 ⇤ |v k v d | if done - 1000 0/10 MPC rnew = r k 100 ⇤ |v k v d | if x k > 0.45 - 1000 0/10 MPC rnew = r k 100 ⇤ |v k v d | if x k > 0.45 rnew = r k 5000 ⇤ |v k v d | if done - 1000 0/10 MPC rnew = r k 25000 ⇤ (v k v d ) 2 if

EQUATION

)

We selected the boundary slightly less than the hazardous threshold (0.033 instead of 0.035) to be safe. Figure 5a shows that with the LLQL policy the car reaches its goal while staying outside of hazardous areas. The MPC based solution keeps the car outside of hazardous areas but cannot deliver the long-term goal (reaching the top of the mountain). The DDPG + modified reward reaches the top but fails to deliver the short-term goal (keeping the car out of hazardous areas). Like the previous section, we apply DDPG network + modified reward function and MPC + modified reward function to compare LLQL with the traditional model-free and model-based reward engineering approaches. We select the model with maximum cumulative rewards during 300 episodes of training. Table 2 shows that unlike LLQL, the modified rewards fail to keep the car below the allowed speed while reaching the top of the mountain. The model-based reinforcement learning algorithm baseline presented in Section E uses the same short-term network as the LLQL network. Table 1 and Table 2 show that even though the short-term part of our solution is useful in achieving short-term goals, it is not enough to solve the entire problem and achieve the long-term goal. 

5.4. EFFECT OF SHORT-TERM GOALS ON LONG-TERM PERFORMANCE:

Using equations ( 18) or (20) for deriving a set of actions is equivalent to solving a sub-optimum solution for the long-term goal in order to satisfy the short-term desired trajectories or constraints.

When the short-term goals are far from the global optimum solution, the long-term performance degrades. Figure 6a shows that lower desired velocities lead to longer traveling time for the Mountain-Car. Similarly, Figure 6b shows that further limiting the maximum velocity degrades the long-term performance.

6. CONCLUSIONS:

In this work, we presented LLQL as a new model-based RL algorithm with the capability of achieving both short-term and long-term goals without requiring complex reward functions. By presenting the advantage function with a locally linear model and separating designing actions from the learning process, our method is capable of adopting control strategies to achieve different short-term goals without retraining the model. This can be very significant for industrial applications where the RL algorithms have not been used due to the necessity of different short-term adjustments.

The LLQL algorithm deviates from the optimal policy temporarily to address local short-term goals (trajectories or constraints). The agent would return to the optimum policy if the deviation is small enough that the agent is still in the environment explored during the training. In the future work, we will investigate conditions where short-term goals are feasible and develop a more analytical approach to set the meta parameters for the controller to guarantee short-term and long-term goals.

Moreover, we will model uncertainties in short-term prediction model and apply robust control theory to design robust control solutions.

A RELATED WORK:

Our work can be categorized as a new model-based RL approach. Model-based RL algorithms use the environment model which represents the state transition function to plan ahead and select actions that lead to higher rewards. Several model-based algorithms assume the environment model is known. Among them, AlphaZero (Silver et al., 2017) is one of the most famous. AlphaZero uses the game's rules (Chess, Shogi and Go) as the environment model to generate a series of selfplay simulated games. During the simulations, the actions for both players are selected using a Monte-Carlo Tree Search (MCTS) algorithm. The MCTS performs as a planning algorithm by generating candidate actions which are superior to the current policy. The neural network parameters are updated at the end of each game to minimize the game prediction error (loss, draw or win) and maximize the similarity of policy vector to the planning algorithm. AlphaZero is limited to the discrete action space problems. Imagination-Augmented Agents (I2As) (Racanière et al., 2017) introduces two paths: 1) model-free path and 2) imagination path. The imagination path learns a transition model and uses this model to generate imagination rollouts. These rollouts are aggregated with the samples in the model-free path.

To generate actions in the imagination path, I2As uses the model-free path policy. Therefore, the rollouts in the imagination path improve as the I2As policy improves. Using the imagination rollouts, I2As converge faster than a model-free network with the same number of parameters. Nagabandi et al. (2018) showed that a two-step control policy based on 1) learning the dynamic model and 2) applying MPC to the learned model is significantly more sample efficient than model-free RL. However, this approach cannot achieve high rewards. To achieve higher rewards and preserve sample efficiency, they proposed a hybrid model-based and model-free (MBMF) algorithm which runs the model-based approach to achieve the initial result in a sample efficient way, it then trains a modelfree policy to mimic the learned model-based controller, and uses the resulting imitation policy as the initialization for the final model-free RL algorithm. Dalal et al. (2018) proposed safe exploration by modeling constraints using a linear model and applied Lagrangian optimization to modify the action in order to guarantee safety. In this work, we also used Lagrangian optimization for short-term constraints. However, our approach is different in two ways: 1) our method does not modify the RL action to achieve the goals. Instead, it derives an action by considering both long-term goals and short-term constraints. This is possible because our algorithm uses a locally linear model to represent the advantage function. 2) Unlike safe exploration, the focus of this paper is in handling new constraints in the application phase without retraining the model. 

B CRANE CONTROL SYSTEM:

Gantry cranes are widely used in industrial production lines and construction projects for transferring heavy and hazardous materials. The objective of the crane system is to convey the payload from the start position to the destination position as soon as possible while keeping the payload sway at the destination minimum. Higher traveling speed improves the efficiency and reduces costs. However, excessive movements at the destination wastes time and energy and can lead to accidents. To move the payload as fast as possible and stop the sway at the destination, skillful operators are required.

Labor shortage in industries, and risk of human error, have motivated us to develop an automated solution for crane control. The crane dynamic system is highly nonlinear. Traditional nonlinear control techniques such as sliding control (Bartolini et al., 2002) and adaptive control (Boustany & d'Andrea Novel, 1992) have been applied to these systems. These methods require detailed mathematical model of the system and its environment, which can be complicated and expensive to derive. When a simulator is available for a crane system, RL algorithms can provide a compelling alternative to traditional control methodologies. This is the case in many industries, where for intellectual property concerns the companies are willing to provide simulators to the costumers but refuse to reveal mathematical models of their products.

Our crane simulator provides us six state variables: 1) trolley location, x trolley 2) trolley velocity, v trolley , 3) payload angle, payload , 4) payload angular velocity, ! payload , 5) payload horizontal location, x payload , and 6) payload vertical location, y payload . The only action is the force applied to the trolley, u trolley . The overall goal is to reach the final destination x pd and y pd in the shortest time possible. We choose the following reward function to learn a policy to do so.

EQUATION

)

where ✏ is a small constant. Figure 7b shows that our learned policy pushes the trolley with maximum force, u trolley is eqaual 1 for the entire episode, till the payload reaches the goal x pd = 6, y pd = 10.

In additional to the long-term goal, our short-term goal is to minimize the object's sway when it reaches to the final destination. Instead of designing complicated reward functions to achieve minimum travel time and minimum sway, we consider ! payload = 0 at the final destination as a short-term desired trajectory. We consider the following hybrid strategy to reach the final destination with close to zero sway. 

Figure 8 shows that our strategy can reach the destination with close to zero swing. C DERIVATION DETAILS FOR SHORT-TERM CONSTRAINTS Consider equation ( 19). Substituting our estimation of the next step from equation ( 14) in ( 19), we have

min findu k 1 2 (h k + d k u k ) 2

such that:

EQUATION

Using Lagrangian method at each time step k, we have

EQUATION

Taking the gradient of L with respect to u k , we can write the Karush-Kuhn-Tucker (KKT) (Kuhn & Tucker, 2014) conditions for optimal solution of equation ( 25), {u ⇤ k , ⇤ } as:

EQUATION

With this assumption, we can show

EQUATION

Note that when there is no constraint: ⇤ = 0, we have

u ⇤ k = (d T k d k ) 1 d T k h k .

This is exactly the input we computed in equation ( 16). When ⇤ 6 = 0, we have k from equation (28) in equation ( 29) we have

EQUATION

EQUATION

E MODEL-BASED REINFORCEMENT LEARNING Nagabandi et al. (2018) proposed a model-based RL based on MPC. Their approach uses the system dynamic model + reward function to generate a sequence of actions that maximize the cumulative reward. To represent the effect of short-term and long-term predictive models in our algorithm, we present a modified version of (Nagabandi et al., 2018 ) algorithm using our short-term predictive model, f and g in equation ( 14). The model based RL uses the learned dynamic model, f and g networks, and generates a sequence of actions U H k = (u k , ..., u k+H 1 ) to maximize reward over a finite horizon, H. 

Solving equation ( 33) for the exact solution is computationally expensive. Therefore, Nagabandi et al. (2018) applied a simple random sampling method and select the candidate action sequence with the highest expected cumulative reward. In this work, we apply the same method to solve equation ( 33). We set H = 25 and select 10000 samples at each step. Following the MPC closedloop control framework, the algorithm only executes the first action in the sequence and calculate a new sequence in the next step. Algorithm 2 presents the model-based RL.

Algorithm 2 Model-based Reinforcement Learning (Nagabandi et al., 2018) 1: Initialize f ang g networks (equation ( 14)) with random weights. for k = 1:T do 8:

Solve equation (33) for the optimum sequence U H k at state x k . 9:

Executes the first action in the sequence, u k . 10:

Store transition (x k , u k , x k+1 , r k ) in R rl .

:

http://gym.openai.com/envs/MountainCarContinuous-v0/

Figures and Tables:

Figure 1: Crane system. The long-term goal is to move the payload to the target location, (x d , y d ) as soon as possible. The short-term goal is to have zero sway at the destination, !p = 0.

Figure 2: Learning the LLQL Network Parameters.

where h(x k |✓ h ) and d(x k |✓ d ) networks model the locally linear advantage function. Note the NAF advantage function is a special case of the LLQL advantage function when d(x k |✓ d ) = I, where I represents the identity matrix.

(a) Average and standard deviation of cumulative rewards during the learning process. LLQL and DDPG algorithms converge in less than 40 episodes. Modelbased reinforcement learning algorithm (MPC) cannot learn a policy that guides the car to the top. (b) State variables and estimated state variables. Using LLQL model, the car reached the top in 78 steps.Using DDPG model, the car reached the top in 110 steps. The car never reached the top when we used the MPC model. The mean absolute error for LLQL position estimation is 0.00078, the mean absolute error for LLQL velocity estimation is 0.000084, the mean absolute error for MPC position estimation is 0.00039 and the mean absolute error for MPC velocity estimation is 0.00014.

Figure 3: LLQL for MountainCarContinuous. The network's parameters are presented in Section D.

(a) With LLQL the car reached the top of the mountain in 82 steps with a short-term goal error of 0.00008. With DDPG + modified reward (3 rd reward in Table1) the car reached the top in 138 steps with short-term goal error 0.0023. The MPC + modified reward cannot guide the car to the top.(b) Comparing the actions in normal case versus with the short-term trajectory. The hybrid strategy (see equation (21)) leads to quick adjustment in LLQL action and therefore, smaller short-term goal error compared to the DDPG + reward modification solution.

Figure 4: MountainCarContinuous with short-term and long-term goals. 1 = 1, 2 = 2000.

With LLQL the car reached the top in 105 steps without violating the constraints. With DDPG + modified reward (3 rd reward in Table 2) the car reached the top in 199 steps and violated the constraints 13 timesteps. The MPC + modified reward cannot guide the car to the top. (b) Comparing the actions in the normal case versus with the short-term constraint. The hybrid strategy (see equation (22)) leads to sharp adjustment in LLQL action when the car gets close to the hazardous areas.

Figure 5: State variables for MountainCarContinuous with long-term goal and short-term constraint. The horizontal red lines, |v k | = 0.035, represent the boundaries. The car reaches the goal in 97 steps.

DDPG rnew = r k 10 if |v k | > 0.033 21.5 100 10/10 DDPG rnew = r k 100(|v k | 0.033) if |v k | > 0rnew = r k (100(|v k | 0.033)) 2 if |v k | > 0MPC rnew = r k 10 if |v k | > 0.033 0 1000 0/10 MPC rnew = r k 100(|v k | 0.033) if |v k | > 0rnew = r k (100(|v k | 0.033)) 2 if |v k | > 0Number of steps to the top vs desired final velocity. 1 = 1, 2 = 2000. (b) Number of steps to the top vs velocity constraint.

Figure 6: Long-term performance vs short-term goals. We run the model with each short-term goal 10 times and present the average and standard deviation of the long-term goal.

(a) Cumulative rewards for LLQL. (b) State variables for the crane system. The crane reaches the goal in 12s.

Figure 7: LLQL for the crane system.

u trolley = ⇢ use equation (16) if x payload < 5.5 use equation (18), otherwise

Figure 8: State variables for the crane system with required short-term trajectory, ! payload = 0. The final error is 0.0001. The crane reaches the goal in 12.5s. 1 = 1, 2 = 1000.

29)We define↵ 1 = g i k d T k (d k d T k ) 1 , and ↵ 2 = g i k (d T k d k ) 1 d T k .↵ 1 and ↵ 2 are scalar. Substituting u ⇤

i , u i ), such that: xk = x k and xi+1 = xi + (f (x i |✓ f ) + g(x i |✓ g )u i )

Create dataset of random trajectories R rand . 3: Create the reply buffer R rl = ;. 4: for episode = 1:M do 5: Randomly select a mini-batch of Ns transition from R rand and R rl . 6: Update ✓ f and ✓ g by minimizing the loss: L1 = 1 Ns P Ns i=1 ||xi+1 xi (f (xi|✓ f ) + g(xi|✓ g )ui)||. 7:



Table content: <table><tr><td/><td/><td/><td>Green boxes: inputs</td><td/></tr><tr><td>𝑓(𝑥 𝑘 |𝜃 𝑓 ) Networks</td><td>𝑓 𝑘</td><td/><td colspan="2">Blue boxes: short-term network Orange boxes: long-term network</td></tr><tr><td/><td/><td/><td>Loss</td><td/></tr><tr><td>𝑔 𝑥 𝑘 𝜃 𝑔 Networks</td><td>𝑔 𝑘</td><td>ො 𝑥 𝑘+1 = 𝑥 𝑘 + Δ(𝑓 𝑘 + 𝑔 𝑘 𝑢 𝑘 )</td><td>function 𝐿 1</td><td>𝑥 𝑘+1</td></tr><tr><td/><td/><td/><td>Loss</td><td/></tr><tr><td>𝑥 𝑘</td><td>Controller</td><td>𝑢 𝑘</td><td>function 𝐿 2</td><td/></tr><tr><td>𝑉 Network</td><td>𝑉</td><td>Q function</td><td/><td/></tr><tr><td>ℎ Network</td><td>ℎ 𝑘</td><td>Advantage function</td><td>Target value</td><td>𝑟 𝑘</td></tr><tr><td>𝑑 Network</td><td>𝑑 𝑘</td><td/><td>function</td><td/></tr></table>

Short-term trajectory performance

Table content: <table><tr><td>RL Method</td><td>Modified Reward function</td><td>Average velocity error</td><td>Average number of steps</td><td>Long-term goal success rate</td></tr><tr><td>DDPG DDPG</td><td>rnew = r k 5000 ⇤ |v k v d | if done rnew = r k 100 ⇤ |v k v d | if x k &gt; 0.45</td><td>0.0232 0.0193</td><td>109.1 183.6</td><td>10/10 10/10</td></tr></table>

Short-term constraint performance

Table content: <table><tr><td>RL Method</td><td>Modified Reward function</td><td>Average number of steps out of boundary</td><td>Average number of steps</td><td>Long-term goal success rate</td></tr></table>

The environment model is typically unknown in real-world applications. Therefore, many model-based RL algorithms learn the state transition model from the data. NAF(Gu et al., 2016) learns a linear model for state transition at each operating point and uses this model to generate additional samples through imagination rollout. World Models(Ha & Schmidhuber, 2018) uses a Variational Auto Encoder (VAE) to map a state variable, x 2 X to a lower dimensional variable z in a latent space Z. It then uses a recurrent neural network (RNN) to learn the state transition model in the latent space. Finally, it applies a simple linear controller to z and the hidden state in the RNN, h, to control the system.

Table content: <table/>

Feinberg et al. (2018) proposed Model-based Value Expansion (MVE) algorithm, which limits the uncertainty in the model by only allowing imagination up to a fixed number of steps, H. MVE uses the learned system dynamic model to generate simulation data up to H steps into the future, and applies these sample points to estimate Q-function. Instead of saving simulated samples in an imagination buffer, MVE retrains the dynamic model and generates a new imagination rollout at each step.Buckman et al. (2018) expanded MVE algorithm by proposing Stochastic Ensemble Value Expansion (STEVE), to generate a solution more robust to model uncertainty.

Table content: <table/>

Peer review (iclr2020):
Thank the authors for the response. I will keep my score. ---------------------------------------- Summary In this paper, the author proposes LLQL (Locally Linear Q-Learning), which separates the action design from prediction models, such that the model can have short-term and long-term goals. The short-term network takes the current state and the action, and predicts the next state (to be specific, the status change) under the assumption that the output is linear to the action. The long-term network takes the current state as input, controls the controller to take action, and also optimizes a Q-function. I lean to vote for accepting this paper, though the experiment part can be further improved. Strengths - The proposed model is novel, which can take a trajectory or a constraint as a short-term goal. As a result, we can control the behavior of the model easily. Otherwise, we have to design a new reward function to guide our model indirectly. As the authors showed in Tables 1 & 2, LLQL outperforms this approach. - The paper is clearly written. The author provides clean formulas throughout the paper, which makes the paper easy to understand. Weaknesses More experiments can be conducted to demonstrate the effectiveness of LLQL. - Only tested in one scenario. In the main text, though it is already very long, there are still only experiments in one scenario (i.e., Mountain Car). - Only compared to one baseline. In order to demonstrate the effectiveness of LLQL taking a short-term trajectory or constraint, the authors compared it to only one method (i.e., DDPG) with a few manually designed reward function. Minor Equation 6 looks incorrect to me. Possible Improvements As mentioned before, it would be great to include more experiments in the main text, comparing LLQL to more baselines, testing LLQL in more scenarios, or doing some ablation studies (removing one part of LLQL to demonstrates the effectiveness of that part).