Continuous Deep Q-Learning in Optimal Control Problems: Normalized Advantage Functions Analysis
Authors: Anton Plaksin
Stepan Martyanov
Venue: ICLR 2022 Submitted (2022)
Keywords: continuous reinforcement learning
deep q-learning
optimal control problems
normalized advantage functions

Abstract: One of the most effective continuous deep reinforcement learning algorithms is normalized advantage functions (NAF). The main idea of NAF consists in the approximation of the Q-function by functions quadratic with respect to the action variable. This idea allows to apply the algorithm to continuous reinforcement learning problems, but on the other hand, it brings up the question of classes of problems in which this approximation is acceptable. The presented paper describes one such class. We consider reinforcement learning problems obtained by the time-discretization of certain optimal control problems. Based on the idea of NAF, we present a new family of quadratic functions and prove its suitable approximation properties. Taking these properties into account, we provide several ways to improve NAF. The experimental results confirm the efficiency of our improvements.

CONTINUOUS DEEP Q-LEARNING IN
OPTIMAL CONTROL PROBLEMS: NORMALIZED ADVANTAGE FUNCTIONS ANALYSIS
Anonymous authors Paper under double-bling review
ABSTRACT
One of the most effective continuous deep reinforcement learning algorithms is normalized advantage functions (NAF). The main idea of NAF consists in the ap- proximation of the Q-function by functions quadratic with respect to the action variable. This idea allows to apply the algorithm to continuous reinforcement learning problems, but on the other hand, it brings up the question of classes problems in which this approximation is acceptable. The presented paper de- scribes one such class. We consider reinforcement learning problems obtained by the discretization of certain optimal control problems. Based on the idea of NAF, we present a new family of quadratic functions and prove its suitable approxi- mation properties. Taking these properties into account, we provide several ways to improve NAF. The experimental results confirm the efficiency of our improve- ments.
INTRODUCTION
The standard reinforcement learning (RL) setup consists of an agent interacting with an environment (Sutton& & Barto 2018). At each step of the interaction, the agent determines an action based on its policy and its current state, gets a reward, and makes a transition to the next state. An aim of the agent is learn the policy that maximizes the sum of rewards.
Q-learning (Watkins & Dayan 1992) is one of the most widespread algorithms for solving RL prob- lems. According to this algorithm, the optimal action-value function (Q-function) being found as a solution of the Bellman optimality equation. After the learning, the agent can act optimally by the learned Q-function. Initially, the Q-learning algorithm was applied for solving RL problems with finite state and action spaces. In this case, the Q-function can be represented by finite ta- ble. For the case of a large or continuous state space, Q-Jearning has recently been extended to Deep Q-learning algorithm (Mnih et al., 2015) that allows to look for the approximate Q-function in the class neural networks by means the stochastic gradient descent. Deep Q-learning and its modifications have shown the efficiency for range of challenging tasks (Wang et al.) 2016)) van Hasselt et af.) 2016 [Schaul et al.) 2016 Hessel et al.) 2018), however, note that this algorithm can not be directly applied for solving RL problems with continuous action spaces. The reason that Deep Q-learning involves maximizing of an approximate Q-function by the action variable on each step of the learning, which is complex problem for continuous action spaces. Among various ap- proaches to overcome this problem (Liliicrap et al. 2016 Haarnoja et al. 2017: Kalashnikov et al., 2018 Lim et 2019 Ryu et al. 2020 Lutter et al. (021), we focus on an idea of the normalized advantage functions (NAF) algorithm (Gu et al. (016). This idea consists in the approximation of the Q-function by functions quadratic with respect to the action variable. allows to get the max- imum quite fast and precisely and solve some challenging control problems (Gu et al. 2017 Dong ctt al., 2018 Ikemoto & Ushio 2021), but on the other hand, it brings up the question classes of RL problems in which this approximation is acceptable. The presented paper describes of the possible answers to this question.
Note that the class of LQR problems (Bradtke at. 1994) has the Q-functions quadratic with respect to the action variable. However, this class, being quite special, is not suitable for the description of complex controlled processes. In the paper, consider wider (in some sense) class of RL prob-

---

lems. We consider RL problems which are obtained by the discretization of certain optimal control problems (Bardi & Dolcetta 1997). The rationale for the consideration of such RL problems is that lot of RL problems with continuous action spaces arise from control problems for mechanical
or robotic systems [Lillicrap et al.) 2016 Gu et al. 2016; Нaarnoja et al. 2017; Gu et al.] (017] Kalashnikov al. 2018), whose dynamic are, in fact, described by ordinary differential equations.
For the considered class of problems, based on the idea of NAF, we present a new family of quadratic functions and prove that, first, this family sufficiently rich approximately solve the Bellman optimality equation (Theorem 1), and second, any sufficiently accurate solution of the Bellman optimality equation allows to approximately obtain the optimal policy the corresponding optimal control problem (Theorem 2). Moreover, we prove that it is impossible to get the same results for the original family of functions from Gu al. (016) (Theorem 3). From the obtained theoretical statements, we get some additional knowledge about the Q-function approximation by our family of quadratic functions and also provide several ways to use this knowledge order to improve NAF. The experimental results confirm the efficiency of our improvements.
BACKGROUND
The standard reinforcement learning (RL) setup consists of an agent interacting with an environment
(Sutton & Barto 2018). This interaction is described by Markov Decision Process (MDP), which is a tuple (S, P, R. Po; 7), where S is a state space, distribution, R(s, is a reward function, po(s) is an initial state distribution, and E [0, 1] is discount factor. An aim of the agent is to learn its optimal policy (s(s) that maximizes the value
an action space, P(s'|s, s, u) is a transition
J(x) =
Σ R'((s, u:) ~ po(so), u, = u(s;), 8:41 ~ P(s uz), = 0, 2,., 1=0
In the general statement of reinforcement learning problems. a policy of the agent can be stochastic, however, within this paper, we assume that the policy is deterministie.
One of the most effective algorithms for solving RL problems is Q-learning (Watkins & Dayan 1992). According to this algorithm, the agent explores the environment and looks for the optimal action-value function (Q-function)
Q. u) = sup E Σ r
RRis ui) 80 = 8, 40 =
H
8;+1 ~ P(sit1lsi, uz), ust1 = p(8:+1), i = 0,
as solution of the Bellman optimality equation
Q. [[R(s, u) + max (s', ~ u(s
In other words, the agent solves the following minimization problem:
sup WES
Q(s u) - B[R(s u) + m Q(5 w')l ~ P(s'ls, + inf.
max n'Ell
(1)
If the agent knows the function Q+ a), it can act optimally by the greedy policy (s())) arg max,eu (s)
Initially, the Q-learning algorithm was applied for solving RL problems with finite state and ac- tion spaces. In this case, the Q-function is represented by finite table and problem (T) is finite- dimensional. For the case of a large or continuous state space, Q-learning has recently been extended to Deep Q-learning algorithm (Mnih et al. 2015)) that allows to look for the approximate Q-function in the class of neural networks Q(z a (7) where 8 is the parameter vector the neural network. During the learning, the experiences (st, us Pi, S1+1) are stored in the buffer D and simultaneously the parameter vector g& is updated by means of the stochastic gradient descent minimizing the loss function
L( [E[(Q(s, (s,,,,,,,,)))~
y =- max "' a'Ell
Q(s',u''00
(2)
2

---

where (D) is the uniform distribution D. Deep Q-learning and its modifications are effective for
a range of challenging tasks (Wang et al. 2016 van Hasselt et al. 2016: Schaul et al. 2016 H Hessel not be directly applied for solving RL problems
et al. 2018), however, note that this algorithm with continuous action spaces. The reason is that Deep Q-learning involves the maximizing in 2 on
each step of the learning. which is complex problem for continuous U. Among various approaches
to overcome this problem (Lillicrap et al. 2016,] Haarnoja et al. 2017, Kalashnikov et al.) 2018
Lim et al., 2019; Ryu al., [020) Lutter al. 2021). we focus on an idea of the normalized advantage functions (NAF) algorithm (Gu et al., (016). This idea consists in the approximation of the Q-function by the following quadratic with respect functions:
Q(s, ul00) V(sl0V) + A(s, u((A),
(3)
A(s, uss = (u m(s[0-)) P(s]0P) (u-(ss(()))
where V(s(0V, [(())0)) and P(s]OP) are neural networks with parameters 0V 0u and OP, spectively; P(s is a positive-definite square matrix for each and 0"; 0사 = f0r 0²) and 09 -0a, 0V) J. Under the condition
mls)0t) E U,
(4)
it allows to get the maximum and argmaximum values directly by values of (ss0Vv and m(s)0t):
Qss, uI00) (ssl0M) Argmax (s,, u108) = p(s(0 wElL
max
well
but on the other hand, brings up the question of classes of RI. problems in which quadratic ap- proximations is acceptable. Below, we describes one such class.
PROBLEM STATEMENT
In this section, consider a certain class of optimal control problems and show that discrete ap- proximations of these problems can be formalized as RL problems.
Consider the following optimal control problem: it is required maximize the functional
J((()) = (x(()) - 1." (a(1, ((t) (())))((((())((()))))))))
(6)
over all u(.), where st.) is the solution (Filippov 1988
$1) of the differential equation
dt =(t) = f(t, (t)) + gt. (t)) e [0,
(7)
under the initial condition
x00)
(8)
Here is the time variable, T > vector, u(t) U is the current control action vector forming the measurable function ul-), U C R the nonempty compact set, z RR" the fixed initial state vector, f(t, x) E R g(t, x)) Raxm <(t,z) x) E IR, rtt,z) E R (t,z) E [0,T" x IR" are continuous with respect to t and continuously differentiable with respect to x functions, r(t, æ) is the positive-defintte matrix for each (t, [0, T R" and (Ix) R, E R is the continuous function. We assume that there exists a constant Cfg > 0 such that
the terminal instant of time, xxt) F" is the current state
[t, + g(t, z)ull < (1 + ||)|))
(t,z) E [0, x R", u u U.
(9)
Note that, under these conditions, for each function w(-), there exists unique solution ax-) of equation ( under the initial condition (8) (Filippov) 1988 $1).
Define the value function in optimal control problem 6)_ by
V.(t., = sup (a(x(T)) //,
suu a(.)
(t,, E 10, T] R"
(10)
3

---

where, for each u(-), xx.) is the solution of equation 7 the interval [tx, T] under the initial condition z(t.) = Z+.
Define the sets
S = (t,,)) E [0,T] x R ||||| & (1+ [1=1)) - 1}, S(t) = fa E (t, E S}. (11)
Let l E N. At, = T/k, and 4 i E 0, Consider the corresponding discrete optimal control problem: it required maximize the function
(ug u1,... . = (xx)
At Σ (g()) of (t),
(12)
over all u, E EU. WU.iE0,k 0, I, where (zo, Th...,, I&) is defined by
30 =, J++1 = + (J(1,, x4) + g(4 )1)4
A4 A4, i 1-
(13)
Let us show that problem 11 and actions spaces, the initial state distribution, and the discount factor as follows:
13) can be formalized the RL problem. First, define the state
= ( x S(4) 8e, == po(*a) = 8(50 = (0,:)), 7=1.
(14)
Here sy is some fictional terminal state, 8 is Dirac delta distribution. Next, for every E 0, k - x E S(t,), and u E U, we define the transition distribution and the reward function by
I,
P(s'ls (t,, u) = 8(s' = x()),,))),, R(s = (t,, x). u) = (g((t,, + u r(t,,, x x))) Ate.
(15)
where x' = x + (f(1,, x) + g(4,, x(41,))u) Alg. Taking into account 9 and (T), one can prove the inclusion (4++1, E S. Hence, the transition distribution P is well-defined. For = k, set
P(s'ls (4x, 2), u) = 8(s' = =T), R(s = (t), u) = (x), I I S(t),, WEU. (16)
In order to make dynamical processes formally infinite, we put
P(8 (1/|sT.u) u) 8(s' sT), R(8T, u) = 0, L U.
(17)
Thus, define MDP which describes the RL problem corresponding to problem T2. 13) Next, show that such RL problems is suitable for using quadratic approximations of the Q-function.
4 QUADRATIC APPROXIMATIONS OF THE Q-FUNCTION
Denote by Q the family of functions Q such that
Q(I, v, u) = Vl,,,) + A(1,z,,), (1,2, (0, (0,T)) RR" x
(18)
where
a(t,z,u) -u- x)) -(,, P(t, (
ū(t,
+(((t x)) P(t, P 2) pl(, x))
(19)
p(l,x)) arg (u' (((, - 2)),,)))
a'EU
Here V(t, j(t, x), and P(t, are continuous functions; P(t, x) is a positive-definite square matrix for each ()) E ()) x Rd.
Note that, functions Q from the family Q satisfy the equalities
max Q u) = V(,,.), Argmax Q(1, = p(l,2:). NEt/
(20)
aEU
as well as (see 5)) quadratic functions from family However, these function families are differ- ent. The difference that do not assume the inclusion Ä(t, U as opposed assumption (4)
The theorems below establish connection between the optimal control problem (
S and mini-
mization problems (T) for MDP (14 17
4

---

Theorem 1. Let the value function (t 2²) be continuously differentiable. Then, for every 0, there exists k, > O such that, for every ko 2 ku, the function Q E Q defined by (18) and I9
where
1
v(t,x) Vt,,,,,, P(t,z) at (),,x) = (1,x)9' (t, (t,
(21)
2
satisfies the inequality
(1,, u) + (g(1i, 2) + +u² + Ale - max Q(i++,, u')l S E w'EU
(22)
='= (f(tt g(t;, x) Ats. uEU, x S(t)) i
where we assume Q(tx, x', v/) (z²).
Theorem 2. Let the value function V. (t, x) be continuously differentiable. Let > 0 and k, > 0 be
defined according to Theorem 1. Take kt and suppose that a function Q Q satisfies inequality
22 Then the following estimate holds:
J& Uo, uI.... uk-1) 2 sup ((u()) 3T',
(23)
=(.)
where the function J& (ug, U1 1 is defined by function futt, x)) is defined by Q according to (18) and
12 with u, = p(t,, 2;), i E 0, and the
Thus, Theorem 1 shows that the function family Q is sufficiently rich to contain approximate solu- tions of problem (T) with predetermined accuracy and Theorem 2 establishes that all such approx- imate solutions contained in Q allow to get the policy, which approximately provides the optimal result in optimal control problem (6 88.
Note that the similar results can be obtained for the wider class of optimal control problems (see Appendix B). However, in this case, we need consider another form of the function 2(,, u). Within the presented paper, we focus on class (6)-8)) because, firstly. this class seems quite general and important for applications and, secondly, the function A(t, x, u). corresponding to this class, has the quite simple (quadratic) form.
Now, let us consider the original family of functions from Gu et 2016)) Denote by QNAP the family of functions Q such that
Q(t,,, u) = at, (u =((, (u x)), (f, x. u) E (0, R" U, (24)
where z),, p(t,, z),, and P(t,2) are continuous functions; P(t, x) a positive-definite square matrix for each (t,z) 2) [0. T) x R"; z(t, U for each (t, x) [0, T) x R". The theorem below establishes that if we take the family QNAP instant of Q then Theorem 1 can not be proved even in the simplest cases of optimal control problem (6) (8)
Theorem 3. Let n = m = 1, = 1, U = [-,, (t,, = (t,,z) = 0, xt, = r(t, = 1, (x) = 2 and = 2. Then, for every h 2 4 and Q E QNAAF there exist E 0, 1 I, S(t)), and u U such that
QQ(t,, 2, u) + u² A2 max Q(t++1, 2', v()l Atv/8, a'= x + uuAt. n'EU
where we assume Q(Ie, x', v/) = (2')²
Proofs of the theorems are given in the Appendix A.
5 EXPERIMENTS
We consider four examples of optimal control problems
6 described in Table
[ where
σ1 (I) = -
23. 0z (r) = -|01 - 0.1 22|
0y = - |m] 4 [2:2| |xg
0.75ml
03(2) = -2 2 (#3 2)² (#4 2)
Van der Pol oscillator is a famous model of a non-conservative oscillator with non non-linear damping. The aim of the control is stabilize the oscillator the terminal time.
5

---

Table I: Parameters in the examples of optimal control problems
Name
7
m
T
f(t,
g(t, x))
q(t, x) r(t, x))
2
Van der Pol oscillator
2
11
[-1,1]
(1 - =x2)
x/2
-21
0
0.05
on (I)
Pendulum
2
5
[-
I2
0
0.01
02(x1)
π
14.7 (1.7sin(x1)
3
0
c00((23)
Dubins car
3
2x
[-0.5, 1]
sin(x3)
0
0.05
03(xr
0
A target problem
6
10
1|,
.75
0.001 o1(2)
J6
2 303
3.
Pendulum is a traditional problem for testing control algorithms. The aim of the control is the stabilization of the pendulum in the top position at the terminal time.
Dubins car is a quite famous model which describes motion of the point particle moving constant speed on the plane. The problem is to find control providing the closeness of the motion with a target point at the terminal time.
A target problem is optimal control problem presented in Munos (006)). The dynamic system describes a hand holding a spring to which is attached a mass. required to control the hand such that the mass achieve the target point at the terminal time.
5.1 BOUNDED NAF
First, we modify NAF algorithm, proposed in Gu et al. (2016), based on the function family Q. Note that, the considered examples have U = [a., mh Denote tanh,,, a+ (-+(1++) (1+tan(()))
E R" a, 3 E RR. where tanh is the hyperbolic tangent for each coordinate. Then, according to and (19). we can use the following approximation of the Q-function, within NAF algorithm:
0tt,z,u/09) V(t, x1+) + A(t, ul0a),
A(t, x. u(0a) -u - (u- 2())) xt!")
+ ( x(0²)) ūtt, p(t,
where (tt, x1, ji(t, 2z )00)) and x(1,)((P) are neural networks with parameters @v 0a and OP respectively: P(t, , x 200)) -esitive-definite square matrix for each (t. x) and 0"; g (OP 0r) and 09 = {0A, 0V J. To be short, we call this algorithm Bounded NAF (BNAF), because it is essential for our modification
NAF that the set U is bounded.
Note that, we can also use the function clipa, (v) = max(o, min v)), E RT a, E R instead of the function tanh,,8. 3 According (19. more correct, however our experiments show that the learning results are slightly better with tanh,, (see Appendix C), A possible reason for this that the function clipa,e not smooth.
5.2 REWARD-BASED BNAF
If know the function r(t, then, according to Theorem 1, use the function r(t, x) Atx instead of the neural network P(t, x))P)) to reduce the number learning parameters. This variant of BNAF is called Reward-based BNAF (RB-BNAF).

---

-20
-15
that f--nna
3-hool
gh-brot
opp2
0099
-20
(a) Van der Pol oscillator
(b) Pendulum
-100
-20
-10
npl
Insf
-500
250
300
750
IC00
1250
2500
3130
2000
200
400
600
800
3000
@2sodes
esisodes
(c) Dubins car
(d) A target problem
Figure 1: Results of NAF, BNAF, RB-BNAF, GB-BNAF, DDPG algorithms averaged 5 seeds.
5.3 GRADIENT-BASED BNAF
we also know the function g(l, x),, then, according to the Theorem 1, we can use the function
1 att, 10 /" (1, ((.
instead of the neural network jitt, (0FR) It also reduces the number of learning parameters. This variant BNAF is called Gradient-bant based BNAF (GB-BNAF).
5.4 EXPERIMENTAL RESULTS
We use the same learning parameters of every our tasks. We apply neural networks with two layers of 256 and 128 rectified linear units (ReLU) and learn their used ADAM with the learning rate Ir 5e-4, We use batch size Tiss 256 and smoothing parameter 7 1 Also we take At = 0.1. All calculations were performed on personal computer in standard way.
We compere NAF, BNAF, RB-BNAF, GB-BNAF algorithms, and DDPG algorithm presented in Lillicrap et al. 2016). Figure I shows learning curves of the algorithms for the considered exam- ples. One can note that GB-BNAF algorithm the most stable and gets the best performance in all examples. RB-BNAF is also capable to get acceptable results in all examples, although it does not have the same stable learning as GB-BNAF, RB-BNAF algorithm gets worse result than GB- BNAF and RB-BNAF in Dubins car (c). NAP demonstrate good results in Van der Pol oscillator (a) and Pendulum (b), however it does not cope with Dubins car (c) and A target problem (d). DDPG algorithm shows poor performance only in A target problem (d).
Thus, taking the presented experiments into account, we can give the following general recommen- dations. For solving optimal control problems 66 8). it is rational to use BNAF, GB-BNAR and

---

especially RB-BNAF algorithms along with NAF algorithm. There is high probability that they will show better results and more stable learning.
RELATED WORKS
Many different ways apply reinforcement learning approaches for solving optimal control problems are investigated (Baird 1994;
Doya 1995 2000; Munos] 2006; Tallec et al.) 2019
Jeongho Kim 2020, Lutter al. 2.21j. Among them, dineeiiscttiization is perhaps the most obvious and widely used tool (Lillicrap et al. 2016 Gu et al. 2016 Haarnoja et al. [017] Gu al.,
2017
Kalashnikov et al. 2018). From the theoretical point of view, it is known (Bardi & Dol-
solution cetta 1997 p.388) that solutions of time-discrete optimal control problems converge
of the initial problem as the discretization step tends to zero. In the present paper, we also use the -isceeesrttiittinnn and study approximating solutions of Bellman equations and the corresponding greedy politics depending on the discretization step (see Theorem and 2). Other studies depen- dencies the discretization step of reinforcement learning methods can be found in Munos (2066);
Tallec et al. (2019).
expand the Q-learning algorithm to optimal control and reinforcement learning problems with continuous actions. Other approaches for solving such problems are investigated in Lillicrap et al. (2016); Haarnoja al. 2017); Kalsshikk al. (2018);
We focus the idea from Gu al. 2016)
Lim et al. (219); ; Ryu et al. (2020): Lutter et al. (2021). The paper Lutter et al. (2021) ) seems the closest the presented paper. In this paper, the similar optimal control problem and feedback
control policy are considered, however, another Bellman equation is used.
19 is included, some sense, to families of Q-functions considered in Wang et al. (2016); Tallec et al.] (2019). It seems expected because more general classes of problems are considered in these papers. Nevertheless, proposed algorithms and results of these papers are very different from presented in this paper.
Let us also note that the family of functions (T8)
REFERENCES
Leemon C. Baird. Reinforcement learning in continuous time: Advantage updating. Proceedings of 1994 IEEE International Conference on Neural Networks, 1994.
Martino Bardi and Italo Capuzzo Dolcetta. Optimal Control and Viscosity Solutions of Hamilton- Jacobi-Bellman Equations. Birkhauser, Boston, 1997.
Steven J. Bradtke, B. Erik Ydstie, and Andrew G. Barto. Adaptive linear quadratic control using policy iteration. Proceedings of American Control Conference, 1994. doi: 10.1109/ACC.1994. 735224.
Xingping Dong, Jianbing Shen, Wenguan Wang, Yu Liu, Ling Shao, and Fatih Porikli. Hyperpa- rameter optimization for tracking with continuous deep q-learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 518-527, 2018. doi: 10.1109/CVPR.2018.000
Kenji Doya. Temporal difference learning in continuous time and space. Proceedings of the 8th International Conference on Neural Information Processing Systems, pp. 1073-1079, 1995.
Kenji Doya. Reinforcement learning continuous time and space. Neural Computation, 12:219- 245, 2000. doi: 10.1 1221666099976000000055090
Alexey Fedorovich Filippov. Differential Equations with Discontinuous Righthand Sides. Springer, Berlin, 1988.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. Proceedings of the 33rd International Conference on Machine Learning, 48:2829-2833 2016.
Shixiang Gu, Ethan Holly. Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off off-policy updates. Proceedings 2017 IEEE Interna- fional Conference Robotics and Automation (ICRA), 2017. doi:
8

---

Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep amergy-baseed policies. Proceedings the 34th International Conference on Machine Learn- ing. 70:1352-1361, 2017.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bital Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. Proceedings of The Thirly-Second AAAI Conference on Artificial Intelligence, pp. 2215-3222, 2018.
Junya Ikemoto and Toshimitsu Ushio Continuous deep q-learning with simulator for stabilization of uncertain discrete-time systems. (XXiv:20010064400 2021.
Insoon Yang Jeongho Kim. Hamilton-jacohi-belinn bellman equations for q-learning in continuous time. Proceedings of the 2nd Conference on Learning for Dynamics and Control, 220:739-748, 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for rision-basek robotic manipulation. 2nd Conference on Robot Learning (CoRL 2018), 87:651-673, 2018.
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Interna- fional Conference on Learning Representations (ICLR), 2016.
Sungsu Lim, Ajin Joseph, Lei Le, Yangchen Pan, and Martha White. Actor-expert: A framework for using q-learning continuous action spaces. arXiv: 1rXiv:1810.091 1810.09103, 2019.
Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, and Animesh Garg. Value iteration in con- tinuous actions, states and time. Proceedings of the 38th International Conference on Machine Learning. 139:7224 7234, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(754)) 2015. doi: 10. 1a1100/naurre12236
Rémi Munos. Policy gradient in continuous time. Journal of Machine Learning Research, 7:
771-791, 2006.
Moonkyung Ryu, Yinlam Chow, Ross Michael Anderson, Christian Tjandraatmadja, and Craig Boutilier. Caql: Continuous action q-learning. Proceedings of the Eighth International Con- ference Learning Representations, 2020.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In- Ternational Conference on Learning Representations (ICLR), 2016.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning An Introduction (Second Edition). The MIT Press, Cambridge, Massachusetts, 2018
Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust time discretization. Proceedings of the 36th International Conference on Machine Learning, 97:6096 6104, 2019.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16), 30 (1):2094-2100, 2016.
Ziyu Wang. Tom Schaul, Mattco Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Ducling network architectures for deep reinforcement learning. Proceedings of The 33rd Interna- tional Conference on Machine Learning, 48:1995-2000 2016.
Christopher Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992. doi: 10.1007/BF000992698.

---

A
APPENDIX
Denote
8, x)) + ,(t, s) =(t,x) u² ((t,,
(25)
Lemma Let the value function (t,,,) z) be continuously differentiable. Then, the following HamiloonJJcobi-eellnn Belbi-n equation holds:
(t, max H(t, V2V (t, E (0, (),T) R
WEU
The proof can be found, for example, in J. Yong. Differential Games: A Concise Introduction.
World Scientific, University of Central Florida, 238 USA, 2015. doi: 10.1142/9121.
Proof of Theorem 1. Note that, due to 199
21 and 25
we have
a(t, u/ H(t, x, V. V. (t, x), u)
max H(t, x, V V2V.(t,2), n'EU
(26)
for any (1, [0, R" U. Due 9 and
have
[|]((,,)) g(, x/ v czg (1 + ||=1)) agg: (1, E u E U.
(27)
Lete> Since (t, x) is continuously differentiable, there exists 8 > 0 such that
V. @v.(t.z)
x' -a) S (11 ++|11=10000 xx'll) + Orfg
(28)
Ōt
for any (1, x), (t', x') satisfying It 1'| || - x'1l S 8. Put k. (1 + +5ag))/
Let 2 k. i E 0, k 1, x E (t.)). u E U, and x' = x + (f x)) + g)t;, Atx. Due to and
we have (4) E S. Then, from (7) and 28
we derive
@V. (V,(t,,))
V. (t,,,,
At. (V V. (fz, f(i, x)+
S <EAts, (29)
Next, according to 18
26 and Lemma 1, we obtain
Q(1,, x; u) (g(4;, x) u² ((11,, Ale min W
Q(4++1,
= VV + A(4,, u) + (g(4, + u r r(1,,, (1,,,)u)) x)))/// A/e V=(1+41,22)
OV. (t,, max H(I,, x., V VV Ale + &eAl el a'EU
The theorem is proved.
Proof of Theorem 2. Let > 0. Define k. according to Theorem 1. Let k: > k. Let Q E Q satisfy
Let Iú, i E 0, & be defined by 113 where ui u(ts, 24). Then, due to have
22
12 20 and 22 we
v(0,2) - Ju(uo....,
(30)
(l(ii,ttuu)) + (9(4,, 24) + ((T(((,1)))) Ale max Q( ((141, w'EU
Te.
Let us detine Q. E Q according to 21 Let क;, i E 0, be defined by
13 where up E
arg max Q. Ii, ui, Then, due the Theorem and 20 have
2tt,, (f(t +
(tttt,,z))) x7)4 At Va
S A
(31)
Note that, according to definition (10) of the value function V., we have V.(t,, = (xx), x E IR..
Then, from 20
22) and (31), we derive
max w'EU
Q(ta, -x, u²) oxx))
max Q(t411, x/) - Q(ti, u²) - (g(te, x;5) + r(t. +V,022 WeU
10

---

(u((t)zz)) + (;) + (tt))))+((((t()r())r("a ((to)))(((,,2333 At - Vu(te+i) - (0,))
27' + V(, - VV)
Due to this inequality and inequality (30), we obtain Ju. (uo, (....... uk-1) 2 V.(00=)) 37e. Taking into account (6) and (10) we have sup [((())) = (O, z). Thus, the theorem proved. x(-)
Proof of Theorem 3. Let k 2 4 and Q E QNAP. Let us take i = h - and x = 2. For the sake of brevity, denote V (ftx-111 2), μ p(te-1, 2), and P P(t--1.2 2). Then, in order to prove the theorem, we have to show that
V - ( 22 (2+ x A4./8.
naax NE(-1,1)
Arguing by contradiction assume that
V (u 11 uf + (2+ ( & Atv/8.
max uE(-1.1]
Then, for u = 1, u 1, and u = 0, we have
At&/8 S V -1 (1+ 11) A4 (2 A1x) S A
(32)
Atx/8 S - (1 -1- -1-1) 44 + (2+ 2t2) Atv/8,
(33)
At4/8 VV 2 P2 Ltk/8
(34)
Adding up incquations 32
33 and twice inequity 34). we derive
2 2 2 < Alx/2.
(35)
Adding up twice incquations 32
34). and inequity 35) we obtain
A/4 < -4up - 8AIe S A/u.
From this estimate, taking into account
35 and the inequality k 2 4, we conclude
7
7
#
I < -1 5 5 4
4P
This inequality contradicts the inclusion μ E U ||-1, 1), which should be valid for Q QNAP-
B APPENDIX
Consider the following optimal control problem: it required to maximize the functional
J(u(-)) = =(((T)) /-" (1, u())ddt
over all w(-), where s(.) is the solution (Filippov 1988
$1) of the differential equation
dt xt) = F(t, u(t)), t [0,T].
under the initial condition
x00) = 2.
Here f is the time variable, T > 0 is the terminal instant of time, xxt) E R" is the current state vector, u(t) E U is the current control action vector forming the measurable function u(-), U R is the nonempty compact set, E Rh" is the fixed initial state vector, Ftt, u) E RR" (t, x,, u) E RR (t,x, u) E [o, T) " x U continuous with respect and continuously differentiable with respect to x functions, and (xx) E I. x E R" is the continuous function. We assume that there exists constant Cfa > 0 such that
|[F(1,,, w)ll (1 [
(1, (0, XR. R"
U.
11

---

-30
-15
1b- (die)
ot-bnet icipl
gbenst icto)
nh teaf franto
debral Ormm)
dc bnaf (tari))
20 g0-anaf naam))
-20
(a) Van der Pol oscillator
(b) Pendulum
200
200
teaf (ctp)
db-brei (diel
go-bnat
ghonee
Itanto
rb brot
250
300
750
SC00
1250
1500
3130
2000
200
400
600
820
3000
@psodes
esisodes
(c) Dubins car
(d) A target problem
Figure 2: Results of RB-BNAF, GB-BNAF algorithms with clip and tangent functions averaged over
5 seeds.
Similar to 25
denote
H(t, x, 8, (F(t, x, u)u, - (t, u).
Theorem 4, Let the value function V= x)) be continuously differentiable. Then, for every > O,
there exists k. >> 0 such that, for every k: 2 k., the function
Q x, u) = vt( + A(t, x, u),
(36)
A(t, u) 7((t, V.V. At max H((, vt) a'El/
satisfies the inequality
Qtt,, u) + F (t, u) max Q(t++1, x',", u() EAti.
x == where we assume Q(t,, ul)) = o(x').
u U.U, x S(t,, i
Proof of Theorem can be obtained similar to the proof of Theorem 1.
For the family of q-function 36 based on Theorem 4, one can also prove the result similar to Theorem 2.
C APPENDIX
Figure 2 shows learning curves of RB-BNAA and GB-BNAF algorithms with the clip and hyperbolic tangent functions for the considered examples. Note that there ing the clip functions show better performance comparing with the corresponding algorithms using the hyperbolic tangent function. Although the difference does not seem very significant.
in which algorithms
12


Reviews:
[{"review_id": "MoF7-rzn2Re", "reviewer": "Reviewer_6AfP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Paper is relatively clear and organized, and it is easy to understand the high level idea of the paper.\n- Great experiments, clear examples of how RL environment fall under the class of control problem\n\nWeaknesses:\n- The paper immediately jumps to a specific class of optimal control problem in Section 3. What is the intuition behind choosing such a class?\n- Needs more seeds, unclear in Figure 1 if RB-BNAF is better than NAF. NAF also seems to have very high variance and hence needs more runs. (Please do at least 5 seeds for all other baselines, if not 10 seeds for NAF)\n- In section 4, the new class of Q functions seems to be arbitrarily proposed. I believe this is the biggest contribution of the paper, so I recommend making this class of functions more intuitive (perhaps explain in Appendix how this class of functions are conceived of).\n- Should ablate on what happens when there is no bounding of Tanh for BNAF.\n\nQuestions:\n- What are the limitations to this approach (beyond limited to a certain class of problems)?\n- What is future work (e.g. extending this to more general classes of problems)?", "summary_of_the_paper": "-The authors prove that a discrete approximation of a class of optimal control problems can be recasted as an RL MDP.\n-The authors prove that the original NAF formulation of the Q-function cannot approximately solve the MDP defined above, and hence propose a new quadratic formulation of the Q-value. They apply their new formulation via Bounded, Reward-based, and Gradient-based BNAF.\n-The authors evaluate their proposed agent over 4 optimal control environments.", "summary_of_the_review": "Overall:\nBased on my limited understandings and mathematical background, the authors propose modifications to NAF, prior work, and prove it can solve certain classes of control theory problems. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0vh3dOKMVTN", "reviewer": "Reviewer_8xxB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript cannot be accepted for publication in the current format. Some, but not all, shortcomings are briefly mentioned below. \n\nFirst, the setting looks artificial and non-realistic, and the title is misleading, as the continuity is converted to discreteness! In Sec 2, the problem is not well explained, e.g., knowns and unknowns are not specified. Optimal policy is referred to as Greedy policy. I do not think that solving the problem in (1) is equivalent to those above (1), as the authors claim. \n\nFurther, since (6) is a classical noiseless optimal control problem and the goal is to find a possibly anticipative control signal, it is not clear how it falls under the umbrella of non-anticipative RL policies in Sec 2. Also importantly, it is unclear why the proposed method is better than the classical/existing solutions for (6). Note that it is assumed that the value function is known. So, directly solving HJB equation looks a better option, especially as it is exact and there is no approximation. \n\nThe proposed framework, by (11) and Thm 2, is applicable only to small T setting, which is not the common setting in control or RL. Putting this together to the other assumptions, the main claim of the paper is that Q-func is continuous with time. However, it seems like an immediate consequence of smoothness of the value function in Thm 1. Thm 3 is a counter example in fact, which is artificial since simply extending the U interval leads to letting \\mu be included in the family of policies.\n\nOverall, the impression of the reader is that the authors are not familiar enough with the frameworks of reinforcement learning and optimal control. \n\nSome concepts and quantities are not defined, e.g., after (3). There are several issues with the presentation including grammatical and writing errors, typos, inaccuracies, and unclear statements:\n\"our improvements\", \"can act optimally\", \"solution the Bellman\", \"For the case ..descent\", \"Deep ...challenging tasks\", \"the presented paper\", \"a wider (in some sense) class\", \"whose dynamic\", \"a new family of quadratic\", \"an aim\", \"as a solution\", (1), \"during the learning\", \"buffer\", \"consists in\", \"as well as (see 5) quadratic\", to name a few.\n", "summary_of_the_paper": "The paper applies the method of quadratic approximation of Q-functions to a continuous-time optimal control problem by discretizing the time. It is shown that under technical assumptions, if the horizon is short and the discrete times are dense enough, the approximation becomes accurate.", "summary_of_the_review": "The manuscript cannot be accepted for publication in the current format. First, the setting looks artificial and non-realistic and the problem is not well explained. Further, in (6), it is not clear how it falls under the umbrella of non-anticipative RL policies in Sec 2. Also importantly, it is unclear why the proposed method is better than the classical/existing solutions for (6). The proposed framework, by (11) and Thm 2, is applicable only to small T setting, which is not the common setting in control or RL. The main claim of the paper is that Q-func is continuous with time, which seems like an immediate consequence of smoothness of the value function in Thm 1. Thm 3 is a counter example, but is artificial since simply extending the U interval leads to letting \\mu be included in the family of policies. Some concepts and quantities are not defined, and here are several issues with the presentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "06zk94OIq8k", "reviewer": "Reviewer_J1Yy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important class of optimal control problems in continuous action spaces that has been traditionally difficult to solve by general RL algorithms, due to the need to solve a non-trivial optimization problem to find the optimal action every time the value function is computed by means of a Bellman back-up. By assuming that the advantage term of the value function is quadratic in the action, this optimization problem can be solved much more efficiently, thus speeding up policy computation significantly. The previously proposed NAF algorithm makes this assumption, as do other algorithms, for example the iLQR algorithm, without necessarily knowing for sure whether this general assumption is correct or not. A question of practical significance then is to try to understand under what circumstances this assumption is justified. In this paper, the authors prove that if the cost functional of the optimal control problem contains a term that is quadratic in the control effort (action), then the approximation is reasonable, and the true value function of the decision problem can be approximated in this way arbitrarily well. This is a good result, because this kind of \"minimum effort\" problems with non-linear dynamics but costs quadratic in the control effort are very common in practice and of high practical significance.\n\nOther than this proof, the contributions of the paper are rather limited. Several variations of the original NAF algorithm are proposed, and are experimentally compared on four reasonably difficult problems of the kind identified in the theoretical analysis. However, I do not see any overwhelming superiority of the new variants in comparison with the original NAF algorithm, except maybe on the Dubins car. At any rate, for practitioners of RL faced with a continuous control problem, the main question is probably not which of the NAF variants is best, but how these variants compare with other RL algorithms that do not leverage the knowledge that the value function must be quadratic in the control effort everywhere in the state space. Maybe a comparison with several such algorithms would better persuade practitioners to use the NAF algorithms and its variants?\n\nAlso, I am not entirely sure what this paper has to do with the main topic of the conference, learning representations. Maybe the way the value function is represented by means of the three neural networks for V, mu, and P is related to representations, but this representation is not learned, but rather deduced based on understanding of the structure of the cost functional of the problem. I like the paper in general, but it might be of more interest to the audience of a control conference than to that of ICML.  \n\nSome minor typos;\nP.2, first paragraph: \"whose dynamic, in fact, described\" -> \"whose dynamics are, in fact, described\"\nP.7, caption of Figure 1: \"BNAD\" -> \"BNAF\"?\nP.7, first paragraph in 5.2: \"learn their using\" -> \"learn their parameters using\"?\n\n", "summary_of_the_paper": "The paper investigates the applicability of quadratic approximations to the advantage term in the value function of RL problems in continuous action spaces, and the effectiveness of the previously proposed NAF algorithm for finding optimal policies for such problems. The main contribution is identifying a class of optimal control problems where the cost is a quadratic form of the control effort (commonly known as \"minimum-effort problems\"), and proving theoretically that the value function for such problems can be approximated arbitrarily well by the kind of quadratic approximations eteh mployed by the NAF algorithm.", "summary_of_the_review": "The paper analyzes under what conditions the value function of an optimal control problem in continuous control spaces can be approximated by a quadratic function of the control effort, and prove that this approximation is reasonable when the cost functional itself includes a term that is quadratic to the control effort. Several modifications to the previously proposed NAF algorithm are described and tested, but it is not clear how much better or faster they learn in comparison with standard RL algorithms that do not make use of such quadratic approximations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper concerns learned Q functions in continuous action spaces wherein the action-value function is assumed to be quadratic in the action variable, and thus can be maximized in closed form. The paper identifies a class of optimal control problems for which the approximation is reasonable and produces a proof to this effect.

Reviewers found the manuscript clear (6AfP). Reviewer J1Yy notes that the main result of the paper is useful and good to have, as "problems with non-linear dynamics but costs quadratic in the control effort are very common in practice and of high practical significance". On the negative side, reviewer J1Yy considered the contribution beyond the central proof rather light and the empirical study inconclusive and questioned the appropriateness of the venue; a comparison to DDPG was added and while a convincing argument was made as to ICLR's appropraiteness, J1Yy was not willing to move their score beyond a 4 (it does not seem the upward adjustment was actually made). 6AfP noted concerns with the presentation and number of seeds, though their concerns seem to have been addressed in an update. 8xxB was the paper's most ardent critic, who found fault with the presentation (starting with the title). The core of 8xxB's criticism seems to be that by narrowing the scope of problems considered, we are left with a problem domain that is already well solved by classical control, as well as contention about the use of "continuous". The authors offered a thorough rebuttal but the authors and 8xxB were unfortunately unable to see eye to eye on these issues. 8xxB requested more explanation, though a request by the authors to specify the precise scope of what further was required went unanswered.

The AC's own reading of the paper matches J1Yy's assessment most closely. There is a contribution here, in the form of connecting previously proposed RL algorithms for continuous action spaces and discretized time to the literature on optimal control, and exploring cases that match NAF's inductive assumptions, but agree that the contribution is of a rather limited nature. I also believe that the paper has improved through the feedback of reviewers J1Yy and 6AfP. I hesitate to recommend acceptance given a universally negative appraisal and in particular the fact that 8xxB was not satisfied in the end. I hope the authors will continue to improve the manuscript with a more thorough empirical interrogation and adjustments in presentation in light of 8xxB and 6AfP's feedback.
Decision: Reject