Continuous Deep Q-Learning in Optimal Control Problems: Normalized Advantage Functions Analysis
Authors: Anton Plaksin
Stepan Martyanov
Venue: NeurIPS 2022 Accept (2022)
Keywords: continuous reinforcement learning
deep q-learning
optimal control problems
normalized advantage functions

Abstract: One of the most effective continuous deep reinforcement learning algorithms is normalized advantage functions (NAF). The main idea of NAF consists in the approximation of the Q-function by functions quadratic with respect to the action variable. This idea allows to apply the algorithm to continuous reinforcement learning problems, but on the other hand, it brings up the question of classes of problems in which this approximation is acceptable. The presented paper describes one such class. We consider reinforcement learning problems obtained by the discretization of certain optimal control problems. Based on the idea of NAF, we present a new family of quadratic functions and prove its suitable approximation properties. Taking these properties into account, we provide several ways to improve NAF. The experimental results confirm the efficiency of our improvements.

Continuous Deep Q-Learning in Optimal Control Problems: Normalized Advantage Functions Analysis
Anton Plaksin
Stepan Martyanov IMM UB RAS, Ural Federal University
IMM UB RAS, Ural Federal University
Yekaterinburg, Russia
Yekaterinburg, Russia
a. a.r. wt..plaaksin@@mai.ccom
martyanovatt@mail.comm
Abstract
One of the most effective continuous deep reinforcement learning algorithms is normalized advantage functions (NAF). The main idea of NAF consists the approximation of the Q-function by functions quadratic with respect to the action variable. This idea allows to apply the algorithm to continuous reinforcement learning problems, but the other hand, brings up the question of classes of problems in which this approximation is acceptable. The presented paper describes such class. We consider reinforcement learning problems obtained by the time-discretization of certain optimal control problems. Based on the idea of NAF, we present a new family of quadratic functions and prove its suitable approximation properties. Taking these properties into account, we provide several ways to improve NAF, The experimental results confirm the efficiency of our improvements.
Introduction
The standard reinforcement learning (RL) [Sutton and Barto, 2018] setup consists of an agent interacting with an environment. At each step of the interaction, the agent determines an action based on its policy and its current state, gets reward, and makes a transition to the next state. The aim of the agent is to learn the policy that maximizes the sum of rewards.
One of the most widespread approach for solving RL problems Q-learning [Watkins and Dayan, 1992], which consist in seeking the optimal action-valus function (Q-function) as a solution of the Bellman optimality equation. After the learning. the agent can act optimally by the found Q-function. Initially, Q-learning approach was applied for solving RL problems with finite state and action spaces. In this case, Q-function can be represented by finite table. For the case of a large or continuous state space, Q-learning has later been extended to Deep Q-learning algorithm [Mnih al., 2015] which allows to seek the approximations of Q O-function in the class of neural networks by means of the stochastic gradient descent. Deep Q-learning and its modifications have shown efficiency for a range of challenging tasks [Wang et al., 2016, van Hasselt et al., 2016. Schaul et al., 2016, Hessel et al., 2018]. However, this algorithm can not be directly applied for solving RL problems with continuous action spaces. The reason is that Deep Q-learning involves maximizing of approximate Q-function by the action variable on each step of the learning. which a complex procedure for continuous action spaces. Among various approaches to overcome this problem [Lillicrap et al., 2016, Haarooja et al., 2017, Wu al., 2017, Kalashnikov et al., 2018, Nachum et al., 2018, Fujimoto al., 2018, Lim et al., 2019, Ryu et al., 2020, Lutter et al., 2021]. we focus on the idea of the normalized advantage functions (NAF) algorithm [Gu et al., 2016], which consists in the approximation of Q-function by the function quadratic with respect to the action variable. allows to calculate the maximum quite fast and precisely and solve some challenging control problems [Gu et al., 2017, Dong et al., 2018, Ikemoto and Ushio, 2021], but on the other hand, it brings up the question of classes of RL problems
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in which this approximation is acceptable. The presented paper describes one of the possible answers to this question.
Note that the class of L.QR problems [Bradtke et al., 1994] has Q-functions quadratic with respect to the action variable. However, this class, being quite special, is not suitable for the description of non-linear controlled processes. In this paper, we study a wider (in some sense) class of RL problems. We consider RL problems which are obtained by time-discretization of finite-horizon optimal control problems [Bardi and Dolcetta, 1997] in dynamical system is described by affine in control ordinary differential equations and the cost functional is quadratic in control. Note that lot of RL problems with continuous action spaces arise from control problems for mechanical or robotic systems [Lillicrap et al., 2016, Gu et al., 2016, Haamoja et al., 2017, Gu et al., 2017, Kalashnikov et al., 2018] which are, in fact, described by affine in control ordinary differential equations. Besides, often, the cost functional in control problem can be determined as quadratic in control. For instance, if the quality index in an initial problem does not depend on control, then one can add small quadratic term such that, on the one hand, the quality index becomes quadratic, and on the other hand, the sense of the problem does not changed much. Thus, the class of control problems under consideration seems quite general.
For this class, based on the idea of NAF, we present a new family of quadratic functions and prove that, first, this family is sufficiently rich to approximately solve the Bellman optimality equation (Theorem 1), and second, any sufficiently accurate solution, from this family, of the Bellman optimality equation allows to approximately obtain the optimal policy in the corresponding optimal control problem (Theorem 2). Moreover, we prove that it is impossible to get the same results for the original family of functions from Gu et al. [2016] (Theorem 3). Next, we provide several ways to utilize the obtained theoretical knowledge in order to improve NAF The experimental results confirm the efficiency of our improvements.
Note that, in contrast to LQR problems, we can not say that Q-function is quadratic in our class of problems. Moreover, it not true in general case. Nonetheless, according to presented statements (Theorem 1,2), Q-function can be approximated by quadratic functions with sufficient order of accuracy to obtain near-optimal policy.
Background
The standard reinforcement learning (RI.) setup consists of an agent interacting with an environment [Sutton and Barto, 2018). This interaction is described by a Markov Decision Process (MDP), which a tpple (s,,, P, R. Po. r)) where S is a state space, distribution, R(s, u) is a reward function, po(s) is an initial state distribution, and y E [0, 1] is a discount factor. The aim of the agent is to learn optimal policy (s) which maximizes the value
an action space, P(s'|s ss u) is a transition
J(n) E Σ R(ss, w/) su po(su), pu(s),, Si+1 P(s141ls) uz), 0, 1,
In the general statement of reinforcement learning problems, a policy of the agent be stochastic, however, within this paper, we assume that the policy is deterministic.
One of the most effective approach for solving RL problems is Q-learning [Watkins and Dayan, 1992], which consist in seeking the optimal action-value function
us) = sup E
Σ R's,, uu) 50 = 5, u0 u.
8:41 ~ P(s ui+1 = ((8++1), i i= 0, 1,2,
as a solution of the Bellman optimality equation
Q+ u) = E R(s, u) + max (= w')ls' ~ u))
w'ell
In other words, the agent solves the following minimization problem:
sup Q(s,u) E[R(s, SES,uEL1
max ((5(()() P(51ls + inf.
(1)
w'ell
Q

---

If the agent knows the function Q~ a). it can act optimally by the greedy policy
(s(s) E argmax Q=(s,u).
MEW
Initially, the Q-learning approach was applied for solving RI. problems with finite state and ac- tion spaces. In this case, the Q-function is represented by a finite table and problem (1) is finite- dimensional. For the case of large or continuous state space. Q-learning has later been extended to Deep Q-lcarning algorithm [Mnih et al., 2015] which allows to seek approximations of Q-function in the class of neural networks Qxx, ul ((09), where 0 is the parameter vector of the neural network. During the learning, the experiences (8s, Us ri; 81+1) 1) are stored in the buffer D and simultaneously the parameter vector d is updated by means of the stochastic gradient descent minimizing the loss function
L(B = [Q((s, ul09) y)² u, ( - maxxmmaxxxaoo Q(s', u v'109). a'Ell
(2)
where (D) is the uniform distribution on D. Deep Q-learning and its modifications are effective for a range of challenging tasks [Wang et al., 2016, van Hasselt et al., 2016, Schaul et al., 2016, Hessel et al., 2018]. However, note that this algorithm can not be directly applied for solving RL problems with continuous action spaces. The reason is that Deep Q-learning involves the maximizing in (2) on each step of the learning, which is complex procedure for continuous U. Among various approaches to overcome this problem [Lillicrap et al., 2016, Haarnoja et al., 2017, Kalashnikov et al., 2018, Lim al., 2019, Ryu et al., 2020, Lutter et al., 2021], we focus an idea of the normalized advantage functions (NAF) algorithm [Gu et al., 2016]. This idea consists in the approximation of Q-function by the following quadratic with respect to u functions:
Q(s, ul09) V(s u(s,
1
A(s, 0A)
- u(s(0"))² (u
where V(s(oV,, )(s(("), and Ps) are neural networks with parameters gV 8", and AP, re- spectively; P(s is a positive-definite square matrix for each and OP; 0 {0a, 0P) and 0² = 10a,00 Under the condition
ms)(0)) E U,
(4)
it allows to get the maximum and argmaximum values directly by values of Vs800V) and p(s)0R):
max Q(s, u|09)= V(s Argmax Q(s, ul09) ()s) well
(5)
uENl
but the other hand, it brings up the question of classes of RL problems in which quadratic approximations is acceptable. Below, we describes one such class.
Problem statement
In this section, consider certain class of optimal control problems and show that discrete
approximations of these problems can be formalized as RL problems.
Consider the following optimal control problem: it is required to maximize the functional
J(u(-))= ((((T)) =((()) - J/" (a(t,=(()) u())
(6)
over all u(-), where x(-) is the solution [Filippov, 1988, $1] of the differential equation
dt x(t) f(t, + g(t, ((t)x()) te [0,T].
(7)
under the initial condition
.=( =
(8)
Here is the time variable, T > 0 is the terminal instant of time, xx() IR" is the current state vector, u(t) E the current control action vector forming the measurable function w(-), U R is the nonempty compact set, _ is the fixed initial state vector, f(t x) R" 2², 2) gaxm.
3

---

q(t, E R. r(t, E Rm*x, (t.z) E [0,T] I" are continuous with respect to and continuously differentiable with respect
functions, r(t, æ) is the positive-defiite matrix for each (t. x) [0. T) RR" and (xx) E IR, a E R" is the continuously differentiable function. We assume that there exists a constant cfg > such that
(l, + g(t, x)u/l s (1 + +xxl))sss
(1, E [0, x RR" U
Note that, under these conditions, for each function u(-), there exists unique solution xx.) of equation (7) under the initial condition (8) [Filippov, 1988, $1].
Define the value function in optimal control problem (6), (7)
(((x(T)) J." (g((,x(()) + -(() +(()))))))))))))))))))))))))))))
V. v..) -sup wa-))
(10)
(1,, E (0, R",
where, for each u(-), 2x-) is the solution of equation (7) on the interval [t,, T] under the initial condition zzt.) T..
Define the sets
S ((4,)) [0, R": |||||| S (1 [1=1)e=ro'' S(1) {s R": (1, E S). (11)
Let l E N. At, T/k, and t = i i E 0, Consider the corresponding discrete optimal control problem: it required maximize the function
k-1
Ju (uo, U1,.. uk ux-1) = (xx) - Atx
Σ
(o(te.z;) + of r(tti x;)),,
(12)
==0
over all u, E U. U.LE 0, - I, where (Zg. #1,...,, Ik) defined by
20 2, I++1 zz + (f,t, (4) g(t,, yu, ) Ate. E 0, I.
(13)
Let us show that problem (12), (13) can be formalized as the RL problem. First, we define the state and actions spaces, the initial state distribution, and the discount factor as follows:
S = ((tt) x S(t)) UT,, u= U, po(so) 8(30 (0,z)), 1
(14)
Here $T is some fictional terminal state, is Dirac delta distribution. Next, for every i E 0, E S(t:), and u E U, we define the transition distribution and the reward function
P(s'1= = x), u) = 5(s'= = (1)+1,2))) R(s = (t,, x), u) = - (g(t;, + u² x)t) Atk.
(15)
where x' = x + (f,,, x) + g(t,, Atx) Taking into account (9) and (11), one can prove the inclusion (t4+1, E S. Hence, the transition distribution P is well-defined. For i = k, we set
P(s'ls (1), u) 8(s' ST), R(s (lx, x). u) (x), x E S(ta), u we U. (16)
In order to make dynamical processes (13) formally infinite, we put P(s'I (sss, u) 6(s' ST), R(sT, O, U.
(17)
Thus, define MDP which describes the RL problem corresponding problem (12), (13). Next, we show that such RL problems is suitable for using quadratic approximations of the Q-function.
Quadratic approximations of the Q-function
Denote by Q the family of functions Q such that (t, u) V(t, A(t, u), u) (0,T) RD U,
(18)
where
a(t,z,u) -u- ((t
+(u(t, (i(1,,))) p(1,))(((t,z)) (ptt, x))
(19)
p(t,x) argmin (u' (((t,))))))))))))))))))) P(t, (((, x)).
n'EU

---

Here ((t, x),, x)(t,z), and P(t, x) are continuous functions; P(t, is pesitive-definite square matrix for each xt E [)) R".
Note that, functions Q from the family Q satisfy the equalities
max Qt,, u) V(t,x), Argmax Q(t, jut,,)), uEU
(20)
wEU
as well as (see (5)) quadratic functions from family (3). However, these function families are different. The difference is that we do not assume the inclusion ju(t, x) E U as opposed to assumption (4), In other words, the vertex of a quadratic form (3) belongs to the set U. while this is not required in the function family Q Thus, Q is wider family of quadratic functions.
The theorems below establish connection between the optimal control problem (6)-(8) and mini- mization problems (1) for MDP (14)
Theorem 1. Let the value function (t x)) be continuously differentiable. Then, for every > O. there exists k, > 0 such that, for every ko 2 k. the function Q E Q defined by (18) and (19) where
V(t, 2t, p(t,z) = r(t, p(t,z)
(21)
satisfies the inequality
Qu1,, u) + (g(4i; x) + at (((,,
max Q(li+1, u/) S EDA w'EU
(22)
x' ='=+ (f(t,, + g(tz, )u) Atk, uEU, x S(t), where we assume Q(4,, x u') = (x¹).
i
Theorem 2. Let the value function (t, .) be continuously differentiable. Let >> and k, > 0 be defined according to Theorem 1. Take 2 k. and suppose that function Q E Q satisfies inequality (22). Then the following estimate holds:
Ju Ui Uk 1k-1) sup (u(()) 3,'s, wl-)
(23)
where the function Jk (ug. U1 - is defined by (12) with u, = ((tt,,)), E 0, - and the function uut,, x²) is defined by Q according to (18) and (19).
Thus, Theorem 1 shows that the function family Q is sufficiently rich to contain approximate solutions of problem (1) with a predetermined accuracy and Theorem 2 establishes that all such approximate solutions contained in Q allow us to get the policy, which approximately provides the optimal result in optimal control problem (8).
Now, let us consider the original family of functions from Gu al. [2016]. Denote by QNAF the family of functions Q such that
((t,z,u) V(t, x)) (u x) P(t, ())tu))t(tttt =(t, (t, u) E [0, R" U, (24) where v(t,z), plt, x), and P(t, x) are continuous functions; P(t, x) is a positive-definite square matrix for each (t, z) [0. x R"; u(t,z) U for each (t, E [0, x R". The theorem below establishes that if we take the family QNAF instant of Q, then Thcorem I can not be proved even in the simplest cases of optimal control problem (6)-
Theorem 3. Let n m 1, T 1, U 1-,, 1), f(t,x) = <(t, = 0, g = a(t, = 1,
(x) = = and = 2. Then, for every 2 and Q E QNAP, there exist i 0, k: - 1, x E 5(4). and u E U such that
QQ u) + Al max Q(1441, v()l Atz/8, x' x 4uAi. n'el
where we assume Q(t,, x' w/) (x')2
Proofs of the theorems are given in the Appendix A.
Note that, in the particular case U IH, the family Q coincides with family QNAF. However, within this paper, is required to be bounded, which significantly for the presented theorems. Indeed, unboundedness U implies unboundedness the set S (see (11)) which make it impossible to obtain inequality (22) uniformly on this set. Thus, these theorems are not applicable to any unbounded U and, in particular, U = m
5

---

Table I: Parameters in the examples of optimal control problems
Name
T
m
T
f(t,
g(t,z)
q(t, r(t, x))
Van der Pol oscillator
2
11
[-,
(1 -
(1-2f)a2 27)o02)
I2
-11
0
0.05
0 (1(x)
Pendulum
2
1
5
|-
D22 14.7 sin(x1)
0
0.01
(2(x)
3
(co((23)
Dubins car
3
2m
|-0.5, 1]
sin(23)
0
0.05
x3(x)
1
0
A target problem
0
6
2
10
[|1,1]
0.001 (ι(2)
J6
233
222
24
5 Experiments
We consider four examples of optimal control problems (6)-(8) described in Table 1, where
(I()) = =x
x2. (2(2) = - | - 0.1 0.1 0z(x) = - |21 4| |2:2| |23 0.75m
(3(2) = x33 (23 2)² (#t4 2)
Van der Pol oscillator is a famous model of a non-conservative oscillator with non-linear damping. The aim of the control is stabilize the oscillator the terminal time.
Pendulum is a traditional problem for testing control algorithms. The aim of the control is the stabilization of the pendulum in the top position at the terminal time.
Dubins car is a quite famous model which describes motion of the point particle moving constant speed on the plane. The problem is to find control providing the closeness of the motion with target point at the terminal time.
target problem is an optimal control problem presented in Munos [2006]. The dynamic system describes hand holding spring which is attached mass. It is required to control the hand such that the mass achieve the target point at the terminal time.
5.1 Bounded NAF
First, modify NAF algorithm, proposed in Gu et al. [2016]. based the function family Q. Note that, the considered examples have U = [a, 3m Denote clip(v) = max (a, min (3, v)), E RT a, B E R Then, according to (18) and (19), use the following approximation of the Q-function, within NAF algorithm:
Q1,,r, u109)- = 2,
A(t, att,0,u004 u/04) = (1t P(t, ((t.:1)P)(u-i(()0)0))))
+(clip(j((1, x (0P) já(,, x00))) P(1, ( (clpp(((())))p(( - ju(l, x00)))
where x((v), ju(t, (00) and P(I, are neural networks with parameters 0V 0² and 02. respectively; P(L, x: x0f) is positive-definit square matrix for each (1, and 0P; 0 = (0P OP) and 0 [04, 0²) To be short, we call this algorithm Bounded NAF (BNAF), because it is essential for our modification of NAF that the set U is bounded.

---

Van der Fol osciliaorr
Pend Juuu
-3.0
-10
A0C
1006
200
1003
Dub
problem
-50
1000
ENA
-110
Ade
BhAT
CR-RNAA
-20
200
Figure 1: Performance of NAF, BNAF, RB-BNAF, GB-BNAF, DDPG algorithms averaged over 10 seeds. The curves are averaged for 20 episode for better visualization.
5.2 Reward-baser BNAF
In practice in many cases, the function r(t, known. For example, if researcher who formalizes a control problem has the right to determine an appropriate cost functional. Then, based on the second equality in (21), the function r(t, At can be used instead of the neural network P(t, ( to reduce the number of learning parameters. This variant of BNAF is called Reward- -baed BNAF (RB-BNAF).
5.3 Gradient-based BNAF
One can also imagine the situation in which, taking into account mechanical laws and experimental measurements, the function g(t, .)) also turss out to be known for the researcher. Then, according to the third equality in (21), the function
1 - (t, (t, 2
a(t, x
can used instead of the neural network ja(t, æ 2 which also reduces the number of learning parameters. This variant of BNAF is called Gradient-based BNAF (GB-BNAF),
5.4 Experimental results
We use the same learning parameters every our tasks. We apply neural networks with two layers of 256 and 128 rectified linear units (ReLU) and learn their used ADAM with the learning rate Ir
- 3 We use batch size nıs = 128 and smoothing parameter т = le 1e- Also we take performed
A/ 0.1. All calculations
personal computer in standard way.
compare NAF, BNAF, RB-BNAF, GB-BNAF algorithms, and DDPG algorithm presented in [Lillicrap et al., 2016]. We utilize the hyperbolic tangent the output layer of the neural network

---

Let us also emphasize that the presented algorithms can be successfully applied to problems without the dependence of the cost functional (6) on u()). ). In this case, the quadratic term with small r(4, xx()) can be added in the initial problem. Then, on the one hand, smallness of this term does not result in corresponding to the policy sus](")) in NAF and DDPG algorithms in order to provide inclusion (4). Figure shows learning curves of the algorithms for the considered examples. One can note that RB-BNAF algorithm has a good performance in all examples. GB-BNAF is slightly inferior to RB-BNAF everywhere except target problem. DDPG and BNAF have excellent results only one example out of four. NAF algorithm shows not the worst performance only in Pendulum example.
Thus, taking the presented experiments into account, we can give the following general recommenda- tions. For solving optimal control problems (6)-(8), it is rational to use GB-BNAF and especially RB-BNAF algorithms along with NAF and DDPG algorithms. There high probability that they will show better results and more stable learning.
poor performance of the algorithms (see pendulum and a target problem examples) and on the other hand the obtained results are close to the solution of the initial problem.
6 Related works
Many different ways to apply reinforcement learning approaches for solving optimal control problems are investigated [Baird, 1994, Doya, 1995, 2000, Munos, 2006, Tallec et al., 2019, Kim and Yang, 2020, Lutter et al., 2021]. Among them, a dineeesscetizationn is perhaps the most obvious and widely used tool [Lillicrap et al., 2016. Gu et al., 2016, Haarnoja et al., 2017, Gu et al., 2017, Kalashnikov et al., 2018]. From the theoretical point of view, it is known [Bardi and Dolcetta, 1997, p.388] that solutions of rime-discrete optimal control problems converge the solution of the initial problem as the discretization step tends to zero. In the present paper, we also use the dine-discrtiization and study approximating solutions of Bellman equations and the corresponding greedy policies depending on the discretization step (see Theorems and 2). Other studies of dependencies on the discretization step of reinforcement learning methods can be found in Munos [2006], Tallec al. [2019].
We focus on the idea from Gu et al. [2016] to expand Q-learning algorithm to optimal control and reinforcement learning problems with continuous actions. Other approaches for solving such problems are investigated in Lillicrap et al. [2016], Haarnoja et al. [2017], Kalashnikov et al. [2018], Lim et al. [2019], Ryu et al. [2020], Lutter et al. [2021]. The paper Lutter et al. [2021] seems the closest to the presented paper. In Lutter et al. [2021], the similar optimal control problem and feedback control policy are considered, but the proposed algorithms require knowledge of the right-hand side of differential equations in contrast to algorithms suggested in presented paper.
Let us also note the family of functions (18), (19) is included, in some sense, to families of Q-functions considered Wang et al. [2016], Tallec et al. [2019]. It seems to be expected because more general classes of problems are considered in these papers. Nevertheless, the proposed algorithms and results are very different from presented in this paper.
7 Conclusion
We consider reinforcement learning problems which are obtained by time-diccretization of finite- horizon optimal control problems described by affine in control ordinary differential equations and quadratic in control cost functionals. Based on the idea of NAF algorithm Gu et al. [2016], we present new family of quadratic functions and prove that, first, this family is sufficiently rich to approximately solve the Bellman optimality equation (Theorem 1), and second, any sufficiently accurate solution, from this family, of the Bellman optimality equation allows to approximately obtain the optimal policy (Theorem 2). Moreover, we establish that it is impossible to get the same results for the function family from Gu al. [2016] (Theorem 3). Next, we utilize the obtained theoretical results to propose algorithms improving NAF, The experimental results confirm their efficiency.
8

---

Acknowledgments and Disclosure of Funding
The work was performed as part of research conducted in the Ural Mathematical Center with the financial support of the Ministry of Science and Higher Education of the Russian Federation (Agreement number 075-02-2022-874).
References
L. C. Baird. Reinforcement learning in continuous time: Advantage updating. Proceedings of 1994 IEEE International Conference on Neural Networks, 1994.
M. Bardi and I. C. Dolcetta. Optimal Control and Viscosity Solutions of Hamilton-lccoii-Belln Bellmba Equations. Birkhauser, Boston, 1997.
J. Bradtke, B. Ydstie, and A. G. Barto. Adaptive linear quadratic control using policy iteration. Proceedings of American Control Conference, 1994. doi: 10.1 (0.100/ACCC..999433222
X. Dong, J. Shen, W. Wang, Y. Liu, L. Shao, and Porikli. Hyperparameter optimization for tracking with continuous deep q q-learning Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 518-522 2018. doi: 10.11099CVPR.221880000)
K. Doya. Temporal difference learning continuous time and space. Proceedings of the 8th International Conference Neural Information Processing Systems, pages 1073-1079, 1995.
K. Doya. Reinforcement learning in continuous time and space. Neural Computation, 12:219-245.
2000. doi: 10.1162/08976600300005555
A. F. Filippov. Differential Equations with Discontinuous Righthand Sides. Springer, Berlin, 1988.
S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. Proceedings of the 35 th International Conference on Machine Learning, 80:1587-1596, 2018.
S. Gu, T. Lillicrap. I. Sutskever, and S. Levine. Continuous deep q-learning with model-based acceleration. Proceedings of the 33rd International Conference on Machine Learning, 48:2829- 2838, 2016.
S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- off-policy updates. Proceedings 2017 IEEE International Conference on Roboties and Automation (ICRA), 2017. doi: 10.1109//RRA22017.79998
Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies. Proceedings of the 34th International Conference Machine Learning, 70:1352-1361, 2017.
M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. Proceedings of The Thirty-Second AAAI Conference on Artificial Intelligence, pages 3215-3222, 2018.
Ikemoto and T. Ushio Continuous deep q- q-learning with simulator for stabilization of uncertain discrete-time systems. arXiv:2101.056400 2021.
D. Kalashnikov, A. Irpan, P. Pastor. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly. M. Kalakrishnan, V. Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-based robotic manipulation. 2nd Conference Robot Learning (CoRL 2018), 87:651-673, 2018.
J. Kim and Yang. Hamilton-jacobi jacbbi-belbmabaaaannnnnaaaaaaaaa equations for lelaarnin in continuous time, Proceedings of the 2nd Conference on Learning for Dynamics and Control, 120:739-748, 2020.
T. P. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations (ICLR), 2016.

---

S. Lim, A. Joseph, L. Le, Y. Pan, and M. White. Actor-expert: A framework for using q g-learning in continuous action spaces, arXiv: 15t.:088100913, 2019.
M. Lutter, S. Mannor, Peters, D. Fox, and A. Garg. Value iteration in continuous actions, states and time. Proceedings of the 38th International Conference on Machine Learning, 139:7224-7234, 2021.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg. and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529-533 2015. doi: 10.10388naturee12555
R. Munos. Policy gradient in continuous time, Journal of Machine Learning Research, 7:771-791, 2006.
O. Nachum, M. Norouzi, K. Xu, and D). Schuurmans. Trust-pel: An off-policy trust region method for continuous control. International Conference on Learning Representations (ICLR), 2018
M. Ryu, Y. Chow, R. M. Anderson, C. Tjandraatmadjaa and C. Boutilier. Caql: Continuous action q-learning. Proceedings of the Eighth International Conference Learning Representations, 2020.
T. Schaul, J. Quan, Antonoglou, and D. Silver. Prioritized experience replay. International Conference Learning Representations (ICLR), 2016.
R. Sutton and A. G. Barto. Reinforcement Learning An Introduction (Second Edition). The MIT Press, Cambridge, Massachusetts, 2018.
C. Tallec, L.. Blier, and Y. Ollivier. Making deep q-learning methods robust to time discretization. Proceedings of the 36th International Conference on Machine Learning, 97:6096-6104, 2019.
H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. Proceedings of the Thirtieth AAAI Conference Artificial Intelligence (AAAI-16), 30(1):2094- 2100, 2016.
Z. Wang. T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas. Dueling network architectures for deep reinforcement learning. Proceedings of The 33rd International Conference on Machine Learning. 48:1995-2003. 2016.
C. Watkins and P. Dayan. Q-learning. Machine Learning, 8:279-292. 292. 1992. doi: 10.1007) BF00992698.
Y. Wu, E. Mansimov, S. Liao, R. Grosse, and J. Ba. Scalable trust-region method for deep reinforce- ment learning using fronecker-factored approximation. 31st Conference Neural Information Processing Systems, 2017.
10


Reviews:
[{"review_id": "LD6W-OdcmFu", "reviewer": "Reviewer_mYi8", "summary": "The paper provides analysis of a method for solving optimal control problems based on approximating the value function by means of deep neural networks that results in a quadratic parametrization of the value function in the control effort, thus allowing for an efficient solution of the optimization step needed for deep Q learning. Several theorems are proved to identify and analyze the class of optimal control problems this method is applicable to. Several variants of the algorithms are investigated empirically on four test problems, in comparison with one well known method for continuous Q learning, DDPG. ", "questions": "Is DDPG really the current state of the art to be used as a baseline? Shouldn't PPO and/or SAC also be included in empirical comparisons?\n\nOn page 2, line47, by \"quality index\", do you mean the cost (respectively, reward) function?\n\nA minor typo: \nP.8 L.199: \"greedy politics\" -> \"greedy policies\"", "limitations": "I think the authors have adequately addressed the limitations of the proposed methods - in fact, a major objective of the paper is to analyze and understand which classes of optimal control problems these methods are applicable (and possibly, limited) to. I do not see any negative societal impact of this work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main contribution of the paper is probably the empirical investigation of the proposed method that illustrates that two variants of the algorithm, RB-BNAF and GB-BNAF, have significantly higher performance than DDPG on all test problems, and as such, could be good candidates for solution methods for optimal control problems of a suitable class. Analysis demonstrates that this class included non-linear control-affine dynamical systems with cost functions quadratic in the control effort. These methods could be a good addition to the toolbox of practitioners. Both methods require at least partial knowledge of the system being controller (the reward function and the control-affine term in the dynamics), so these methods cannot be considered proper reinforcement learning methods, but as optimal control methods, they appear to be valid and quite competitive.\n\nHowever, because the basic NAF algorithm is already known in the literature, this paper's contribution is relatively minor. The class of optimal control problems the methods are applicable to is important in practice, but not that universal. The authors are arguing that by adding a quadratic term to the cost function, many control problems can be made to be of the kind they are considering, but this is a somewhat contrived way of claiming the generality of the method.  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K6NAHW6koGA", "reviewer": "Reviewer_e5w3", "summary": "Q-learning is a popular approach to reinforcement learning that is difficult to extend to continuous action spaces. Normalized advantage functions (NAFs) address this issue by assuming that the Q function approximation is quadratic with respect to the controls. This enables us to determine the policy by efficiently computing the argmax with respect to controls of our approximated Q function. While it seems to work in practice, which systems this approximation is suitable for is an open question. To address this concern, the paper analyzes the case where the dynamics are control-affine, the cost is quadratic in the controls, and the problem is formulated in continuous time. First, the authors map the continuous-time optimal control problem to a discrete-time reinforcement learning problem. Next, they propose a slight variation of the NAF approximation family and show how to set the terms given knowledge of the known optimal control problem to bound the Q-function approximation error. They then present how this translates into a bound on the performance. Additionally, they show that these bounds do not hold if we use the original NAF approximation via a counterexample. Finally, the authors evaluate a couple variants of their new approximation family on a set of benchmarks and compare their performance to both the baseline NAF implementation and a policy gradient method.", "questions": "*  If the controls are unconstrained, would Theorems 1 and 2 hold for the original NAF approximation or is there something more that would prevent this?\n* Are the observed benefits of BNAF and its variants really due to this new approximation family? Would I observe similar benefits if I used the modifications to NAF I discussed above?", "limitations": "The authors do discuss the limitations of their work in the introduction. Specifically, they mention that the analysis in this paper only holds for a specific class of problems. Moreover, the Q function approximation they propose is not the true form, even in the more limited class of problems they consider. However, they are able to bound the approximation error and performance, which significantly adds to the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The question of which control problems the NAF approximation is acceptable to is a relevant and important question. Moreover, the approximation and performance bounds presented in this paper are applicable to a useful class of problems that apply to a wide variety of real-world robotics tasks. The authors also present a number of variants of their Q-function approximations which can incorporate knowledge about the reward and dynamics if known. The experimental results illustrate the benefits of this extended approximation family and the variants in terms of the overall reward achieved on a number of benchmarks. Additionally, the paper is fairly well organized and explains most of the relevant background material.\n\nHowever, the addition of control constraints to the problems considered seems to be changing the question the paper originally asked. Specifically, there's the question of which problems the NAF approximation is suitable for, and then there's the question of how to adapt NAF in the case of control constraints. Both are useful questions to ask, but they need to be disentangled somehow. For instance, it is unclear if the proposed extended approximation family would be important in the case of unconstrained controls. The main difficulty in applying Theorem 1 and 2 to the original NAF approximation appears to be that we cannot set the policy directly to the optimal as in (21) due to the presence of these constraints. This is what motivates the introduction of the new family in (18, 19). However, in the case where the controls are unconstrained, it seems like the original NAF approximation should be sufficient. While the theorems presented in this paper are interesting and useful regardless, the authors should elaborate on this point.\n\nMoreover, even if the controls are constrained, it is hard to tell from the experiments if the benefits observed really come from this new family or just from incorporating additional information into the Q function. For instance, in the case of box constraints on the controls, one could easily set $\\mu$ in the original NAF family such that the neural network output is also clipped or passed through a scaled sigmoid. Similarly, for GB-BNAF, one could also set $\\mu$ to be $\\frac{1}{2} r^{-1}(t,x) g^T(t, x) \\nabla_x V(t, x | \\theta^V)$ but clipped or passed through a scaled sigmoid. It would greatly strengthen the paper to disambiguate the source of benefits by performing some ablations. \n\nFinally, there are a number of grammatical errors throughout the paper that need to be addressed. A sentence or two describing other approaches besides NAF to tackle continuous actions in Q learning would add to the paper. It would also be nice for the authors to include a conclusion section instead of just ending on the related work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GGxi-uW1uOQ", "reviewer": "Reviewer_RkHa", "summary": "This data identifies a class of (discretized) optimal control problems for which a quadratic (with respect to actions) approximation to the problem's Q-function is accurate and gives rise to near-optimal policies.  The proposed quadratic parameterization of the Q-function is similar to, but slightly more general than, that proposed by the original Normalized Advantage Functions paper, and this additional generality is necessary.  The authors test several implementations of this approximation on control problems and find it to be effective.", "questions": "(1) How does the performance of BNAF (+ variants) compare to more state-of-the-art RL algorithms (e.g. SAC) for continuous action spaces, and to what extent is BNAF compatible with various implementation details that improve performance in these baselines?\n\n(2) How gracefully does BNAF handle violations of the assumptions in equation (6)?", "limitations": "The authors are clear about the limitations of BNAF and related algorithms, unambiguously defining a set of problems to which these approaches are applicable.\n\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper provides a useful theoretical contribution by characterizing a set of RL tasks for which NAF-style Q-function approximations are suitable.  The explanations and derivations are fairly clear and the experiments are easy to follow.\n\nMy principal concern with the paper is that its main contribution, while useful, is rather limited in scope.  It is unclear to me whether NAF-style approaches are competitive with the state of the art in continuous action-space RL problems.  The authors do compare against DDPG, which is nice, but not against stronger baselines like SAC.  The relevance of this paper's theoretical contribution to the community is ultimately somewhat dependent on the performance of NAF-style algorithms relative to alternatives.\n\nI think the paper has another major shortcoming independent of the above concern, which is that there is no analysis of problems that *do not* lend themselves to a quadratic Q-function approximation.  Obviously the space of all MDPs is too large to categorize in a single paper, but the authors could help clarify the applicability of NAF by analyzing which aspects of the problem formulation in (6) are essential to the paper's theoretical and empirical results.  For instance, how does the suitability of the NAF approximation, and its actual performance, scale as the dependence of cost on u(t) becomes increasingly non-quadratic?  \n\nFinally, two comments on presentation: (1) I think the motivation for the equation (19), and the three implementations in section 5, could beimproved.  As I understand it, the deviations from the original NAF algorithm are made to account for the possibility that the optimal action in a given state may not belong to the set of actions to the agent.  This point could be made clearer in the text.  (2) The reward-based BNAF and gradient-based BNAF algorithms are introduced rather abruptly, without much motivation.  Under what conditions are we likely to know r(t, x) or g(t, x)?  When should we intuitively expect (without seeing the empirical results) these algorithms to outperform vanilla BNAF?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept