Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning
Authors: Harley Wiltzer
Marc G Bellemare
David Meger
Patrick Shafto
Yash Jhaveri
Venue: NeurIPS 2024 poster (2024)
Keywords: distributional reinforcement learning
reinforcement learning
continuous time
advantage updating
stochastic differential equations
Primary area: reinforcement_learning
TL;DR: We establish theory governing the difficulty of policy optimization in high-decision-frequency distributional RL.

Abstract: When decisions are made at high frequency, traditional reinforcement learning (RL) methods struggle to accurately estimate action values. In turn, their performance is inconsistent and often poor. Whether the performance of distributional RL (DRL) agents suffers similarly, however, is unknown. In this work, we establish that DRL agents *are* sensitive to the decision frequency. We prove that action-conditioned return distributions collapse to their underlying policy's return distribution as the decision frequency increases. We quantify the rate of collapse of these return distributions and exhibit that their statistics collapse at different rates. Moreover, we define distributional perspectives on action gaps and advantages. In particular, we introduce the *superiority* as a probabilistic generalization of the advantage---the core object of approaches to mitigating performance issues in high-frequency value-based RL. In addition, we build a superiority-based DRL algorithm. Through simulations in an option-trading domain, we validate that proper modeling of the superiority distribution produces improved controllers at high decision frequencies.

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning
Harley Wiltzer*
Mare G. Bellemare!
David Meger McGill University
Mila-Queee AI Institute
Mila- Hilaee AI Institute
McGill University
McGill University
Patrick Shafto
Yash Jhaveri'
Rutgers University-Newar
Rutgers University-Newart
Abstract
When decisions are made at high frequency, traditional reinforcement learning (RL) methods struggle to accurately estimate action values. In turn, their performance is inconsistent and often poor. Whether the performance distributional RL (DRL) agents suffers similarly, however, is unknown. In this work, establish that DRL agents are sensitive to the decision frequency. We prove that action-conditioned return distributions collapse their underlying policy's return distribution the decision frequency increases. We quantify the rate of collapse of these return distributions and exhibit that their statistics collapse different rates. Moreover, we define distributional perspectives on action gaps and advantages. In particular, we introduce the superiority
probabilistic generalization of the advantage- the core object of approaches to mitigating performance issues in high-frequency value-based RL. addition, we build a superiority- -ased DRL algorithm. Through simulations in an option-trading domain, we validate that proper modeling of the superiority distribution produces improved controllers high decision frequencies.
Introduction
In many real-time deployments of reinforcement learning quintttttttttiive finance, robotics, and autonomous driving, for instance-the state the environment evolves continuously time, but policies make decisions discrete timesteps (h units of time apart) [28]. In such systems, the performance of value-basee agents is sensitive to the frequency w := 1/4 with which actions are taken. In particular, action values become indistinguishable
the time between actions decreases. In turn, in high-freag frebhrryyyncc settings, Baird demonstrated that action value estimates are susceptible to noise and approximation error [20]. Moreover, Tallec et al. exhibited that the performance of popular deep Q-learning agents is inconsistent and often poor [34].
In order remedy this sensitivity, Baird proposed the advantage function and advantage- based variants of Q-learning, Advantage Updating (AU) [20] and Advantage Learning (AL) [2]. Unlike action values, advantages (appropriately rescaled) do become indistinguishable
decision
frequency increases. As a result, Baird, in [20, 2]. demonstrated that advantage-based agents can learn faster and be resilient noise than their action value-based counterparts. Furthermore, Tallec al.. in [34], exhibited that their extension of AU, Deep Advantage Updating (DAU), works efficiently over wide range of timesteps and environments, unlike standard deep Qelamnnigg approaches.
While advantage- based approaches to RL have demonstrated robustness to decision frequency, in this work, we establish that they nevertheless sensitive the frequency with which actions
'Equal contribution. Correspondence har.oy..viltzer@mmne #illzeer@mail.mmgill.. 'CIARR AI Chair.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

are taken. This discovery arises as we answer the question: to what extent is the performance of distributional RL (DRL) agents sensitive to decision frequency? To this end, we build theory within the formalism of continuous-time RI. where environmental dynamics are governed by SDEs, as in [25]. Additionally, we validate our theory empirically through simulations. Specifically, we make the following four contributions:
Distributional Action Gap. First, we extend notions of action gap the realm of DRL. Precisely, we consider the minimal distance between pairs of oction-conditions distributions under metrics on the space of probability measures on RR. We observe that some metrics are viable for this extension, while others are not. This formalism sets the stage for analyzing the influence of individual actions as well as decision frequency on, for example, an agent's 's return distributions.
Collapse of Distributional Control at High Frequency. Second, we establish tight bounds on the distributional action gaps of h--dpendent centionconditioned return distributions- induced by applying a specific initial action for h units of time. We prove that these distributional action gaps not only collapse, as h tends to zero, but do so a slower rate than action-value gaps. On one hand, therefore, distributional Q-learning algorithms are susceptible to the same failures as Q- learning in continuous- time RL. On the other hand, however, remedies to these failures transliterated to distributional Q-learning algorithms are unlikely to succeed, because the means of these return distributions collapse faster than their other statistics.
distributions
Distributional Superiority. Third, we propose an axiomatic construction of a distributional analogue of the advantage. which we call the superiority. Leveraging our analysis of h-dependent action- conditioned returns and their distributional action gaps, we present frequency-scaled superiority distribution that enables greedy action selection at any fixed decision frequency.
Distributional Action Gap-Preserving Algorithm. Fourth, we propose an algorithm that learns the superiority distribution from data. Empirically, we demonstrate that our algorithm maintains the ability to perform policy optimization at high frequencies more reliably than existing methods.
2 Setting
Notation: Spaces will either be subsets of Euclidean space or discrete. Measurability, in the former case, will with respect to the Borel sigma algebra; the latter case, it will be with respect to the power set. The set of probability measures over space Y will be denoted by P(Y). Functions on spaces are assumed to measurable. For f Y - Z and μ E (Y),, the push forward of j o f-1 For random variable X, defined
through/by f. f# E (Z), is defined by f# implicitly on some probability space (9,, F, P). we write law(X) (X)= ;= n P to denote the law of X; the notation X llaw Y is shorthand for law(X) law(Y). For any μ P (R), the quantile function of μr, Fy is defined by Fj- (T) == inf, (2) 2 r}. where (2) is the CDF of μ.
2.1 Continuous-Timm RL
Here we give a brief introduction to the technical aspects of continuous-tii RI., à la [25]. We provide additional exposition and references in Appendix A. For any reader looking to defer some of this technical introduction, we summarize the core objects of interest at the end of Scction 1.1.1.
2.1.1 MDPs
Continuous-time Markov Decision Processes (MDPs) are defined by three spaces and four measurable functions: time interval T := [0, 2'I with T' E (0, oc) or T := (0,T) with T oo, state space C R", action space A, a drift b T x X x A + R", a diffusion reward r T x X + R, and a terminal reward f X + R.² The pair (b, a) govern the environment's dynamics by family of SDEs parameterized by A,
T x X x A + Ra
dx. = X;, dt + ot, XT. a) dB,
(2.1)
Here (Bt) is an n-dimensional Brownian motion. In turn, any solution to (2.1) collects the state paths of an agent that chooses action a at every time, regardless of the state they are
As is done in discrete-time RL, an agent might consider the induced Markov Reward Process (MRP) derived from policy T X > 3 (A)." The dynamics of policy-induced MRP (with policy x)
3 We work with iidion--ndepeddent rewards. This is the norm continuous-time DRI. We assume that (t) + rTB1 f, x) is measurable for every E C A and every (t, x) E T

---

are governed by the SDE
dx; 6" (t, dt + o² of (t,
(2.2)
Here, following [38, 15, 16], the policy-averaged coefficients bo and 0" are defined by
1/2 ( f at. [t,x)
6" (t, J b(t, b(t,z, (da t, and o (t,
(2.3)
Thus, solutions to (2.2) collect the paths of an agent following policy π.
class of policies central to study is those that fix action from some time for given persistence horizon h.
Definition 2.1. Given h >> O and a E A, policy is said to be peerssttt at time / E T T( 8, 5a for all (8,y) [1, + h) x X.
In particular, given a policy n, we will consider (h, a)-persistent modifications of x: for E T,
oh (,,,, = da
if E h))
(-(1118))
if $ (t,
These policies will help understand the influence of taking actions relative others well those taken by æ. We assume h small enough so that + h T.
In order to guarantee the nobal--n-timm existence and uniqueness of solutions to our SDEs (2.1) and (2.2), we make two sets of assumptions.
Assumption 2.2. The functions and a have linear growth and are Lipschitz in state, uniformly in time and action: a finite, positive constants C22.2 exists such that
sup b(t,x,a)) + sup lo(t, C2.2(1 + (x]) Wx E X; and
taa
taa
sup
2, b(t, y. sup f,a
(a,, ott,y, S C2.22= yl Vx,yeX. t.a
Assumption 2.3. The averaged coefficient functions bo and 6a Lipschitz in state, uniformly in time: a finite, positive constant C2.3 exists such that
sup 10 (t,z) (t. 8) + sup la (oa (t, ot (t, v)1 S C2.3/ yl Vz,yeX.
These assumptions are standard in the analysis of continuous-time RL, optimal control, and SDEs [12, 26, 38, 16, 42]. Since consequence of Assumption 2.2. The coefficients brt and o satisfy the conditions of Assumption 2.3 provided and satisfy some (also standard) additional regularity conditions and satisfies some regularity conditions. These technical details are discussed in Appendix A.2.
function of state, we note that Assumption 2.3 is not a direct
In summary, in continuous-tims RL, there are three stochastic processes of interest: (X.) with E {a, ग, π a all beginning at some time These processes collect the state paths an agent in one of three scenarios: 1. choosing action at every state and time; 2. following a policy n, 3. choosing a at every state and time for the first h units of time and following thereafter.
2.1.2 Value Functions and their Distributions
Given policy-induced state process (XT)s2e the (discounted) random return (t, x) earned by m starting from state X time E T is defined [41] by
Ga (t, 1. 70'tr(s, + with X²
(2.4)
where f when = [0, oo). We distinguish returns earned by (h, peersisetn modifications of policies. We call these ha-dependent action-comm conditiondd returns and denote them by 2 (t, x, a). Given æ, they are defined by
(t, x, a) = 1: 208 10-18,,55lla x.ln ds + TTx=f(Xx with X = I. (2.5)
3

---

(t, V* : E[G (t) or the h--ependent cartonnvvalte function Q7 (t, 2, a) := [t x, a)]. As distributional approaches in RL estimate the laws of returns, following [41], we define
Value-based approaches in RL estimate either the value function V
17 xt, (t (t) and
C (1, == law (Z (1, a)).
It is important to note that only the laws of random returns (and not their representations as random variables) are observable and modeled in practice.
2.2
Q- Q-Learning in Continuous Time
The failure of action-value- based RL in continuous-time stems from the collapse of action values at given state to the value of that state. Precisely, Tallec et al. and Jia and Zhou established that
QT, x. V (F
(HT x. a) (log V) (VVV,((() (t, x)) + oh))
(2.6)
where HD E R is independent of h (see [34] and [16] respectively). In a discrete action space, given state and time I, concise way capture the asymptotie information of (2.6) is by considering the action gap [11. 5] of the associated h-dependent action values
gap 2, = min Q7% (t, x. a1) (t, x, x2)2 a1702
Anticipating (2.6), which implies that gap(Q7, t. 2) Qh), Baird proposed AU wherein he esti- mated the rescaled advantage function AT, in place of Q7:
(T, x. a) Q7 2, V (t,
v(t,z, T
(2.7)
h
Note that gap(A r) 0()) Tallec al. [34] and Jia and Zhou [16], following Baird, also estimated A to ameliorate Q-learning in continuous time.
3 The Distributional Action Gap
In this section, we define a distributional notion of action gap; we prove that h-dependent action- conditioned return distributions collapse their underlying policy's return distribution as h vanishes; and we quantify the rate of collapse of these return distributions.
Definition 3.1. Consider an MDP with discrete action space and let p T x X A (R). for a metric The action gap of yu at state x and time is given by
distgapa(u, x,)
min d(p(t, 2, a1), x(t, a1f42
While IR has canonical metric, induced by |-|, the space (R) does not. So a choice must be made. and some metrics are unsuitable. For example, in deterministic MDPs with deterministic policies, return distributions are identified by expected returns: (t, x a) is the delta Q (t, x, a). for all (f, a) E T x X x A. Thus, distgap, f,z) should vanish as h decreases to zero if gap (Q%, f, x) vanishes decreases to zero. With the total variation metric d TV, for instance, this case, making TV unsuitable. Indeed, suppose we have a deterministic MDP with A = {a, a2 and such that C7 (1, xr (1) = 6U. and G7 (1, 22, a2) do, for some state æ and time i (see, e.g., [34]). Then h.
the
distgaprv (47, (, = 1, for all h > 0, yet (
The W, p distances from the theory of Optimal Transportation (see [36]), however, suitable. They are defined via couplings of distributions.
Definition 3.2. Let μ.v (R)) A k (R²) a coupling of μ and v if its first and second marginals are μ and respectively. We denote the set of these couplings by E E(u, v).
Definition 3.3. Let u.v (RR) and p E (0). The W, distance between i and is
1/p
(111=- w| k(dzdw)
Wo(n,)) == inf kE(E().))
(3.1)
$ In [38] and [16]. the authors established V and QT, as continuous- time RL analogues of the classic value and action-value functions. We note. however, that this done without explicitly defining GF and 2T, 6 The function H² is the Hamiltonian of the classic stochastic optimal control problem defined by our MDP.
7 Here assume that μ and v have finite absolute pth moments.

---

Any coupling attaining the infimum in (3.1) is called a p-ootiiall coupling. Henceforth, we write distgap, when considering Wp action gaps. j and v are deltas Q7 (t, x,aa) a1) and (t, , 2, 02) respectively, then the right-hand side of (3.1) is equal to Q zz @1) Q% (t, x az)l. Hence, in deterministic MDPs with deterministic policies, Wp action gaps of SK are identical to action gaps of Q7 making the W, distances suitable in the above sense. non-deterministie MDPs, the relationship between distgap, t, x²) and gap(Q t, x:) is opaque.
The following results study the W, action gap of CK as a function of h, lending some color to the relationship between distgap, (, and g,, These results all hold under Assumptions 2.2 and 2.3. Henceforth, we suppress mention of these assumptions; we do not restate them explicitly. First, we observe that Wp action gaps of 57 are bounded from below by action gaps of Q7.
Proposition 3.4. For all (t E T X X, we have that distgap, < t, x) 2 gap(Q7, f,
For a proof of this statement and any other made in this work, see Appendix B. Our next result establishes that W action gaps of 57 like action gaps of Q%. vanishes for large class of MDPs.
(t, (t) = 0, for all (t,z,a) Theorem 3.5. If and f are bounded, then lima W, (SR (t, (,, n7
TxXx x A; hence, lim distgap, (t),
While Theorem 3.5 shows that the Wp distance between ST x, a) and 1f" (t. x) (and the Wp action gap of S at (t, T x X) does indeed vanish as h decreases, it does not identify the rate at which it does so. Our next two theorems establish this rate.
Theorem 3.6. MDPs and policies exist in and under which, for all (t, a) E T x x A, we have that Wp WSE (f, 2. a), n7 (t, z)) 2 h2 and distgap, (ST, t, 2 h/2
Finally, we prove that for large class of MDPs (different from but overlapping with the class of MDPs captured in Theorem 3.5), the lower bound found in Theorem 3.6 is an upper bound.
Theorem 3.7. If r is Lipschitz in state, uniformly in time, f is Lipschitz, and T < 0o, then 2242,fo for all (t, E X A; hence, distgap, t, & hVv
Wt,S a), n7 (t, & h²²
Theorems 3.6 and 3.7 demonstrate that the Wp distance between (t, a) and 17 (t, and the distance between Q% (t, x. a) and (t, are of different orders in terms of h. Thus, we see that distgap, (C,,, and gap(Q?, x,) in stochastic MDPs are fundamentally different.
Distributional Superiority
In this section, we introduce a probabilistic generalization of the advantage. We define this random variable-whhe which we call the superiority and denote by 5% via pair of axioms.
A natural construction of the superiority at (1, a) is given by Z (1, x, a) G (*,,,, x).. The law of this difference, however, depends on the joint law of (2g (1, 2r, a), G" (I,z)), which is unobservable in practice and ill-defined (cf. Section 2.1.2), Yet, the set of all possible laws of this difference is casily characterized; is the set of coupled difference representations of CT (t, 2,a) a) and 17" (t, x).
Definition 4.1. Let p. V (RR) A coupled difference representation (CDR) v6 E ( of and takes the form 16 = Ag where & E E(u, u) and A R²² + IR is given by A(z, w) == w. The set of all coupled difference representations of μ and v will be denoted by 2 ((,,)) v).
Our first axiom places the superiority's 's law in this set, 9 (t, x, n), (t,
Axiom The law of S (t, x, a) a coupled difference representation of CT. (t, x, a) and r7" (t, x)..
Our second axiom encodes a type of consistency for deterministic policy behavior.
Axiom 2. ST. (t, x, a) is deterministic whenever π (h, aad---esistem at time t.
To see how Axiom 2 encodes notion of deterministic consistency, first consider its discrete-time analogue: the superiority at (z, a) for a policy π deterministic if I at z deterministically chooses a. In this situation, our a-following agent makes the same choices as a r-following agent-botl take action a in state æ initially and then follow æ thereafter-, and we posit that the superiority should not be random. The continuous-time analogue of the situation just described occurs precisely when policy π is (h, a) perrsssstt at starting time l. Given a starting time and state x,, an agent that chooses action between f and t+ h and then follows is following the (h, a) perssistent modification of at
5

---

time t. By definition, they make the same choices as a T--following agent when : is (h, a) peersistent starting at time Axiom 2 stipulates that, in this case, S7 (t, I, a) should be deterministic.
By construction, if v% (4,2, a) E (CE (4, x a), 7" x)), then its mean is (i, xt, a) (*
(see Appendix B.2 for a proof of this claim). Axiom 2 then says that any determining coupling (t, (t, æ, a) when π is (h, a) -)-pessistent at time must be such that 4 (t, (t, a) = ôg. In particular, Axiom 2 nontrivially restricts (S7 (t, I, a), (t,
Example 4.2. Let 2t, == Agt (t, for 1o x. x. = Sh @ (t, If is (h, a) a)-persistent time then Var( (1, 2Var(1 ι 's return is deterministic. Hence, 5% (1,2, a) may be nontrivial even when conditioning on a reflects the policy's behavior exactly. We posit, via Axiom that this should be prohibited.
This variance is 0 only when
In fact, Axiom 2 determines a single coupling, if we want a consistent choice across all time-state- action triplets and all MDPs.
Theorem 4.3. Let K E 6 E(u, p)) for some E (R)) The push-forvard of K by A the delta at zero, Ayr do. if and only if K is Wp-optimal coupling, for some p E (o). Moreover, there is only one such coupling. It is given by Ky == (id, id) # equivalently, Kp == (F 1F)')+ (0, 1). Here U(00 1) is the uniform distribution on [0, 1].
The second definition of Ki corresponds, more generally, to the Wp-optimal coupling, for all p 2 of μ and given by Ky.,v (PT- Fr # ((,0, 1). As A AgKp.' 's quanitle function is FT- Fy we have in hand everything we need to define the superiority distribution (via its quantile function).
Definition 4.4. The superiority distribution 45% at (t,x, E T
X A
vo 2t, := (FFglt...)) FF)((.)) 2(0 1).
As eig (t, a) has the smallest possible central absolute pth moments among all CDRs of G (t, z, a) and 17 (t, z). heuristically, it captures more of the individual features both return distributions than other such CDRs (like o (t, 2,, a) in Example 4.2). We illustrate this by example in Figure 4.1.
1.0
Returns
CDRs (1,,, a) (f)
0.5
n" (f,x)
(g(t,
4
3
-2
-1
0
1
2
3
4
4-4
4
-3
-2
-1 0
2
3
4
Figure 4.1: PDFs of Return Distributions and Two Candidate CDRs.
4.1
The Rescaled Superiority Distribution
From Section 3, we know that the Wp distance between (1, xr, a) and (t, for every p 2 1, vanishes as h vanishes. Moreover, we know the rate at which this distance disappears. As a result, by construction, the central absolute pth moments of vे (t, æ, a) collapse as collapses, and, for a large class of MDPs, we understand the rate at which these moments collapse. More generally and precisely, we consider the neeqque q-rescaled superiority distribution defined by
Vh. = (h~9 i
we see that Theorems 3.6 and 3.7 translate to the follow statements on W, action gaps of Whagi
Theorem 4.5. MDPs and policies exist satisfying Assumptions 2.2 and 2.3 in and under which, for all (t,x) E x X. we have that distgap,, (v (1 t, x) 2 h/229
Theorem 4.6. Under Assumptions 2.2 and 2.3, ifr Lipschitz in state, uniformly in time, f is Lipschitz, and T < oc, then distgap, (v 1,x) S h //-2,, for all (1, T X.
These two theorems tell us how to preserves the Wp action gaps of g-rescaled superiority distributions (as a function of ht). They identify q 1/2. For < 1/2, W action gaps vanish as h vanishes. Whereas for > 3/2, Wo action gaps blow up as h vanishes. These behaviors are undesirable. When
Here C.. (t, 2,, a) vy" 2);; the mean of A 1 (t, a) is

---

< 1/2, the influence of an action an agent's 's superiority becomes indistinguishable from any other action. For q > 1/2, ever larger sample sizes are need to obtain any statistical estimate of an agent's superiority with the same level of accuracy. These scenarios are untenable.
Another consideration regarding rescalings of eig. is whether they upend rankings of actions deter- mined by some given measure of utility. This would be counterproductive. In DRL, agents often use distortion risk measures [1] to rank actions ([10, 22, 4, 17]).
Definition 4.7. Given 3 E ( I]}, the distortion risk measure PB (RR) + R is defined by Pa(p)= = {B, Fr on μ E (R), its value is given by the integral of Fy" with respect to B.
A family of Ps is the a-conditional value-at-risk measures (a-CVaR) [29], where Sa U(O, a:) for a E (0, 1]; a = 1 is the expected-value utility measure. Crucially, 1h preserves p3-valued utility.
Theorem 4.8. Let Pg be a distortion risk measure, q 2 O, and h > If Ps (1, oc, then arg EA P (t, a)) arg m Pa(CT (t, a)).
In turn, the 1/2-rescaled superiority distribution is not only Wp action gap preserving but matches Ç in its greedy choice of action as measured by a distortion risk measure.
4.2 Algorithmic Considerations
We now turn to building DRL algorithms based on theory. Our algorithms leverage the quantile TD-learning framework [10] to learn Pa-greedy policies, for a given distortion risk measure pa, just DAU [34] leverages the Q-learning framework learn greedy policies. Full pseudocode and implementation details are given Appendix C.
At the heart of our algorithms is an equality of quantile functions, which holds by construction,
F-1 (t.a,)) = = (t,,)) + F 0% (t.r,o)
(4.1)
Indeed, given 1) and hing as models of 17" and vh% respectively, equation (4. 1) justifies the application of quantile TD-Leaming to Sn. as a model for ST, defined via the quantile function
F-1 (t,,,a) == F + h (t,,aa)
(4.2)
That said, we cannot realize quantile TD-Lcarning without defining predictions and bootstrap targets in terms of m-quantile representations" [10, 4] of Sn, via those of 17 and "hn."""
While we may freely parameterize the m-quantile representation of with neural network (with interface) 0 x X RR' have to be careful when parameterizing the m-quantile representation of Wha Given a neural network Φ T x X
x A + R", we set
F-1 (nnota == @tt, x. a)
@ ar a')
with a" E Earg max P3 (s(ott, a)).
(4.3)
aEA
This ensures we identify pa-greedy policy; is 0 at the Pa-greedy action at (cf. [34, Eq. 27J).
With appropriate parameterized m-quantile representations of 1j and Whag in hand, we derive our predictions and bootstrap targets. By (4.2), recalling 0 and (4.3), we compute our predictions via
F-1 (t,,a) ((1,,) h2 (0(t,, 6(t, ax)))
(4.4)
By Wiltzer [40], as X. and x X observe that
-law h-' 7'r(t x ds + G X (hhi
law hr(t, + GH h, XP Yn.
where E| C||Y|p= o(h) 10 for all e N. So upon getting a sample statc/realization 2ith of X as in [30], we compute our bootstrap targets via
JF-1 (t = hr(t, x) + 200(t + h, x.+n).
(4.5)
An m--uuantle representation of a given distribution in (R) is vector in R whose ith component encodes the 1-1/2 quantile said distribution.
This holds if r is continuous, standard assumption in continuous cittinuous-timmeeee control (see, e.g., [25, 34, 16]).
7

---

In summary, the predictions (4.4) and the bootstrap targets (4.5) together characterize family of QR-DQN-based algorithms called DSUP(q), whose core update outlined in Algorithm
Algorithm (SUU(() Update
Require: 9 (rescale factor), (step size), D (replay buffer), ps (distortion risk measure) Require: 0: T x X + In (m-quuntile approximator of ) Φb T x X A - R. (see (4.3))
Sample (f, Fe, de, Tt, Itthh done+th) from D
a' + arg max, Pa m 2n=1 )s(t,ara))))
Risk-sensitive greedy action
F (Ø, 0) +- ((1, + (6(1,, 0z) ) a())
Di Prediction (4.4)
JF." + + +
done h, (1411 ( done (z(th) f(ó. () - QuantileHuber((? (0, (T, (
Target (4.5)
See [10, Eq. (10)
+ Vol(o, and 0 + aVe(()
Gradient updates
One theoretical drawback of DSUP(2) for mean-return control is that the mean of the q-rescaled superiority distribution is (1) only when 1, by (2.6) and (2.7). Thus, we propose modeling AW simultaneously. This yields novel form of two-timescale approach to value-based RL (see, e.g.. [8]. In particular, we estimate hig defined by
F (t.,0t.
F (t..a)) + +1 - -1
We call 0%.yg the advantage-shipe q-rescaled superiority. Note that its mean is Ag. which is 0(1). To realize this, approximate AT, using DAU and employ parameter sharing between the approximators of AØ and Whag We call this family of algorithms DAU+DSUP((). We note that AØ, is used only for increasing action gaps; it does not change the training loss for and Whag"
Simulations
The empirical work herein two-fold in nature: illustrative and comparative. First, we simulate an example that illustrates Theorems 3.6/4.5 and Theorem 3.7/4.6 and their consequences. Second, in an iitiion environment, we compare the performance of 6े, -bsed agent(s) against QR-DQN [10] and DAU [34] in the risk-neutral setting and against QR-DQN in risk-sensitive setting.
5.1 The Rescaled Superiority Distribution Revisited
Consider an MDP with time horizon 10, a two element action space, 0 and when action is executed, the system follows I-dimensional Brownian dynamics with constant drift of 10, and otherwise, the state is fixed-, a reward that equals the agent's signed distance to O, and a trivial terminal reward. We estimate four distributions at (t, a) (0, 0, 1) for the policy that always selects O. Figure 5.1 shows these estimated distributions for sample of frequencies (kHz), w 1/h.
Ch 1/
16.
0
Kht/a
hu/a
66k
44kHz
33kH2
26kH2
22kH2
10KHZ
16kHz
14kHz
13kHz
0.3
0.0
0.3
10000
0
10000
-40
40
100
150
Figure 5.1: Monte-Carlo estimates of Vhggg for q = 0, 1, 1/2, and oF 1/ function of w 1/h.
First (from the left), we see that vij, collapses to 6g. as h tends to 0. Thus, accurate action ranking distributional or otherwise becomes impossible in the vanishing h limit. Second, we see that rescaling by hi produces distributions with O(1) mean but infinite non-mean statistics in the vanishing / limit. Here the (11) means are imperceptible face of the large variances. So while this rescaling permits ranking actions by action values, it does at the expense of producing high vigiincrencce distributions.

---

Third, we see that rescaling by h yields distributions with ((1) non menn statistics but vanishingly small means, ))) Hence, this rescaling permits ranking actions by non-mean statistics, even action values again becomes indistinguishable in the vanishing limit. That said, the vanishing rate of the means here is slower than when no rescaling is considered, O(h). Fourth, we see that rescaling by h 122 and then shifting it by (1 - A produces distributions with O(1) mean and non-mean statistics. In turn, this two-timescale approach permits ranking actions by either action values or non-mean statistics (but not both by Theorem 4.8). However, the mean estimates here are inaccurate and imprecise-ratien than uniformly being 100, they oscillate substantially.
In risk-neutral control, we are left with number of questions. What effect do the high variance distributions in DAU/DSUP(1) have on performance? What effect do the O(h*/2) means have on the performance D/U? What effect does the instability of the mean estimates DAU+DSUP(//2) have on performance? In Section 5.2, we begin to answer these questions and others by testing our superiority- algorithms against appropriate benchmarks in an option-trading environment.
5.2 High-Frequency Option Trading
The tiriinn environment in which we run our comparative experiments is commonly used benchmark (see, e.g., [22, 17J). We use an Euler-Maruyama, discretization scheme [23] at high resolution to simulate high-frequency trading. Returns are averaged over 10 seeds and 10 different dynamics models (corresponding data from different stocks). Additionally, following [22], we use disjoint datasets to estimate the dynamics parameters for simulation during training and evaluation.
First, we consider the risk-neutral setting. Here we compare QR-DQN, DAU, and three algorithms based the g-rescaled superiority distribution with g = 1, 1/2: DSUP(1), DAU+DSUP(/2), and DSUP(1/2). Figure 5.2 summarizes their performance at a sample of frequencies (Hz).
Discounted Return
0.03
QR-DQN
0.02
DAU
DSUP(1)
0.01
DAU DSUP(1/))
0.00
(SUP(1/2)
1Hx
5Hs
10Hs
15Hr
20Hr
26Hs
30HH
36Hz
Decision Frequency
Figure 5.2:
algorithms on high-frequency srdiiig as a function of
We that DSUP(1/2) is not only the most consistent performer, but outperforms every competitor all but the two lowest frequencies. Even then, its performance very close to the best performer. We also that DAU+DSUP((/2)' preservation of both action gaps and W, action gaps does not lead to the strongest performance. In particular, its performance is inconsistent and sometimes poor. We believe this is because the tested frequencies are low enough that DSUP(1/2) maintains large enough action gaps to learn performant policies, but high enough that the variances of the distributions underlying AE cause estimation difficulty. Indeed, the three methods that estimate AE, (explicitly in DAU and DAU+DSUP(!/2) or implicitly in DSUP(1)) exhibit almost identical behavior.
Our results highlight a dichotomy existing (ours included) methods for value-based, high- figh-rreuency, risk-neutral control. They can either maintain ((1) expected return estimates or O(1) return variance estimates, but not both. We observe better performance estimating small means from O(1) variance distributions than in estimating ((1) means from receipricolly large variance distributions.
To qualitatively illustrate the appeal of the 1/2-rescaled superiority, Figure 5.3 presents examples of learned cetiionconditioned distributions used by DSUP(I/2) and QR-DQN agents make decisions.
Our code available https: :
attps://githus htt///gtttuub/te////////e wwt..//wwwwtt//aaaaeseeeeeeaeeee

---

In this environment, action 1 taken in the start state terminates the episode, yielding the smallest return, O, making this action inferior its alternative ac- tion, 1. We see that DSUP(1/2) infers this fact. QR-DQN, on the other hand, has difficulty distinguishing these ac- tions. This because the 1/2-rescaled superiority preserves Wp action gaps, while 5% does not.
Action
Action
Action
Action
-12
0.01
0.51
0.02
Figure 5.3: CDFs of Wh/y/ from DSUP( (left) and Sh from QR-DQN (right) at the start state at w 35Hz.
Second, we consider risk-sensitive set- ting. Here compare QR-DQN and DSUP(1/2) using a-CVaR for greedy action selection. We do this because Theorem 4.8 does not hold with hhgg" and preserving means is less critical in risk- sensitive control than is risk-neutral control. Figure 5.4 depicts our results = 35Hz (see Appendix D for results across range of w).
a--VaR(Retumi
QR-DQN DSUP(1/2)
0.01
0.01
0.23
0:34
0.45
0.56
0.67
0.78
0.89
&r
Figure 5.4: Risk-sensitive algorithms high-frequency option-trading 35Hz.
Again, that DSUP(1/2) is conclusively the best performer.
Related Work
Notions of action gap and ranking have long been of interest in RL. (see, e.g., [11]). Action gaps related sample complexity in RL-indeed, ietance-dependen sample complexity rates inversely proportional to the divergence between action-conditioned return distributions ([13, 19, 37J). Bellemare al. [5] argue for the consideration alternatives the Bellman operator that explicitly devalue suboptimal actions, and they show that Baird's AL [2] operator falls within this class of operators. On the other hand, Schaul al. [31] implicitly question Bellemare et al.'s position. They demonstrate that stochastic gradient updates in deep value- -ased RL algorithms induce frequent changes in relative action values, which turn mechanism for exploration.
The advantage function is commonplace in RL (see, e.g.. [39, 32, 27, 35, 24]). [24], Mésnard al. employ a distributional critic that is closely related to our (unscaled) distributional superiority. Their choice of critic stems from desire minimize variance. We note that the distributional superiority is a posteriori characterized as minimal variance coupled difference representation of action-conditioned return distributions and policy-induced return distributions.
Lastly, DRL in continuous-tinn time MDPs in its infancy. There are only three works to mention. Wiltzer et al. [40, 41] give characterization of return distributions for policy evaluation, and Halperin [14] studies algorithms control. That said, neither work considers distributional notions action gaps or advantages. Moreover, Halperin does not consider any of the challenges of estimating the influence actions high decision frequency settings.
Conclusion
We establish that DRL agents are sensitive to decision frequency through analysis and simulation. In experiments, DSUP(L/2) learns cell-peffommnng policies across range high decision frequencies, unlike prior approaches. DSUP(1) and DAU-DSUP((/) are less robust. Given our analysis, the performance of DSUP(1) expected. Building an alternate algorithm DAU+DSUP(!2)) that both tailored to risk nistralutral control and robust to f is an important avenue for future work.
10

---

Acknowledgments and Disclosure of Funding
The authors are very grateful Yunhao Tang for fruitful correspondence ahout distributional ana- logues to the advantage. Additionally, thank Mark Rowland, Jesse Farebrother, Tyler Kastner, Pierluca D'Oro, Nate Rahn, and Arnav Jain for helpful discussions. HW was supported by the Fonds de Recherche du Québec and the National Sciences and Engineering Research Council of Canada (NSERC). MGB was supported by the Canada CIFAR AI Chair program and NSERC. This work supported in part by DARPA HR001 23-9-0050 to PS. YJ was supported by in part by NSF Grant 2243869. This research was enabled in part by support provided by Calcul Québec, the Digital Research Alliance of Canada (alliancecan. ca), and the compute resources provided by Mila (mila. quebec).
References
[1] Carlo Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance, 26(7):1005-1518, July 2002.
[2] L.. Baird. Reinforcement Learning Through Gradient Descent. PhD thesis, May 1999.
[3] Mare G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):777888 82, December 2020.
[4] Mare Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. The MIT Press, May 2023.
[5] Mare G Bellemare, Georg Ostrovski, Arthur Guez, Philip Thomas, and Rémi Munos. Increas- ing the action gap: New operators for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016
[6] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin. George Necula, Adam Paszke, Jake VanderPlas, Skye Mandderman-Miinge and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[7] Gerard Brunick and Steven Shreve. Mimicking an Itô process by a solution of a stochastic differential equation. The Annals of Applied Probability, 1384):1584 August 2013.
[8] Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for non- linear value function approximation. In International Conference on Learning Representations, 2018.
[9] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, pages 1096-1105. PMLR, 2018.
[10] Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional Reinforce- ment Learning with Quantile Regression. In AAAI, 2017.
[11] mairuud Farahmand. Action-gap phenomenon in reinforcement learning. Advances in Neural Information Processing Systems, 24, 2011.
[12] Wendell H Fleming and Halil Mete Soner. Controlled Markov processes and viscosity solutions, volume 25. Springer Science & Business Media, 2006.
[13] Todd L. Graves and Tze Leung Lai. Asymptotically Efficient Adaptive Choice of Control Laws in Controlled Markov Chains. SIAM Journal on Control and Optimization, 35(3):715-743, May 1997.
[14] Igor Halperin. Distributional offline continuous-time time reinforcement learning with neural physics- informed pdes (sciphy rl for doctr-l). Neural Computing and Applications, 36(9):4643-466 2024.
[15] Yanwei Ji and Xun Yu Zhou. Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms. Journal of Machine Learning Research, 23(275):1-50, 2022.

---

[16] Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. Journal of Machine Learning Research, 24(161):1-61, 2023.
[17] Tyler Kastner, Murat A Erdogdu, and masr-massoud Farahmand. Distributional Model Equiva- Ience for Risk-Sensitive Reinforcement Learning. In Advances Neural Information Processing Systems (NeurIPS), 2023.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2014.
[19] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[20] Leemon C. Baird. Advantage Updating- Technical report, Defense Technical Information Center, Fort Belvoir, VA, November 1993.
[21] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
[22] Shiau Hong Lim and Ilyas Malik. Distributional Reinforcement Learning for Risk-Sensitive Policies. In NeurIPS, October 2022.
[23] Gisiro Maruyama. Continuous markov processes and stochastic equations. Rendiconti del Circolo Matematico di Palermo, 4:48-90 1955.
[24] Thomas Mesnard, Wenqi Chen, Alaa Saade, Yunhao Tang, Mark Rowland, Theophane Weber, Clare Lyle, Audrunas Gruslys, Michal Valko, Will Dabney, et al. Quantile credit assignment. In International Conference on Machine Learning, pages 24517-24531. PMLR, 2023.
[25] Rémi Munos and Paul Bourgine. Reinforcement Learning for Continuous Stochastic Control Problems. In Advances Neural Information Processing Systems (NeurIPS), 1997.
[26] Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
[27] Hsiao-Ru Pan, Nico Gürtler, Alexander Neitz, and Bernhard Schölkopf. Direct Advantage Estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[28] Simon Ramstedt and Christopher Pal. Real-time reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
[29] R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. Journal of Banking Finance, 26(7): 1471, 2002.
[30] Mark Rowland, Rémi Munos, Mohammad Gheshlaghi Azar, Yunhao Tang, Georg Ostrovski, Anna Harutyunyan, Karl Tuyls, Mare G. Bellemare, and Will Dabney, An Analysis of Quantile Temporal-Difffrrnce Diffference Learning. Journal of Machine Learning Research (JMLR), 25:1-47, 2023,
[31] Tom Schaul, André Barreto. John Quan, and Georg Ostrovski. The phenomenon of policy churn. Advances in Neural Information Processing Systems, 35:2537-2549, 2022.
[32] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. CoRR, abs/1707 006347, 2017.
[33] Daniel W. Stroock and S. R. Srinivasa Varadhan. Multidimensional Diffusion Processes. Springer, 2006.
[34] Corentin Tallec, Léonard Blier, and Yann Ollivier. Making Deep Q-learning methods robust to time discretization. In International Conference Machine Learning (ICML), 2019.
[35] Yunhao Tang, Rémi Munos, Mark Rowland, and Michal Valko. VA-learning more efficient alternative to Q-learning. In International Conference on Machine Learning (ICML), 2023,
[36] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
12

---

[37] Andrew J. Wagenmaker and Dylan J. Foster. Instance- Pestance-Oplimliiy in Interactive Decision Making: Toward Non-Asymptotic Theory. Proceedings of Thirty Sixth Conference on Learning Theory, pages 1322-1472. PMLR, July 2023.
[38] Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 21(198):1-34, 2020.
[39] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Dueling Network Architectures for Deep Reinforcement Learning. In International Conference on Machine Learning (ICML), 2016.
[40] Harley Wiltzer. On the Evolution of Return Distributions in Continuous- Reinforcement Learning. McGill University (Canada), 2021.
[41] Harley Wiltzer, David Meger, and Mare G. Bellemare. Distributional Hamitton-JccoiiBellle Equations for Continuous- Time Reinforcement Learning. In International Conference on Machine Learning (ICML), 2022.
[42] Hanyang Zhao, Wenpin Tang, and David Yao. Policy optimization for continuous reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
13

---

A Formalism of Continuous-Time RL Controlled Markov Processes
Expected-value RI. a data-driven approach to solving the (classic) optimal control problem: find an action (control) process (As)szr and an associated state process (then determined by the environment) (X,)a2t with X, = 2, for a given t 2 O, that maximize the expected return earned by following the state-action process (X... A2) In particular, RL agents search the space of state-action processes via policies т : state-action processes. Indeed, (X,,, Ag)e2( is the state-action process of an agent following if and only if, for each s 2 t. the set {- s, x) Ixx is the set of conditional probabilities of law (( with respect to law (Xa).
+ ((A). Policies prescribe the conditional probabilities of the laws of
Continuous-time RI a data-driven approach to stochastic optimal control. Whence, environmental dynamics are assumed to arise from an action- action- parameterized family of SDEs determined by drift b: T X x A R² and diffusion a T X A - TR. "x* 12 Thus, the goal of expected-value RL. (and stochastic optimal control) is to find an expected-return maximizing state-action process among state-action processes (X., 22) that satisfy
dX, b(s, X., A.) ds + o(s, X. AB, with X, x.
(A.1)
We note that the MDPs defined Section have equivalent formulations in terms of transition kernel, exactly as they formulated in discrete-time RL. We refer the reader to [33] for an in-depth discussion regarding this fact.
A.1 Justification Random Returns
Given the above formalism, the "true" distribution of returns of agent following policy is the law of
hr ( ( 'rs, Xs) ds +77==((x))
where (x.AA))s is a state-action process associated to a and solves (A.1). That said, by the definition of
b5,, = E(b(s, X,, and O"OO")
E(oo'
TeooT(s.)) (s, X,, A,)
where 5r and TT exactly the coefficients defined in (2.3). Hence, by [7]. provided that ba and TT are regular enough to guarantee that (2.2) is well-posed in law, the processes X = (X7)22 and X = (X,)s2t are equal in law. Here (XT)szz satisfies (2.2) with X; = æ. Consequently,
G (3, x) (1, llww Jr
X 'r X.) do +
This, formally, justifies justiiiy G and Zr , as defined in (2.4) and (2.5).
A.2 On Assumption 2.3
Here provide some conditions under which Assumption 2.3 is established. These conditions presented as additional assumptions. Assumptions A.1 and A.2 are common in stochastic control theory (see, e.g., [12]) and SDE theory in general (see, e.g., [26]). Assumption A.3 is ubiquitous in the contimuous-time RL literature [i5, 16, 42]. Notably, these conditions together guarantee the existence of transition probabilities for molicy-induced state processes arising from (2.2).
Assumption A.1. The coefficients b and a are uniformly bounded: a finite, positive constant CA.1 exists such that
sup |b(t, a) + sup |a((, 21 S CA. f,z,a
t,z,a
Assumption A.2. The matrix To is uniformly elliptic: a positive, finite constant XA.2 exists such that
inf inf T "oo" (t,I, N221.
Without loss of generality, as the the laws of our stochastic processes are the objects of interest, we may assume that our diffuusion matrix is square. Indeed, for any SDE with n x m-dimensiional diffusion a, the matrix (ar T) which uniquely defined by ao and not a the diffusion matrix that determines the law of the solution that SDE.
The assumptions under which work guarantee that X -law X.
14

---

A consequence of Assumption A.2 is
inf inf v. "=""((,z00 2 AA2l
(A.2)
t 1o|=1
In other words, o= is also uniformly elliptic. Assumption A.3. A finite, positive constant CA.3 exists for which sup (t) & CA.3/x - yl Ve,y
where TV is the total variation metric on 3 (A).
Observe if satisfies Assumption A.3, then ala,a, also satisfies Assumption A.3. Indeed, x), n ( (1,9)) & (V(x x (-11,,))
(V
since
sup TV (a (8 ग a v))
and TT (- x) x(- s, x) for all 8 T [t, h).
Proposition A.4. If Assumptions 2.2, A.I, and A.2 hold and π satisfies Assumption A.3, then Assumption 2.3 holds.
Proof. Observe that
- - (c)))) s 1 - 6(1, a)) r(da (,,))
+ 11 b(t, y. x(da t, - f b(t, us a) (ddalt,))
S (C2.2 +
y).
by Assumptions 2.2, A.1, and A.3. Here we have also used Kantorovich duality to computed TV and invoke Assumption A.3.
Let be an eigenvalue of α (1) oth (t, v) with with unit eigenvector v. Observe that
v (o o² (t,1) (o (t) o" (t,u))o" (t, + o² (t,u)(o" (1, - o² (t, 1),
Au (0 (t, ot x))
Hence, by (A.2),
| In o y)
o² o
I 2A
(o (t,)))))l.
v (T + o.h (t, æ))b
In turn, by Assumptions 2.2, A.1, and A.3, as done to prove that 6t was Lipschitz above,
| - Aa.2 (O2.2 + CALCAR)) w/-
Assumption 2.3 follows, since À was arbitrary eigenvalue and all norms on finite dimensional spaces are equivalent.
We conclude this section with one final fact: under Assumption 2.2, the policy-averaged coefficient (2.3) have linear growth. Indeed,
(1, / a)),, < C2. (x]]
and
1 (t,x)) < f (1((t,2,)) x $ C2 + 1x])2.
15

---

A.3 Action-Independen Rewards
In this paper, we assume that the rewards do not depend on actions. This is theoretical limitation of not just our work, but continuous-ting DRL in general (see [41, 14]). In the following sections, we discuss the nature of this theoretical limitation. However, many MDPs have action- vetion-independeed reward functions. For example, MDPs encoding goal-reaching problems, tracking problems, and commodity-tradinn problems all have action-independent rewards.
A.3.1 Continuous- Time, Expected Return
In continuous- time, expected-vas RL, when the reward function r depends on actions, the standard approach to analysis involves considering the averaged reward function nM T x X - IR given by
r" (t, (t, := J r(t, 2, a) rdaa t.).
In continuous-time RL specifically, the averaged reward r is justified exactly as the coefficients br and ṭ" are justified. However, the "true" return distribution is not equal to the law of
L 7-'' X,) ds +1b++-'''(Xp
(A.3)
To see this, it suffices to consider an MDP with a single state x. In this case, the expression in (A.3) is deterministic. However, if is nondeterministic and is dependent on actions, then the "true" return distribution is nondeterministic. Hence, the law of the expression in (A.3) cannot be the return distribution associated ...
return
A.3.2 Discrete-Time, Random Return
In discrete-time RL, the distribution of returns given an action-dependent reward analyzed through the state-action process induced by a policy. This processes is defined by extending the action- parameterized family of transition probability kernels on X, which define the dynamics of given MDP, to single transition probability kernel on X x A. In the time timehhomggeneou setting, for instance, with transition kernels (P(dyl x, a)) eEA, this amounts constructing
p" (dydb I, a, == (db y) @ P(dy) 2, a),
provided that map y (EE yy) measurable for all y E X. In continuous-time environments, such a constructing has yet to be discovered.
We note that trying to analogously extend the action- action-paramenn parameterized family of transition semigroups on X, which define the dynamics transition semigroup X x A by defining
given time- homoooooggenuus MDP in continuous time, to a single
P. (dydb) æ, a) := π (db)) y) @ Payyy a.))).
where Pt (dyy 2) is transition semigroup, may fail to satisfy the Chapman-Kolmogong identity. Indeed, suppose A has two elements and = R. Let (da xx) be the uniform measure on for all E X. If P. (dy æ, as) = (dy) and (:yy x, ag) = (2m)-1// exp(-ly x 2 2/2t) dy, then Chapman Kolmogorov identity fails, for example, any tuple (s, (s,t,I, as. E' x F) where E C Xis open, x + t + & & E, and (F) + 0. On one hand,
Pm ( x, as) π (F)84+++s(E)) =0
On the other hand,
f P. P. x Fl PT xr, x,)5)
x(F)
+ + > 0.
2
So
(E (E +
v. (dyda | as).
At present, the question of how to generate well-defined (even in law) action process in any continuous-time MDP framework given a stochastic policy is generally open. Of course, is deterministic, then the state-action process is (X,, ग (s, X Xz))szt- If b and are Lipschitz in state and action, uniformly in time, and и is Lipschitz state, uniformly time, then (A.1) with A, xx) is well-posed.
16

---

B Proofs
B.1 The Distributional Action Gap
In this section, we prove the statements made in Section 3.
Before proving any of the statements made in Section we recall identity that relates the W, distance between two probability measures and y and the absolute central pth moments of the differences of random variables distributed according to f and v/:
Wo(, = inf (X.Y)
[El YP! law(X) = Iu and law(Y) = =n..
(B.1)
This identity will be used a number of times, including in the proof of Section 3's first result, which we restate here for the readers convenience.
Proposition 3.4. For all (t,z)= E T X. we have that distgap, (SR t, t, t,,
Proof. Let (Z1, 7z) be any random vector with such that law (4) 5% (t. 0:) for i = 1,2. Then, by Jensen's inequality,
E| 11 - /2 2 ElIZ1 -2]l 2 B|Z Z2]1 (Q% - (2))))
Hence, since (71, 72) arbitrary, by (B.1):
Wp(CE (t, x. 01)). ST (t. x, 02)) 2 (T, QT x, 02)1.
Finally, taking the minimum over pairs of actions (a1, a2) such that a₁ * 02 concludes the proof.
Now move to the proofs of Theorems 3.5 and 3.7. We defer the proof of Theorem 3.6 until after the proof of Theorem 3.7 as the proofs of Theorems 3.5 and 3.7 are similar. For clarity's 's sake, first prove collection of lemmas.
Lemma B.1. Under Assumption 2.2, let (X4)s2) be the unique strong solution to (2.2) with X = P-a.s. Then, for all q 2 1 and all s
El|x-
x 2 S C,
|=|) (() t)a + 1)(s C
where C. = 424-1
Proof. Let Cg = 424-1, By Jensen's inequality and Itô's isometry, observe that
IX x/22 < Cs 1)24-1 S. 16(s', x, 24 ds' Cg(s 1)8-1 f. a))
+ Cg(s 1)20 1 S: 16(8', (9.', - b(s', a ds'
+ -C1( - 1) 1. 10(5 (1,5, - 008 a) 2 ds
< < 1)24 + (s 1)9) (1 |z|)
+ CC(((s 2)2 + (s t) 1) J.
[X - x: ds'.
Thus, the lemma follows after taking expectation and applying Gronwall's inequality.
Lemma B.2. Under Assumptions 2.2 and 2.3, let (X)) with E {,,, π T|n.a.)) be the unique strong
solution to (2.2) with X P-a.s. Then, for all < 1 h
X
E
21
C(1 + (>) 1)a + 1)(s ((=00((-))+
where C. some finite positive constant depending on q. Cz and C2.3-
17

---

Proof. Let Cg g2q-1 Note that Xa X². P-a.s. for all s' < h, by the definition of T at, and the uniqueness of strong solutions to (2.1). So, by Jensen's 's incquality and Itö's isometry, observe that
X Ia - & > - -1)20- + ((--t)=-)) ( (c22 + - x cs'
(C22 + (22))(1 + |x) )2 ds
J. C2
x C2 177 X3 24 ds'
Thus, after taking expectation, we deduce that
E X bl
xx Cgl(s - t)2g + (s - -]]) (1224 + +22.) max E[ x| 3'(.x.)]
+ (C22 + 21 + |z|)22
+ C,(((s - 2- ()) + (s - t)a 1 C |"
E E Shee 2
And so, the lemma follows after applying Lemma B.1 and Gronwall's inequality.
Lemma B.3. Under Assumptions 2.2 and 2.3, les (X:)s2) with E {I, x xla,,))) be the unique strong solution to (2.2) with X; P-a.s. Then, for all > t h,
El|x; heatt x S C(1 =0) (h ((s-
where C is some finite positive constant depending on 4 C2S and C2.3-
Proof. Let Cg = 32 Observe that
x = Xith* + [ X)) ds' + l
1th
1th
So. as x/1.,,o(- s' x(- ss (1.1s'.v0
for all (5',)) E T [t,t h) X. by Jensen's incquality and
Itô's isometry,
|x_ Thaae haat
X < Cy
Th. bth
X bth
20 + Cg
(s-1- ( 11)
s |x
+ (s - - h) C23
X3 29 ds'.
++h Thus, after taking expectation, applying Gronwall's inequality, and considering Lemma B.2, the lemma follows.
One consequence of Lemmas B.2 and B.3
E E|
x - as 4,0,
for s and for all p E [1, oo). And so, if f bounded, then f(X - f(X)) is bounded and converges to zero P-a.s. as h converges to zero. The dominated convergence theorem implies that
EX f(X7))
h O.
(B.2)
Similarly, we see that the functions 9h T [0, o0) defined by
gh (8) = E[|r(s, x: - r(s, XT)
are uniformly bounded (in h) and converge zero as h 0 every E Hence, by the dominated convergence theorem, again,
Ellr(s,
(n Xi (x, (( - as h40
(B.3)
where I'(ds) == - /Cr ds for Cr (n Tt- - 1)/ logo y. We now prove Theorem 3.5,
18

---

Theorem 3,5. I and f are bounded, then Jim (, = 0, for all 2,
T A; hence, lim distgap, (5% 2t
Proof. Observe that distgapp (Shigapp(Siikee 2,) min (l,, (1,,,)))
a1
I min (p (f, + min 22)) a13es
z02
S max w (t, 1f (t,,))
Thus, as claimed, suffices to show that
as 4 0, for all (1, ETN)xAXXXA A. By (B.1), 1), it suffices show that
WS (t, xz (t,
7 (,, GT (1, - 0 as h 4.
(B.4)
for E
Since (s,w) + (w) is measurable for E (T, by Jensen's inequality and Fubini's theorem, we see that
G",,,) Cr T.t.
X -(t
(B.5)
+ 2 (0- E[((( ( (iit) f(x7)))
In turn, (B.4) follows from (B.2) and (B.3).
If < and h 1, the incqualities in the statements of Lemmas B.2 and B.3 yield the following inequality: for all E o))
E x
x3I) S P/. Vs T,
(B.6)
for some finite positive constant C(R.) depending on p. |x), 1, T, C2.2, and C2.3. With this inequality in hand, we now restate and prove Theorem 3.7.
Theorem 3.7. If r is Lipschitz in stale, uniformly in time, I is Lipschitz, and T < Xi, then Wo xt, xx, a), 17" (t, h22 for for all (t,z,a) E T x x A; hence, distgap, t, x) h/2
Proof. Arguing as in the proof of Theorem 3.5, we see that
E| El - G (1,x)))) < 2 CP Tei
I + 2D n ((T-1)
with
X.lhiitt) -(s and 11: BII f(x)))
This is (B.5). turn, by (B.6) and that r Lipschitz in space, uniformly in time and f is Lipschitz, deduce that
I I1 < (C. + CF)
as desired, where C, and Cy are the Lipschitz constants of and f respectively.
Recall r: x + R is Lipschitz in state, uniformly in time if finite positive constant C, exists such that
sup |r(t, r(t, w)1 Cylar vl Vz X.
As promised, we now prove Theorem 3.6.
Theorem 3.6. MDPs and policies exist in and under which, for all (t, a) E T x A. we have (TT t, x) 2 1/2
that W, (C (t, a), 17 (t,xt) 2, f//2 and (Ss
19

---

Proof. Let X = R and A = {0, 1}; set b = 0 and o = (a=1); for all (s.y) y) E T x v)) y: and set f = 0. In words, our action space has two elements, and when action 1 is executed, the system follows Brownian dynamics, and otherwise, the state is fixed. Now consider the policy which always selects the action O: .(- s, y) = og, for all (8, TTXX
Case 1: and T' < co. Observe that
/ h
2% (t, 2, -) 23 (t, -,
B, ds + -t- -t- B,,
where (Bz)a20 is Brownian motion. Hence, (t, 27 of two zero mean Gaussian random variables with variances of 1//31//3 and d3 = (T tt- h respectively. And so, it is also Gaussian, and its variance is 0% of + 0 + 200 for some -1 c S 1. In particular, of 2 hP/22 Recall that central absolute pth moment of a Gaussian random variable is proportional to its standard deviation to the power p. for all p 2 1. Therefore, by (B.1), we deduce that distgap, (CT, f, r) 2 h /2, for h < 1, as desired.
is equal in law to the
Case 2: T E [0, oo] and E (0, 1). Observe that
T-t
//" hh > ds,
(1, 27 (1, xr 0) llw law N(h)
with N(h)
log7
for some Brownian motion (Bz).20. We claim that (h) is mean zero Gaussian random variable with variance of = h²/3. Then the concluding argument in the proof of Case I also concludes the proof of this case.
To prove our claim, first note that
E[N(M)] = S/" "'E ds = 0 and E[N(A)) 1/" 700
1/- y0th ElB, B,) dads'.
As
h
2h hh
h 1-/ 1/'' min(s, s'} dsds' < E[N(A)* < L0 fo minfs, s') dsds' S
/
2h
3
3
we see that N (h) has the claimed statisties. Second, observe that N (h) lim (h) where
N₂ lim
81 Ba,dh with de,n and 8; idn,n-
(This is simply Riemann approximation of the integral that defines (h))) As the sum of any finite number of Gaussian random variables a Gaussian random variable, N₂ (h) is Gaussian. Furthermore, as the limit of sequence of Gaussian random variables whose sequences of means and variances converge (to finite values) Gaussian, (h)) is Gaussian, which proves our claim.
B.2 Distributional Superiority In this section, we prove the statements and claims made Section 4. First, we prove that the mean of every (f, E (t, (t.z)) is -V
Lemma B.4. 1f x, a) (t, (7 ()), then its mean is Q, - (²(t
Proof. If (t, a) be such that A xr = v (t, a), then
I (00T / - w) x (1, x; (d2dw)
(1, a)) (1,
-,(1,, a) v=(,,)),
as desired,
20

---

Second, we prove Theorem 4.3.
Theorem 4.3. Let E (1,,1) po) for some μ. E (R)). The push-forward of K by A is the delta at
zero, Ayr= 60. if and only if k. is a Wy-optimal coupling, for some p E [1). Moreover, there is only one such coupling, is given by Ky == (id, id) # or, equivalently, Ki == (FF F U(0, 1). Here U[(0) 1) is the uniform distribution on [0. 1].
First, we establish that there is only one Wp optimal coupling between a given μ E (R) and itself, for every p [1. 00).
Lemma B.5. Let μa E P (R). There is only one Wp optimal coupling between μ and itself. for every p E 00). It is Ki = id) (1,
Proof. Let E 66 (p, ni), and suppose there exists e for which
c. := (11== w| 2, E R}) O.
Then
/ w| k(dzdw) 2
/
12 wIl k(dzdw) 2 lel > 0 / 12 12- Ky (dzdw).
(1x-w|2e)
Hence, as considered, is not optimal. Since was arbitrary, it follows that W, optimal coupling concentrated on {z w) Therefore, every optimal coupling of the form (id, id) 4 for some (R). As the marginals of such coupling are v and v, we deduce that μ., desired. 0
Proof of Theorem 4.3. Let E (1,, be such that Ag = ōg. Observe that
f w|" n(dzdw)) f 50(dr))
/ w/l Ky (dzzue)
where, again, Hi (id, æd) Hence, Ki,, by Lemma B.5. On the other hand, since Ky is concentrated on {a = w). for any bounded, continuous function g, we see
I ((()) A / g(2= - w) ka(dzdw) = (2mm) g(2 w) Ku kud(dzw)) = g(0).
In turn, Ayti = do, as desired.
Remark B.6. Under the hypotheses of Theorem 3.7, we see that the 1/2-reccaled superiority distribu- tions any (1, a) for h E (0, a family of probability measures with uniformly bounded second moment. Hence, this family is tight. So, up to subsequences, these rescalings converges to limiting probability measure as & O. An interesting open question, whether or not these subsequential limits are the same.
Third, prove Theorems 4.5 and 4.6. The proofs of these theorems are a consequence of the following expression for the W, distance between and y when μr, E (R):
Wz(n,v)P S/ -F)
Fr)
Theorem 4.5. MDPs and policies exist satisfying Assumptions 2.2 and 2.3 in and under which, for all (t,z) E T x X. we have that distgap, (1/: t, x) 2 12-9.
Proof. Recall that F-1 (1g,))))))))) h 1~ (((,2)))) F=1 (t.))) for all a E A. Hence, for every an # a2
W Wy(Wh (1, x, ai), Whig (1, a2))p
F-1 (1,,,,))
(T) (T)
h hr / 1 ( F=1 (2 (T)
h ())
21

---

Thus, taking the pth root of both sides of the above equality, we deduce that
Wp ( (t, x. 01), Whe (t a2)) = "Wp(ST
The example presented in Theorem 3.6 is such that WC, (1),, (1, 02)) 2 h Whence, Wo(Whig (1, x, a1) (1), Why (1, x. a2)) 2 h//2-9
Theorem 4.6. Under Assumptions 2.2 and 2.3, if Lipschitz in state, uniformly in time, f is Lipschitz, and T < sc, then distgap, (1 t,æ) S. h h. all (f,z) E T x X.
The proof of Theorem 4.6 almost identical to the proof of Theorem 4.5.
Proof. Arguing as in the proof of Theorem 4.5, we see that
Wp w 14 x, (1), Vheg (4, 02)) = "W (,,) a2)). By Theorem 3.7, it follows that W, (17 (1, 01), (h, (1, 02)) 1//9-99
Finally, we prove Theorem 4.8
Theorem 4.8. Let P3 be a distortion risk measure, 2 0, and h > 0. If pa 2)) < oc, then
mag Pa ( (t, 2, a)) arg maX P=(CT (t,
Proof. Observe that
P3 (t,,,a))
F-1 Wiy (t,,a) (T) ( 3(dr)
1- (t (T) 3(dr)
= h 1'' (F (F (T) - F (T)) ((dr)
= h "h xt, h <pppp( (17 (t,
Since ps( ()) independent and 0,
arg max Pa (1. (1, x. a)) arg max ps (1, a)).
C
Algorithms and Pseudocode
Here we discuss methods for policy optimization via distributional superiority. practice, computers operate at finite frequency-as such, all policies we consider here will be assumed to apply each action for units of time, as in the settings of [34, 16].
Before describing the superiority learning algorithms, we first remark on the form of exploration policies used in our approaches. We consider policies of the form
π: explore
x R + A.
The first argument to such a policy is a vector of "action-values"" In our case, given a superiority distribution oh (t, it),,,), the action values may be (pa (v (t, a))) for a distortion risk measure Ps. This generalizes the notion of Q-values for distributional learning.
The second argument represents noise variable, in order support stochastic policies. As input to our algorithms, we require probability measure Pact on processes (tz) This framework generalizes common exploration methods in deep RL. For example, recover e-greedy policies, Pact represent a 2-dimensional white noise, and
arg max, Va if S
explore (v.
(C.1)
(are. A|l
otherwise.
22

---

Alternatively, one might choose to correlate the action noise. Approaches such as DDPG [21] and DAU present such examples, where action noise evolves over time as an Omstein-Uhlenbeer process. This can be implemented in the framework above by choosing Pact as the distribution of an |Al-dimensional Ornstein-Uhlenbeck process, and defining
xxpplore (v, {)
arg max (u1 + E) aEA
(C.2)
In our experiments, we found that e-greedy exploration was sufficient for approximate policy optimiza- tion. To keep our implementation closest to the QR-DQN baseline, therefore, our implementations use the definition of π mxplore from equation (C.1).
The remainder of this section details the implementation of DSUP(() and DAU+DSUP(() as introduced in Section 4.2. Additionally, we provide source code for our implementations at https:// thub.
C.1 Distributional Superiority
Generally, our goal is to learn a Pa-greedy policy (see Definition 4.7). Since all policies apply actions for h units of time, in order to satisfy Axiom 2, we want
S x. (t), =0
where I b arg max, (s( (t, x. a)) is the pa-greedy policy. As such, following the DAU algorithm of [34], our algorithms will model quantile functions ø(t, a) R that aim satisfy
F (1,, = ott, xr a) o(t, x,))) for some a" E arg max ps
Our primary algorithmic contribution integrates such model, with proper superiority rescaling, into the QR-DQN framework of [10] for estimating action-conditioned return distributions. It is outlined in Algorithm 2.
To deal with the increased time-resolution of transitions, [34] modified the step size by a factor of h. More recently, [3] found that subsampling transitions before storing them in the replay buffer was most effective in their Rather than only storing every h transitions, randomly select transitions to store according to independent Bernoulli(h) draws in order to avoid the possibility of only capturing cyclic phenomena in the replay buffer. We found that this strategy worked similarly to that of [34], but is far less computationally expensive. Likewise, as h decreases, we extend the number of training interactions by factor of h-1 1 This corresponds to training for constant amount of time units across decision frequencies, and likewise, a constant number of gradient updates across decision frequencies.
sonddecciiio-frruenee domain. Thus, we opt for such a strategy here.
C.2 Two-Timescale Advantage-Shifee Distributional Superiority
An astute reader might recognize that while Vhy may have nonzero distributional action gap, since h9 is asymptotically larger than h for q < 1, the works of [34, 16] would suggest that the expected action gap under vih,, should vanish. To account for this, we propose shifting the rescaled superiority quantiles by the advantage function, as estimated c.g. in DAU [34]. It is clear that such a procedure cannot cause the distributional action gap to vanish. The resulting procedure is depicted in Algorithm 3, with the modifications relative to Algorithm 2 highlighted in blue. practice, we employ a shared feature extractor in the representations of A₁, and & to reap the representation learning benefits of DRL [4] when approximating the advantage.
C33 Influence of the Rescaling Factor
The algorithms 2 and
parameterized by a rescaling factor g (0, 1]. which is meant to compensate for the collapse of the distributional action gap. Larger values of q correspond to larger compensation. In this work, we argue that the distributional action gap collapses at rate h/',, leading natural choice of 1/2, which theoretically preserves constant order action gaps with respect to the decision frequency. We also test the approach with g = 1, which corresponds to the well-known scaling rate for preserving expected value action gaps.
For any q > 1/2, the distributional action gap theoretically grows without bound as h 4 O, leading to distributional estimates with arbitrarily large variance. On the other hand, for any q < 1/2, the distributional action gap decays to 0 as h 4 O, which makes difficult to identify the best actions in the presence of approximation error.
23

---

Algorithm 2 Distributional Superiority Quantile Regression
Require: (number of quantiles) and q (rescale factor)
Require: h- (decision frequency), (initial state distribution)
Require: nexpllre (exploration policy) and pa (distortion risk measure)
Require: Fact (action noise distribution)
Require: ((t, 2),, ((t, E IR".. parameterized quantile representations for (t, 2), o (t, a).
Require: (tt, E RT target quantile representations for (t,
Require: nhp (target update period), a (step size), N (batch size)
D 0- 0
reset +- True
for i N do
// Gather data with exploratory policy
(4).20 Pact
Sample from stochastic process for exploration noise
for j E fo.....,hh | do
if reset True then
t 0 and xa
reset False end if
a' c arg max, PB ( Em= )))))))))))))
F-1 (a) t Dtt,I;:00 a) olt, Iz, a*) for each a E A Phuu
Va + PB 1 (w)m.
for each a E
at ~ oxplore ((Va)aEA,55i
Perform action Or for units of time, observe (res Fr4h done;4x):
Yj ~ Bernoulli(h) if Yj V done;+1 True then
> Subsample transitions for replay
D + D U {(1, It, az, rt, &r done
end if
reset + done and t + + end for
// Update return/superiority distributions Sample minibatch ((t,, Ik, ak, Tk donex) from D for à E [N] do
at + arg max, PB ( Σm= ) Fr (o. 0) t ((x, v4) + h (0(k,, I2, ax) o(te, ak, JF-1 hr, + 21 done; 104ee h 11,²)) + 7 d
Risk-sensitive greedy action
D Prediction (4.4)
Target (4.5)
(1, 0) - QuantileHuber 1 ( end for E 0) N
See [10, Eq. (10))
Σm Lu(e,0000) l)(
N
+ Φ a ol( 0) and + 0 - avol().0)
> Gradient updates
if no then
0 9-6 0
end if
end for
24

---

Algorithm 3 Advantage-Shifted Distributional Superiority Quantile Regression
Require: (number of quantiles) and y (rescale factor)
Require: h- (decision frequency), ) (initial state distribution)
Require: ne (exploration policy) and ps (distortion risk measure)
Require: Pact (action noise distribution)
Require: 0)), (1, a) E IR"", parameterized quantile representations for 17" (1, 1h (1,2,, a).
Require: ((t, E R"", target quantile representations for 17" 2t)
Require: (t, a) (parameterized representation of advantage function)
Require: ne (target update period), a (step size), .N (batch size)
② 4-0 0
reset +- True
for i N do
// Gather data with exploratory policy
(E).20 Pact
Sample from stochastic process for exploration noise
for j E {o...., 1....,,h-1 [ do
if reset is True then
t 0 and To
reset False
end if'
a" t arg maga PB ( Em=1 5
F=1 (a) t ,(t, a) ott, Fz, a²) for each E A
Viu
A (t, + (t: (t,
xe,
> see [34, DAU]
Vo t PB Em= 6 8p (4)m-(1-h' (1)-- for each
ar oxplore ((V))EA::(j))
Perform action Or for h units of time, observe (Te, Zeth, done
Y; Bernoulli(h))
> Subsample transitions for replay
if Y V done True then
+ D U {(t, F2, ar, rı, It-h, done
end if
reset + done and t < + end for
// Update return/superiority distributions
Sample minibatch ((t,, Th, ak, PR, 2 done,)) from for k E [N] do
a + max, Pg
Em 6 o(te + +(1 (1
// Advantage Loss
(tk, xb. ax). t- An Th: Ik:
(t,,
(ttx, ax) + 1 Σn=1 Ix)n QttR Ih, ax)
Vtx,, ax) + hAp Ik. ax)
JQ(tk, Lbe, .x) hrrk + + Σx= A(t k h, h,,
gddv (Ax) t (Q(t2, ax) - JQ Fk, 04))
b Bellman error
// Superiority Loss F. (0,0) + o(t,, 24) h (0(tz, Ib, ax) - Octx, Ik, a*))
> Prediction (4.4)
JF + hrs + 1 done; (1x + h1 x1)) + donex
> Target (4.5) See [10, Eq. (10)
(0(0, 0) + Quantiletuberr(( (0, JF
end for
((bb, 0)
1 EN (x( 0) and padv (An) t 1
2N Σボ= padv (An)
N
+ a V
0) and + 0 Vol(6, 0) and An f An a gadv (An)
if i no then
0
end if
end for
25

---

D Additional Experimental Results
Figure D.I include results comparing the performance of DSUP(!/2) and QR-DQN for risk-sensitive option trading across a variety of decision frequencies.
20Hz
a-CVaR(Return
0.04
0.03
0.02
QR-DQN
0.01
DSUP
0.00
0.01
0.12
0.23
0.34
0.45
0.56
0.67
0.78
0.89
Cx
25Hz
a-CVaR(Return)
0.04
0.03
0.02
QR-DQN DSUP(1/2)
0.01
Q
30Hz
(-CCVa(RReurn)
0.01
0.03
QR-DQN DSUP(1/2) /2)
0.00
0.01
0.12
0.23
0.45
0.56
0.67
0.78
0.89
Cx
w 35Hz
Q2- (
0.05
0.02
QR-DQN
0.01
DSUP(1/2) (1)2
0.00
0.01
0.12
0.21
0.34
0.45
0.56
0.67
0.78
0.89
Q
Figure D.I: Risk-sensitive option trading performance for various decision frequencies
E
Simulation Details
Here we collect further information about the setup for the simulations described in Section 5.2.
E.1 Option Trading Environment
The environment used for the high-frequency option-trading setup is identical to that of Lim and Malik [22]. The enviromment emulates policies that decide when to exercise American call options. The state space is modeled as X IR and with T [0, T'. Notably, existing works such [22] and [17] describe the state space by X = g² where one dimension represents time-in our setup, generally condition policies and returns on time, so we indeed model policies and returns as functions on 2 The state X; represents the price of the option time and evolves according a geometric Brownian motion.
26

---

There are two actions in the environment. Action O "holds" the option, while action represents "execute". Upon taking action (or equivalently, once the time reaches the option executed, and the agent receives a reward ((x) max(0, x) and the episode terminates; here æ represents the price the time of execution, No rewards are incurred otherwise.
Following the setup of [22]. the dynamics of the prices are simulated based on data collected between years 2016 and 2019 from 10 commodities on the DOW market. Lim and Malik proposed method for estimating the most likely parameters of geometric Brownian motion to fit the data for each commodity, which is then used to simulate many environment rollouts for training and evaluation. This is particularly convenient for our setup, where we additionally scale the decision frequency, corresponding finer time discretizations of the Euler-Maruyame scheme for the estimated geometric Brownian motion. Like [22], separate dynamics parameters are estimated (for each commodity) between training and evaluation: the dynamics used for training are estimated on prefixes of the data, and those for testing (post-training) are estimated on suffixes of the data. Results are reported on the testing dynamics, averaged over the 10 commodities.
As is standard [22, 17], we simulate the environment with 7 = 100 and Xo 1. The simulations from [22] correspond dynamics with timestep h < 1.
in this setting. In our figh-freyeency simulations, discretize the
E.2 Hyperparameters
In Table I, we list the hyperparameters used in the simulations for the tested algorithms. We note that although the original DAU implementation scaled the learning rate with h, take alternative approach by updating every h-1 environment steps akin to [3]. This discussed in more detail in Appendix C.
Table 1: Hyperparameters
Method All
Parameter
Value
Replay buffer sampling
Uniform
Replay buffer capacity
20000
Batch size
32
Optimizer
Adam [18]
Learning rate
le-4
Discount factor
0.999
Target network update period
1000
e-greedy exploration parameter
Linear decay from 1 to 0.02
DAU [34]
Value network
MLP (2 hidden, 100 units, ReLU)
Advantage network
MLP (2 hidden, 100 units, ReLU)
QR-DQN [10, 22]
Quantile network torso
MLP (2 hidden, 100 units, ReLU)
Number of atoms
100
Quantile Huber parameter K.
1
DSUP
r network torso
MLP (2 hidden, 100 units, ReLU) MLP (2 hidden, 100 units, ReLU)
(superiority proxy) network torso
Number of atoms
100
Quantile Huber parameter
1
DAU+DSUP
Advantage network torso
(shared with Φ)
E.3
Compute Resources
Our implementations are written in Jax [6] and executed with single NVidia V100 GPU. At highest decision frequencies, experiments took longer to execute, averaging out a maximum of roughly four hours.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe our abstract and introduction outline and faithfully summarize the content of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have included discussions of the limitations and scope of our results throughout our work, rather than within specific "Limitations" section. See, for example, Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex. system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
28

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
We have provided complete statements, proofs, and references to used results in our proofs our theoretical results. Please see Appendix for restatements and proofs. Statements can be found in the main body of our work.
Justification:
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We believe we have included enough information that others can faithfully reproduce our main experimental results. Please see Sections and 5 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
29

---

(a) If the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided data, code. and experimental details. We believe we have included enough information that others can faithfully reproduce our main experimental results. Please see Section and the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc Belliic/guies/Cee (CoodeeSaisssioonn..o
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: // httpps.cc/pbblll
Cileee//ooSSubbiese ionooiccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provided these details Section 5 and the Appendix.
Guidelines:
30

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: When appropriate, have included error bars and information regarding the statistical significance of our experiments. Please see Section and the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included these details in Section 5 and the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
31

---

The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://uuurrss..ppppppppie wctppublcc/hiiicsuuuuieeee.eee
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and believe our work conforms to it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: We believe our work is foundational research without direct path to negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly. harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
32

---

Answer:
[NA]
Justification:
We believe work poses risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Data for the option-trading environment was downloaded from an open source Github repository, and cited the work that introduced the dataset [22].
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithe thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
33

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work neither involves crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work neither involves crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
34


Reviews:
[{"review_id": "pbGxmMgR9B", "reviewer": "Reviewer_TDv5", "summary": "The paper investigates the issues around continuous-time distributional reinforcement learning. In traditional RL, the advantage becomes less informative as the frequency of actions increases, vanishing at the limit and making it impossible to distinguish between actions. This work extends this result to distributional RL in several ways: it first proposes a framework based on Wasserstein distances between return distributions, then proves in this framework that a similar problem appears in DLR, and gives tight bounds on the asymptotic convergence rate of the distgap (an action gap analogue for continuous DLR), establishing that the rate differs from the traditional RL case.\n\nAfter that, the paper constructs a notion of superiority - a DLR analogue of advantage, and considers its different variants (transformations) that ensure that action gaps are preserved in the limit. Finally, two families of algorithms are constructed, and shown to outperform the baseline QR-DQN on a benchmark of high-frequency options trading task.", "questions": "1. The Wasserstein distances considered are for $p \\in [1, \\infty)$. What breaks if we put $p = \\infty$?\n2. Why is the behavior of $\\vartheta$ mean non-monotone wrt frequency in the example considered in sec 5.1?\n3. Apart form distributional RL, could a similar method be applied to continuous-time max-ent RL?", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper investigates a very interesting question, and gives a theoretically satisfactory answers. The formal analysis gives rise to a practical algorithm, which is empirically shown to outperform the QR-DQN baseline. The presentation is very readable while being maximally mathematically precise, which I really appreciated.", "weaknesses": "The theoretical part requires quite a lot of background from the reader. While authors give a short intro to SDEs in the appendix, it is, in my opinion, nowhere near the depth required to understand the paper. While it's not really possible to present a comprehensive intro to stochastic analysis, the paper would benefit, in my opinion, from a slight expansion of the relevant appendix. Also in terms of presentation, I found the explanation of the algorithmic part of the paper really dense and confusing, and I cannot say I understood it particularly well from the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough assessment of our work, their interest in its results, and insightful comments.\n\n**Re: Background on SDEs.**\n\nThank you for pointing this out. We appreciate this suggestion. We will happily expand our background on SDEs. We plan to incorporate discussion on and references to the relationship between MDPs governed by SDEs and MDPs governed by transition probabilities, as they are presented more traditionally in RL. We feel this might make readers primarily familiar with discrete-time RL, for instance, more comfortable with the setting of the theory part of our paper as well as some of our technical assumptions. \n\nIs there something else explicit you feel would benefit our work?\n\n**Re: Density of Algorithm Section.**\n\nAgain, thank you for pointing this out. We appreciate this as well. We will happily expand this section too. In our revision, we will add detailed descriptions (in Section 4.2) and pseudocode (in Appendix C) for both DAU and QR-DQN. We feel this should help clarify how our proposed methods deviate and model the rescaled superiority distribution.\n\nDo you have specific suggestions that add to what we have proposed to incorporate?\n\n**Q1**: $p = \\infty$.\n\n**A1**: Theorems 3.7 and 4.8, for example, break in the case of $p = \\infty$. That said, to freely work with $W_\\infty$, our state processes should have bounded sample paths, which is violated even in the case of Brownian sample paths. (See, e.g., Section 5.5.1 of *Optimal Transport for Applied Mathematicians* by Santambrogio for more details on working with $W_\\infty$.) However, since $W_\\infty \\geq W_p$ for all $p$, some of our analysis does hold: Proposition 3.4, Theorem 3.6, Theorem 4.5, and Theorem 4.7.\n\n**Q2**: Non-monotonicity of the mean of $\\vartheta^\\pi_{h;1/2}$ in Section 5.1.\n\n**A2**: Good question, thank you for pointing this out! The true distributions should actually all have the same mean, roughly 100. They do not because of Monte Carlo approximation error (MCAE). Furthermore, as we are operating at extremely high decision frequency, errors are amplified quite a bit. This is less apparent in the other subplots because the variance blow-up and/or mean collapse dominates the MCAE.\n\n**Q3**: Continuous-time MaxEnt RL.\n\n**A3**: This is a really interesting question! Yes, our theory provides insight into the distributional properties of returns (and the issues with their estimation from data) in this setting, as a function of the decision frequency, because MaxEnt RL can be seen as an instance of expected-value RL with a policy-dependent reward function. Thus, we believe our DSUP algorithms would be effective for learning return distributions in continuous-time MaxEnt RL as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XW1CJRfJMx", "reviewer": "Reviewer_ZgER", "summary": "This paper investigates the action gap in distributional RL, where the decisions are made at high frequency. The authors showed that the distributional RL is also sensitive to the decision frequency. In particular, they proved the action-conditioned return distribution collapse with different rates for the statistics. Also, they introduce the generalization of the advantage function in continuous-time MDP, i.e., superiority, based on which a novel algorithm is designed. The authors finally conducted extensive experiments in the high-frequency option-trading domain to demonstrate their theory and the efficacy of their algorithms.", "questions": "1. While the authors stated that DAU+DSUP(1/2) is inferior to DSUP(1/2), it is indeed a clear drawback of the proposed algorithm as this empirical result is consistent with the theory stated in Section 4.2, and hard to understand to me. Although the authors gave some explanations, they did not rigorously validate their hypothesis. Also, I am not convinced why DSUP(1/2) performs poorly in low-frequency domains. \n\n2. Why only consider 30HZ and DSUP(1/2) in the risk-sensitive setting? My suggestion is to include DAU+DSUP(1/2) as well for an extensive comparison across different frequencies, which could be more convincing. \n\n3. It is suggested that more examples be given to emphasize the significance of studying continuous-time MDP and the motivation to explore distributional RL.\n\n4. It would be better to provide the pointers to the proof of each theorem to allow readers to check the proof instantly.\n\n5. In what sense does the approximation hold in Eq. 4.2?", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The writing is rigorous with detailed and clear definitions and theorems. \n\n* Algorithmic practice in this setting is creditable. \n\n* Experiments are relatively extensive, considering both illustrated and comparative settings.", "weaknesses": "* The motivation can be strengthened. What is an action gap in continuous-time MDP, and what is an algorithm's outcome when encountering high-frequency decision-making settings? Why do we investigate the distributional RL in this setting? More explanation would be better.\n\n* The clarity can be enhanced. Although I appreciate the rigorous writing, it would be better to emphasize the main conclusions of this paper. For instance, the statistics in distributional RL also decrease at different rates. How do readers learn from these observations? Some rigorous theorems may be put in the appendix and replaced with more straightforward explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough assessment of our work, their interest in its results, and insightful comments.\n\n**Re: Motivation and Clarity.**\n\nThank you for pointing to these potential areas of improvement. We very much appreciate your suggestions. We will be sure to update our revision with this in mind. Please see the general response for some more detail on how we will do this.\n\n**Q1a**: DAU+DSUP(½) vs DSUP(½).\n\n**A1a**: This is an important point to clarify. Thank you for making it! Please see the general response.\n\n**Q1b**: DSUP(½) at low decision frequency.\n\n**A1b**: We do not agree that DSUP(½) performs poorly at low decision frequencies. Figure 5.2 shows that DSUP(½)’s performance is similar to the best performer in the lowest two frequencies (the only two where it is not the best performer). These lowest frequencies are the only ones where the baselines are competitive, since action gaps are not an issue here.\n\n**Q2a**: Other frequencies in CVaR experiment.\n\n**A2a**: Among the higher decision frequencies, we felt that QR-DQN stood a fighting chance only in the 30Hz setting. This choice was based on how significantly DSUP(½) out-performs QR-DQN (see Figure 5.2). We have since run the experiment at several decision frequencies as you suggested. A PDF of these results appears with the general response document. We felt they would be of interest to all the reviewers. Note that the trend observed in Figure 5.4 persists across the full range of now-considered decision frequencies.\n\n**Q2b**: DAU+DSUP(½) in CVaR experiment.\n\n**A2b**: We note that DAU+DSUP(½) *is not* principled for risk-sensitive control. In particular, DAU+DSUP(½) does not preserve the ranking of actions with respect to CVaR relative to the action-conditioned return distributions (i.e., Theorem 4.10 only holds for DAU+DSUP(½) when $\\beta$ is the uniform measure on $[0,1]$, that is, when we are in the risk-neutral or expected-value setting). However, Theorem 4.10 shows that DSUP(½) *is* principled for general $\\beta$.\n\n**Q3**: Additional motivating examples.\n\n**A3**: We will be sure to point to more motivating examples in our revised draft.\n\n**Q4**: Pointers to proofs.\n\n**A4**: We will hyperlink the proofs directly beneath the theorems in our revision.\n\n**Q5**: Approximation in Eq. 4.2.\n\nA5: The approximation holds in the sense that the support of the law of the error $Y_h$ is contained within a $o(h)$ radius of $0$. Please see the exact equation just above line 247.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2IvVFWY1HL", "reviewer": "Reviewer_dYRz", "summary": "This paper focuses on the scenario of continuous RL with high decision frequency. They show that it is hard for distributional RL agents to accurately estimate action values when the decision frequency increases, just like in the case of ordinary RL agents. They propose a distributional analogy of the action gap in ordinary RL called superiority. Based on this notion they try to mitigate this issue by the rescaling technique and propose a superiority-based algorithm. They also validate the proposed methods by numerical simulations.", "questions": "* I note that the superior distributions are actually rescaled by $h^{-q}$, where $q$ is a tuning parameter. In the empirical studies, $q$ is set to be $1$ or $1/2$. Are there any possible heuristic rules for the choice of $q$?\n\n* It seems that in the simulation studies, the performances of some methods (DAU, DSUP(1), DAU+DSUP(1/2)) are not monotone w.r.t. the decision frequency, but oscillate a lot as the decision frequency increases. Why would this happen? Can the authors give a more detailed explanation?", "limitations": "It seems the authors do not explicitly addressed the limitations of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is overall well-written and easy to follow.\n\n* The idea of superiority distributions is interesting and intuitive, with solid theoretical justifications.\n\n* The work is quite complete. Besides the discussion about the action gap in distributional RL and the notion of superiority distributions, the author also proposed new algorithms and performed extensive empirical studies.\n\nTherefore, I choose to give a positive rating of this work. However, given that I am unfamiliar with continuous RL, I choose to assign a low confidence score to my review.", "weaknesses": "* It seems that the basic idea of the superiority distribution is that for two distributions $\\eta_1,\\eta_2$, we define an object $\\Delta$ that can be viewed as the difference between $\\eta_1$ and $\\eta_2$. And then given a risk-sensitive objective $\\phi$, one may choose actions according to rescaled $\\phi(\\Delta/h^q)$, where $h$ denotes the decision frequency. But why not simply use the criterion $(\\phi(\\eta_1)-\\phi(\\eta_2))/h^q$ for decision making? I think the authors should discuss this issue in later versions of this paper.\n\n* I suggest the author add some explanations about the problem setting to the intro section, which would make the paper more readable for those not familiar with the specific field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough assessment of our work, their interest in its results, and insightful comments.\n\n**Re: Discussion on using $(\\phi(\\eta_1) - \\phi(\\eta_2))/h^q$ for decision making.**\n\nThank you for pointing this out. We appreciate your suggestion here. We will be happy to add text that speaks to this in our revised draft. In practice, it is generally difficult to estimate $(\\phi(\\eta_1) - \\phi(\\eta_2))$ without estimating $\\eta_1$ and $\\eta_2$ (see, e.g., Chapter 7 of *Distributional Reinforcement Learning*, Bellemare, Dabney, and Rowland 2023). This is one reason why distributional RL is so attractive. Moreover, there are other benefits to modeling return distributions. For instance, modeling distributions is helpful in obtaining second order regret bounds (see *More Benefits of Being Distributional: Second Order Bounds for Reinforcement Learning* by Wang et al., 2024). Finally, even if we could estimate $(\\phi(\\eta_1) - \\phi(\\eta_2))$ directly, some form of rescaling would still be necessary. This is one of the novel contributions of our work.\n\n**Re: The problem setting.**\n\nThank you for raising our awareness to this. We very much appreciate this suggestion, and we will be sure to update our revision with this in mind. We discussed this in further detail in our general response.\n\n**Q1**: Tuning $q$.\n\n**A1**: In reality, $q$ isn’t a hyperparameter to tune. The theoretical section of our work shows that the only appropriate/principled choice of $q$ is ½. For all other choices, distributional action gaps either blow up or vanish, as $h$ decreases. Both of these behaviors are undesirable. We model $q = 1$ for benchmarking purposes, since this corresponds to rescaling in AU/DAU. That said, if one insists on treating $q$ as a hyperparameter, choosing larger $q$ will further increase the distributional action gap, but at the cost of extremely large variance; this will likely hinder performance, as seen by the performance of DSUP(1), for example.\n\n**Q2**: Oscillatory performance of DAU, DSUP(1), DAU+DSUP(½).\n\n**A2**: This is an important point to clarify, thank you for making it! Please see the general response.\n\n**Re Limitations**: \n\nAs noted in our checklist, limitations are addressed throughout the text where relevant. For instance, note all technical restrictions stated formally as Assumptions. The only other restriction is on the class of reward functions, which we specify in Footnote 1. These Assumptions and Footnote 1, while stated in full precision in the main text, are elaborated on in Appendix A. \n\nIf there is something explicit you would like us to address, please let us know; we are happy to do so.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We are thankful for the interest in our work as well as the time and effort taken to review it. 

Reviewers praised our work for the problem that we have highlighted and the completeness of our theoretical treatment of it (TDv5, ZgER, dYRz), our general clarity and rigor of exposition (TDv5, dYRz), and our extensive empirical evaluation (TDv5, ZgER, dYRz). In their assessments, the reviewers suggested our paper would benefit from more heuristic discussion of the problem setting, asked for clarity regarding the behavior of some of the algorithms as a function of decision frequency, and requested for our risk-sensitive experiments to be conducted over additional decision frequencies.

We speak to these suggestions and requests here; other queries posed by reviewers are addressed in our individual responses to them. 

**Problem Setting: Motivation and Clarity**

Reviewers ZgER and dYRz suggested our paper would benefit from more heuristic discussion of the problem setting and motivations for its study to aid in its accessibility.

We appreciate this feedback, and we will amend our introduction accordingly. We propose to preface the introduction with content regarding the scope of the problem setting along the lines of the following text.

“Our work investigates the performance of RL agents in systems where states evolve continuously in time, but policies make decisions at discrete time steps ($h$ units of time apart). Many real-world deployments of RL are within such systems, since the world is continuous in time, yet computerized RL policies operate in discrete time. Within these systems, many factors influence the decision frequency, such as the quality of sensors and the speed of CPUs. And while more responsive policies should perform better, this is not always true in practice. As such, our goal is to design algorithms that perform well across the continuum of decision frequencies.”

Furthermore, in our revised introduction, we will be sure to use the prompts given by ZgER as guides to further expand the accessibility of our work.

**Performance of DAU, DSUP(1), and DAU+DSUP(½)**

Reviewers ZgER and dYRz were curious about the performance of DAU, DSUP(1), and DAU+DSUP(½). 

The issue here is purely with respect to estimating $h$-rescaled advantages in stochastic environments: the $h$-rescaled superiority, as we showed theoretically and via simulation, has unbounded variance as $h$ decreases. Thus, estimating its mean (the $h$-rescaled advantage) is challenging and sensitive to noise when $h$ is small. Our work is the first, to our knowledge, to highlight this issue.

We see the oscillatory performance of DAU, DSUP(1), and DAU+DSUP(½) as additional symptoms of this issue. Indeed, their individual performances track one another, because they all use $h$-rescaled advantage estimates for control. In DAU and DAU+DSUP(½), this is done explicitly. In DSUP(1), it is done implicitly (by modeling the $h$-rescaled superiority distribution and acting according to its expectation). 

Overall, our work is the first (to our knowledge) to highlight the statistical challenges of modeling $h$-rescaled advantages in stochastic environments. Our main practical focus was to design an algorithm that maintains distributional action gaps, which is accomplished by DSUP(½). Further investigation into improved methods for modeling $h$-rescaled advantages in stochastic environments is an important direction we leave to future work.

In our revised draft, we will expand our presentation of DAU, DSUP(1), and DAU+DSUP(½) to contain what we have briefly outlined here. 

**Risk-Sensitive Control Across Decision Frequencies**

Reviewer ZgER suggested that our analysis of the performance of our risk-sensitive superiority-based algorithms can be strengthened by testing across more decision frequencies. 

We include some further results to this end in the attached PDF. In particular, we test DSUP(½) and QR-DQN at 3 additional frequencies, to round out the 4 highest frequencies plotted in Figure 5.1. Note that DSUP(½) out-performs QR-DQN in these 3 additional decision frequencies as well. Also, these results confirm our expectations from our theoretical results: DSUP(½) is stable across decision frequencies, while QR-DQN struggles at these high decision frequencies (even more so than we expected).

Meta-review: This paper investigates the challenges arising from dealing with continuous-time control in distributional RL. In particular, whenever decision are made at a high frequency, it comes harder to compare the relative advatage of actions, thus making it difficult or impossible to distinguish between them. This paper introduces a framework exploiting Wasserstein distances between return distributions to provide bounds on the convergence rate of the gap between actions and generalizes the notion of advantage functions to continuous-time MDPs. 

All reviewers had an overall positive view of this work, highlighting the importance of the problem being tackled by the authors and the thoroughness of the presented theory. ZgER praised the relatively extensive experiments conducted by the authors. Reviewer ZgER suggested that the empirical analyses presented in this work be strengthened by testing the proposed idea across additional decision frequencies, which the authors did. Reviewers brought up a few (relatively minor) points of contention, and the discussion phase was fruitful: the authors did address the most important concerns mentioned in the reviews and pointed out concrete ways in which their work can and will be improved in terms of presentation and clarity.

The most common and pressing concern regarding this paper was its clarity. Although reviewers agreed that the writing is rigorous, the consensus is that the paper is not necessarily accessible to a broad audience due to the highly dense text and the authors not always discussing intuitions that underlie the ideas being investigated and proposed. This will most likely limit how accessible the contributions introduced in this paper are to the NeurIPS community. The reviewers strongly encouraged the authors to improve their document by adding further intuition and discussion and by improving clarity to ensure that the insights introduced in this work become more apparent to a broader audience of readers who may not be, a priori, experts on all theoretical concepts exploited in this paper.
Decision: Accept (poster)