Title: Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: distributional reinforcement learning

reinforcement learning

continuous time

advantage updating

stochastic differential equations
Abstract: When decisions are made at high frequency, traditional reinforcement learning (RL) methods struggle to accurately estimate action values. In turn, their performance is inconsistent and often poor. Whether the performance of distributional RL (DRL) agents suffers similarly, however, is unknown. In this work, we establish that DRL agents *are* sensitive to the decision frequency. We prove that action-conditioned return distributions collapse to their underlying policy's return distribution as the decision frequency increases. We quantify the rate of collapse of these return distributions and exhibit that their statistics collapse at different rates. Moreover, we define distributional perspectives on action gaps and advantages. In particular, we introduce the *superiority* as a probabilistic generalization of the advantage---the core object of approaches to mitigating performance issues in high-frequency value-based RL. In addition, we build a superiority-based DRL algorithm. Through simulations in an option-trading domain, we validate that proper modeling of the superiority distribution produces improved controllers at high decision frequencies.

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning

Abstract

When decisions are made at high frequency, traditional reinforcement learning (RL) methods struggle to accurately estimate action values. In turn, their performance is inconsistent and often poor. Whether the performance of distributional RL (DRL) agents suffers similarly, however, is unknown. In this work, we establish that DRL agents are sensitive to the decision frequency. We prove that action-conditioned return distributions collapse to their underlying policy's return distribution as the decision frequency increases. We quantify the rate of collapse of these return distributions and exhibit that their statistics collapse at different rates. Moreover, we define distributional perspectives on action gaps and advantages. In particular, we introduce the superiority as a probabilistic generalization of the advantagethe core object of approaches to mitigating performance issues in high-frequency value-based RL. In addition, we build a superiority-based DRL algorithm. Through simulations in an option-trading domain, we validate that proper modeling of the superiority distribution produces improved controllers at high decision frequencies.

1 Introduction

In many real-time deployments of reinforcement learning (RL)-quantitative finance, robotics, and autonomous driving, for instance-the state of the environment evolves continuously in time, but policies make decisions at discrete timesteps ( h units of time apart) [28]. In such systems, the performance of value-based agents is sensitive to the frequency ω := 1 / h with which actions are taken. In particular, action values become indistinguishable as the time between actions decreases. In turn, in high-frequency settings, Baird demonstrated that action value estimates are susceptible to noise and approximation error [20]. Moreover, Tallec et al. exhibited that the performance of popular deep Q -learning agents is inconsistent and often poor [34].

In order to remedy this sensitivity, Baird proposed the advantage function and advantage-based variants of Q -learning, Advantage Updating (AU) [20] and Advantage Learning (AL) [2]. Unlike action values, advantages (appropriately rescaled) do not become indistinguishable as decision frequency increases. As a result, Baird, in [20, 2], demonstrated that advantage-based agents can learn faster and be more resilient to noise than their action value-based counterparts. Furthermore, Tallec et al., in [34], exhibited that their extension of AU, Deep Advantage Updating (DAU), works efficiently over a wide range of timesteps and environments, unlike standard deep Q -learning approaches.

While advantage-based approaches to RL have demonstrated robustness to decision frequency, in this work, we establish that they are nevertheless sensitive to the frequency with which actions

are taken. This discovery arises as we answer the question: to what extent is the performance of distributional RL (DRL) agents sensitive to decision frequency? To this end, we build theory within the formalism of continuous-time RL where environmental dynamics are governed by SDEs, as in [25]. Additionally, we validate our theory empirically through simulations. Specifically, we make the following four contributions:

Distributional Action Gap. First, we extend notions of action gap to the realm of DRL. Precisely, we consider the minimal distance between pairs of action-conditioned distributions under metrics on the space of probability measures on R . We observe that some metrics are viable for this extension, while others are not. This formalism sets the stage for analyzing the influence of individual actions as well as decision frequency on, for example, an agent's return distributions.

Collapse of Distributional Control at High Frequency. Second, we establish tight bounds on the distributional action gaps of h -dependent action-conditioned return distributions -return distributions induced by applying a specific initial action for h units of time. We prove that these distributional action gaps not only collapse, as h tends to zero, but do so at a slower rate than action-value gaps. On one hand, therefore, distributional Q -learning algorithms are susceptible to the same failures as Q -learning in continuous-time RL. On the other hand, however, remedies to these failures transliterated to distributional Q -learning algorithms are unlikely to succeed, because the means of these return distributions collapse faster than their other statistics.

Distributional Superiority. Third, we propose an axiomatic construction of a distributional analogue of the advantage, which we call the superiority . Leveraging our analysis of h -dependent actionconditioned returns and their distributional action gaps, we present a frequency-scaled superiority distribution that enables greedy action selection at any fixed decision frequency.

A Distributional Action Gap-Preserving Algorithm. Fourth, we propose an algorithm that learns the superiority distribution from data. Empirically, we demonstrate that our algorithm maintains the ability to perform policy optimization at high frequencies more reliably than existing methods.

2 Setting

Notation: Spaces will either be subsets of Euclidean space or discrete. Measurability, in the former case, will be with respect to the Borel sigma algebra; in the latter case, it will be with respect to the power set. The set of probability measures over a space Y will be denoted by P ( Y ) . Functions on spaces are assumed to be measurable. For f : Y → Z and µ ∈ P ( Y ) , the push forward of µ through/by f , f # µ ∈ P ( Z ) , is defined by f # µ := µ ◦ f -1 . For a random variable X , defined implicitly on some probability space (Ω , F , P ) , we write law ( X ) := X # P to denote the law of X ; the notation X = law Y is shorthand for law ( X ) = law ( Y ) . For any µ ∈ P ( R ) , the quantile function of µ , F -1 µ , is defined by F -1 µ ( τ ) := inf z { F µ ( z ) ≥ τ } , where F µ ( z ) is the CDF of µ .

2.1 Continuous-Time RL

Here we give a brief introduction to the technical aspects of continuous-time RL, à la [25]. We provide additional exposition and references in Appendix A. For any reader looking to defer some of this technical introduction, we summarize the core objects of interest at the end of Section 2.1.1.

2.1.1 MDPs

Continuous-time Markov Decision Processes (MDPs) are defined by three spaces and four measurable functions: a time interval T := [0 , T ] with T ∈ (0 , ∞ ) or T := [0 , T ) with T = ∞ , a state space X ⊂ R n , an action space A , a drift b : T × X × A → R n , a diffusion σ : T × X × A → R n × n , a reward r : T × X → R , and a terminal reward f : X → R . 3 The pair ( b, σ ) govern the environment's dynamics by a family of SDEs parameterized by a ∈ A ,

Here ( B t ) t ≥ 0 is an n -dimensional Brownian motion. In turn, any solution to (2.1) collects the state paths of an agent that chooses action a at every time, regardless of the state they are in.

As is done in discrete-time RL, an agent might consider the induced Markov Reward Process (MRP) derived from a policy π : T × X → P ( A ) . 4 The dynamics of a policy-induced MRP (with policy π )

are governed by the SDE

Here, following [38, 15, 16], the policy-averaged coefficients b π and σ π are defined by

Thus, solutions to (2.2) collect the paths of an agent following policy π .

A class of policies central to our study is those that fix an action a from some time t for a given persistence horizon h .

Definition 2.1. Given h > 0 and a ∈ A , a policy π is said to be ( h, a ) -persistent at time t ∈ T if π ( · | s, y ) = δ a for all ( s, y ) ∈ [ t, t + h ) × X .

In particular, given a policy π , we will consider ( h, a ) -persistent modifications of π : for t ∈ T ,

These policies will help us understand the influence of taking actions relative to others as well as to those taken by π . We assume h is small enough so that t + h ∈ T .

In order to guarantee the global-in-time existence and uniqueness of solutions to our SDEs (2.1) and (2.2), we make two sets of assumptions.

Assumption 2.2. The functions b and σ have linear growth and are Lipschitz in state, uniformly in time and action: a finite, positive constants C 2 . 2 exists such that

Assumption 2.3. The averaged coefficient functions b π and σ π are Lipschitz in state, uniformly in time: a finite, positive constant C 2 . 3 exists such that

These assumptions are standard in the analysis of continuous-time RL, optimal control, and SDEs [12, 26, 38, 16, 42]. Since π is a function of state, we note that Assumption 2.3 is not a direct consequence of Assumption 2.2. The coefficients b π and σ π satisfy the conditions of Assumption 2.3 provided b and σ satisfy some (also standard) additional regularity conditions and π satisfies some regularity conditions. These technical details are discussed in Appendix A.2.

In summary, in continuous-time RL, there are three stochastic processes of interest: ( X · s ) s ≥ t with · ∈ { a, π, π | h,a,t } , all beginning at some time t . These processes collect the state paths of an agent in one of three scenarios: 1. choosing action a at every state and time; 2. following a policy π ; or 3. choosing a at every state and time for the first h units of time and following π thereafter.

2.1.2 Value Functions and their Distributions

Given a policy-induced state process ( X π s ) s ≥ t , the (discounted) random return G π ( t, x ) earned by π starting from state x ∈ X at time t ∈ T is defined [41] by

where f ≡ 0 when T = [0 , ∞ ) . We distinguish returns earned by ( h, a ) -persistent modifications of policies. We call these h -dependent action-conditioned returns and denote them by Z π h ( t, x, a ) . Given π , they are defined by

Value-based approaches in RL estimate either the value function V π ( t, x ) := E [ G π ( t, x )] or the h -dependent action-value function Q π h ( t, x, a ) := E [ Z π h ( t, x, a )] . 5 As distributional approaches in RL estimate the laws of returns, following [41], we define

It is important to note that only the laws of random returns (and not their representations as random variables) are observable and modeled in practice.

2.2 Q -Learning in Continuous Time

The failure of action-value-based RL in continuous-time stems from the collapse of action values at a given state to the value of that state. Precisely, Tallec et al. and Jia and Zhou established that

where H π ∈ R is independent of h (see [34] and [16] respectively). 6 In a discrete action space, given a state x and time t , a concise way to capture the asymptotic information of (2.6) is by considering the action gap [11, 5] of the associated h -dependent action values

̸

Anticipating (2.6), which implies that gap ( Q π h , t, x ) = O ( h ) , Baird proposed AU wherein he estimated the rescaled advantage function A π h in place of Q π h :

Note that gap ( A π h , t, x ) = O (1) . Tallec et al. [34] and Jia and Zhou [16], following Baird, also estimated A π h to ameliorate Q -learning in continuous time.

3 The Distributional Action Gap

In this section, we define a distributional notion of action gap; we prove that h -dependent actionconditioned return distributions collapse to their underlying policy's return distribution as h vanishes; and we quantify the rate of collapse of these return distributions.

Definition 3.1. Consider an MDP with discrete action space and let µ : T × X × A → ( P ( R ) , d ) for a metric d . The d action gap of µ at a state x and time t is given by

̸

While R has a canonical metric, induced by | · | , the space P ( R ) does not. So a choice must be made, and some metrics are unsuitable. For example, in deterministic MDPs with deterministic policies, return distributions are identified by expected returns: ζ π h ( t, x, a ) is the delta at Q π h ( t, x, a ) , for all ( t, x, a ) ∈ T × X × A . Thus, distgap d ( ζ π h , t, x ) should vanish as h decreases to zero if gap ( Q π h , t, x ) vanishes as h decreases to zero. With the total variation metric d = TV , for instance, this is not the case, making TV unsuitable. Indeed, suppose we have a deterministic MDP with A = { a 1 , a 2 } and such that ζ π h ( t, x, a 1 ) = δ h and ζ π h ( t, x, a 2 ) = δ 0 , for some state x and time t (see, e.g., [34]). Then distgap TV ( ζ π h , t, x ) = 1 , for all h > 0 , yet gap ( Q π h , t, x ) = h .

The W p distances from the theory of Optimal Transportation (see [36]), however, are suitable. They are defined via couplings of distributions.

Definition 3.2. Let µ, ν ∈ P ( R ) . A κ ∈ P ( R 2 ) is a coupling of µ and ν if its first and second marginals are µ and ν respectively. We denote the set of these couplings by C ( µ, ν ) .

Definition 3.3. Let µ, ν ∈ P ( R ) 7 and p ∈ [1 , ∞ ) . The W p distance between µ and ν is

Any coupling attaining the infimum in (3.1) is called a W p -optimal coupling . Henceforth, we write distgap p when considering W p action gaps. If µ and ν are deltas at Q π h ( t, x, a 1 ) and Q π h ( t, x, a 2 ) respectively, then the right-hand side of (3.1) is equal to | Q π h ( t, x, a 1 ) -Q π h ( t, x, a 2 ) | . Hence, in deterministic MDPs with deterministic policies, W p action gaps of ζ π h are identical to action gaps of Q π h , making the W p distances suitable in the above sense. In non-deterministic MDPs, the relationship between distgap p ( ζ π h , t, x ) and gap ( Q π h , t, x ) is opaque.

The following results study the W p action gap of ζ π h as a function of h , lending some color to the relationship between distgap p ( ζ π h , t, x ) and gap ( Q π h , t, x ) . These results all hold under Assumptions 2.2 and 2.3. Henceforth, we suppress mention of these assumptions; we do not restate them explicitly. First, we observe that W p action gaps of ζ π h are bounded from below by action gaps of Q π h .

Proposition 3.4. For all ( t, x ) ∈ T × X , we have that distgap p ( ζ π h , t, x ) ≥ gap ( Q π h , t, x ) .

For a proof of this statement and any other made in this work, see Appendix B. Our next result establishes that W p action gaps of ζ π h , like action gaps of Q π h , vanishes for a large class of MDPs.

Theorem 3.5. If r and f are bounded, then lim h ↓ 0 W p ( ζ π h ( t, x, a ) , η π ( t, x )) = 0 , for all ( t, x, a ) ∈ T × X × A ; hence, lim h ↓ 0 distgap p ( ζ π h , t, x ) = 0 .

While Theorem 3.5 shows that the W p distance between ζ π h ( t, x, a ) and η π ( t, x ) (and the W p action gap of ζ π h at ( t, x ) ∈ T × X ) does indeed vanish as h decreases, it does not identify the rate at which it does so. Our next two theorems establish this rate.

Theorem 3.6. MDPs and policies exist in and under which, for all ( t, x, a ) ∈ T × X × A , we have that W p ( ζ π h ( t, x, a ) , η π ( t, x )) ≳ h 1 / 2 and distgap p ( ζ π h , t, x ) ≳ h 1 / 2 .

Finally, we prove that for a large class of MDPs (different from but overlapping with the class of MDPs captured in Theorem 3.5), the lower bound found in Theorem 3.6 is an upper bound.

Theorem 3.7. If r is Lipschitz in state, uniformly in time, f is Lipschitz, and T < ∞ , then W p ( ζ π h ( t, x, a ) , η π ( t, x )) ≲ h 1 / 2 , for all ( t, x, a ) ∈ T × X × A ; hence, distgap p ( ζ π h , t, x ) ≲ h 1 / 2 .

Theorems 3.6 and 3.7 demonstrate that the W p distance between ζ π h ( t, x, a ) and η π ( t, x ) and the distance between Q π h ( t, x, a ) and V π ( t, x ) are of different orders in terms of h . Thus, we see that distgap p ( ζ π h , t, x ) and gap ( Q π h , t, x ) in stochastic MDPs are fundamentally different.

4 Distributional Superiority

In this section, we introduce a probabilistic generalization of the advantage. We define this random variable-which we call the superiority and denote by S π h -via a pair of axioms.

A natural construction of the superiority at ( t, x, a ) is given by Z π h ( t, x, a ) -G π ( t, x ) . The law of this difference, however, depends on the joint law of ( Z π h ( t, x, a ) , G π ( t, x )) , which is unobservable in practice and ill-defined (cf. Section 2.1.2). Yet, the set of all possible laws of this difference is easily characterized; it is the set of coupled difference representations of ζ π h ( t, x, a ) and η π ( t, x ) .

Definition 4.1. Let µ, ν ∈ P ( R ) . A coupled difference representation (CDR) ψ ∈ P ( R ) of µ and ν takes the form ψ = ∆ # κ where κ ∈ C ( µ, ν ) and ∆ : R 2 → R is given by ∆( z, w ) := z -w . The set of all coupled difference representations of µ and ν will be denoted by D ( µ, ν ) .

Our first axiom places the superiority's law in this set, D ( ζ π h ( t, x, a ) , η π ( t, x )) .

Axiom 1. The law of S π h ( t, x, a ) is a coupled difference representation of ζ π h ( t, x, a ) and η π ( t, x ) .

Our second axiom encodes a type of consistency for deterministic policy behavior.

Axiom 2. S π h ( t, x, a ) is deterministic whenever π is ( h, a ) -persistent at time t .

To see how Axiom 2 encodes a notion of deterministic consistency, first consider its discrete-time analogue: the superiority at ( x, a ) for a policy π is deterministic if π at x deterministically chooses a . In this situation, our a -following agent makes the same choices as a π -following agent-both take action a in state x initially and then follow π thereafter-, and we posit that the superiority should not be random. The continuous-time analogue of the situation just described occurs precisely when a policy π is ( h, a ) -persistent at starting time t . Given a starting time t and state x , an agent that chooses action a between t and t + h and then follows π is following the ( h, a ) -persistent modification of π at

time t . By definition, they make the same choices as a π -following agent when π is ( h, a ) -persistent starting at time t . Axiom 2 stipulates that, in this case, S π h ( t, x, a ) should be deterministic.

By construction, if ψ π h ( t, x, a ) ∈ D ( ζ π h ( t, x, a ) , η π ( t, x )) , then its mean is Q π h ( t, x, a ) -V π ( t, x ) (see Appendix B.2 for a proof of this claim). Axiom 2 then says that any determining coupling κ π h ( t, x, a ) when π is ( h, a ) -persistent at time t must be such that ∆ # κ π h ( t, x, a ) = δ 0 . 8 In particular, Axiom 2 nontrivially restricts D ( ζ π h ( t, x, a ) , η π ( t, x )) .

Example 4.2. Let ι π h ( t, x, a ) := ∆ # κ π h ( t, x, a ) for κ π h ( t, x, a ) = ζ π h ( t, x, a ) ⊗ η π ( t, x ) . If π is ( h, a ) -persistent at time t , then Var ( ι π h ( t, x, a )) = 2 Var ( η π ( t, x )) . This variance is 0 only when π 's return is deterministic. Hence, ι π h ( t, x, a ) may be nontrivial even when conditioning on a reflects the policy's behavior exactly. We posit, via Axiom 2, that this should be prohibited.

In fact, Axiom 2 determines a single coupling, if we want a consistent choice across all time-stateaction triplets and all MDPs.

Theorem 4.3. Let κ ∈ C ( µ, µ ) for some µ ∈ P ( R ) . The push-forward of κ by ∆ is the delta at zero, ∆ # κ = δ 0 , if and only if κ is a W p -optimal coupling, for some p ∈ [1 , ∞ ) . Moreover, there is only one such coupling. It is given by κ µ := ( id , id ) # µ or, equivalently, κ µ := ( F -1 µ , F -1 µ ) # U (0 , 1) . Here U (0 , 1) is the uniform distribution on [0 , 1] .

The second definition of κ µ corresponds, more generally, to the W p -optimal coupling, for all p ≥ 1 , of µ and ν given by κ µ,ν := ( F -1 µ , F -1 ν ) # U (0 , 1) . As ∆ # κ µ,ν 's quanitle function is F -1 µ -F -1 ν , we have in hand everything we need to define the superiority distribution (via its quantile function).

Definition 4.4. The superiority distribution ψ π h at ( t, x, a ) ∈ T × X × A is

As ψ π h ( t, x, a ) has the smallest possible central absolute p th moments among all CDRs of ζ π h ( t, x, a ) and η π ( t, x ) , heuristically, it captures more of the individual features of both return distributions than other such CDRs (like ι π h ( t, x, a ) in Example 4.2). We illustrate this by example in Figure 4.1.

4.1 The Rescaled Superiority Distribution

From Section 3, we know that the W p distance between ζ π h ( t, x, a ) and η π ( t, x ) , for every p ≥ 1 , vanishes as h vanishes. Moreover, we know the rate at which this distance disappears. As a result, by construction, the central absolute p th moments of ψ π h ( t, x, a ) collapse as h collapses, and, for a large class of MDPs, we understand the rate at which these moments collapse. More generally and precisely, if we consider the q -rescaled superiority distribution defined by

we see that Theorems 3.6 and 3.7 translate to the follow statements on W p action gaps of ψ π h ; q :

Theorem 4.5. MDPs and policies exist satisfying Assumptions 2.2 and 2.3 in and under which, for all ( t, x ) ∈ T × X , we have that distgap p ( ψ π h ; q , t, x ) ≳ h 1 / 2 -q .

Theorem 4.6. Under Assumptions 2.2 and 2.3, if r is Lipschitz in state, uniformly in time, f is Lipschitz, and T < ∞ , then distgap p ( ψ π h ; q , t, x ) ≲ h 1 / 2 -q , for all ( t, x ) ∈ T × X .

These two theorems tell us how to preserves the W p action gaps of q -rescaled superiority distributions (as a function of h ). They identify q = 1 / 2 . For q < 1 / 2 , W p action gaps vanish as h vanishes. Whereas for q > 1 / 2 , W p action gaps blow up as h vanishes. These behaviors are undesirable. When

q < 1 / 2 , the influence of an action on an agent's superiority becomes indistinguishable from any other action. For q > 1 / 2 , ever larger sample sizes are need to obtain any statistical estimate of an agent's superiority with the same level of accuracy. These scenarios are untenable.

Another consideration regarding rescalings of ψ π h is whether they upend rankings of actions determined by some given measure of utility. This would be counterproductive. In DRL, agents often use distortion risk measures [1] to rank actions ([10, 9, 22, 4, 17]).

Definition 4.7. Given β ∈ P ([0 , 1]) , the distortion risk measure ρ β : P ( R ) → R is defined by ρ β ( µ ) := ⟨ β, F -1 µ ⟩ ; on µ ∈ P ( R ) , its value is given by the integral of F -1 µ with respect to β .

A family of ρ β is the α -conditional value-at-risk measures ( α -CVaR) [29], where β α = U (0 , α ) for α ∈ (0 , 1] ; α = 1 is the expected-value utility measure. Crucially, ψ π h ; q preserves ρ β -valued utility.

Theorem 4.8. Let ρ β be a distortion risk measure, q ≥ 0 , and h > 0 . If ρ β ( η π ( t, x )) < ∞ , then arg max a ∈ A ρ β ( ψ π h ; q ( t, x, a )) = arg max a ∈ A ρ β ( ζ π h ( t, x, a )) .

In turn, the 1 / 2 -rescaled superiority distribution is not only W p action gap preserving but matches ζ π h in its greedy choice of action as measured by a distortion risk measure.

4.2 Algorithmic Considerations

We now turn to building DRL algorithms based on our theory. Our algorithms leverage the quantile TD-learning framework [10] to learn ρ β -greedy policies, for a given distortion risk measure ρ β , just as DAU [34] leverages the Q -learning framework to learn greedy policies. Full pseudocode and implementation details are given in Appendix C.

At the heart of our algorithms is an equality of quantile functions, which holds by construction,

Indeed, given η and ψ h ; q , as models of η π and ψ π h ; q respectively, equation (4.1) justifies the application of quantile TD-learning to ζ h , as a model for ζ π h , defined via the quantile function

That said, we cannot realize quantile TD-learning without defining predictions and bootstrap targets in terms of m -quantile representations 9 [10, 4] of ζ h , via those of η and ψ h ; q .

While we may freely parameterize the m -quantile representation of η with a neural network (with interface) θ : T × X → R m , we have to be careful when parameterizing the m -quantile representation of ψ h ; q . Given a neural network ϕ : T × X × A → R m , we set

This ensures we identify a ρ β -greedy policy; it is 0 at the ρ β -greedy action a ⋆ (cf. [34, Eq. 27]).

With appropriate parameterized m -quantile representations of η and ψ h ; q in hand, we derive our predictions and bootstrap targets. By (4.2), recalling θ and (4.3), we compute our predictions via

By Wiltzer [40], as X π | h,a,t t = x and X π | h,a,t t + h = X a t + h , observe that

where E [ | Y h | p ] = o ( h ) 10 for all p ∈ N . So upon getting a sample state/realization x t + h of X a t + h , as in [30], we compute our bootstrap targets via

In summary, the predictions (4.4) and the bootstrap targets (4.5) together characterize a family of QR-DQN-based algorithms called DSUP( q ), whose core update is outlined in Algorithm 1.

Algorithm 1 DSUP( q ) Update

One theoretical drawback of DSUP( q ) for mean-return control is that the mean of the q -rescaled superiority distribution is O (1) only when q = 1 , by (2.6) and (2.7). Thus, we propose modeling A π h simultaneously. This yields a novel form of a two-timescale approach to value-based RL (see, e.g., [8]). In particular, we estimate ϑ π h ; q defined by

Wecall ϑ π h ; q the advantage-shifted q -rescaled superiority . Note that its mean is A π h , which is O (1) . To realize this, we approximate A π h using DAU and employ parameter sharing between the approximators of A π h and ψ π h ; q . We call this family of algorithms DAU+DSUP( q ). We note that A π h is used only for increasing action gaps; it does not change the training loss for η and ψ h ; q .

5 Simulations

The empirical work herein is two-fold in nature: illustrative and comparative. First, we simulate an example that illustrates Theorems 3.6/4.5 and Theorem 3.7/4.6 and their consequences. Second, in an option-trading environment, we compare the performance of ψ π h -based agent(s) against QR-DQN [10] and DAU [34] in the risk-neutral setting and against QR-DQN in a risk-sensitive setting.

5.1 The Rescaled Superiority Distribution Revisited

Consider an MDP with time horizon 10 , a two element action space, 0 and 1 -when action 1 is executed, the system follows 1 -dimensional Brownian dynamics with a constant drift of 10 , and otherwise, the state is fixed-, a reward that equals the agent's signed distance to 0 , and a trivial terminal reward. We estimate four distributions at ( t, x, a ) = (0 , 0 , 1) for the policy that always selects 0 . Figure 5.1 shows these estimated distributions for a sample of frequencies (kHz), ω = 1 / h .

ζ

π

h

=

ψ

π

h

ψ

π

h

;1

ω

ψ

π

h

;

1

/

2

ϑ

π

h

;

1

/

2

First (from the left), we see that ψ π h collapses to δ 0 , as h tends to 0 . Thus, accurate action ranking distributional or otherwise becomes impossible in the vanishing h limit. Second, we see that rescaling by h produces distributions with O (1) mean but infinite non-mean statistics in the vanishing h limit. Here the O (1) means are imperceptible in face of the large variances. So while this rescaling permits ranking actions by action values, it does so at the expense of producing high-variance distributions.

Third, we see that rescaling by h 1 / 2 yields distributions with O (1) non-mean statistics but vanishingly small means, O ( h 1 / 2 ) . Hence, this rescaling permits ranking actions by non-mean statistics, even if action values again becomes indistinguishable in the vanishing h limit. That said, the vanishing rate of the means here is slower than when no rescaling is considered, O ( h ) . Fourth, we see that rescaling by h 1 / 2 and then shifting it by (1 -h 1 / 2 ) A π h produces distributions with O (1) mean and non-mean statistics. In turn, this two-timescale approach permits ranking actions by either action values or non-mean statistics (but not both by Theorem 4.8). However, the mean estimates here are inaccurate and imprecise-rather than uniformly being 100 , they oscillate substantially.

In risk-neutral control, we are left with a number of questions. What effect do the high variance distributions in DAU/DSUP( 1 ) have on performance? What effect do the O ( h 1 / 2 ) means have on the performance of DSUP( 1 / 2 )? What effect does the instability of the mean estimates in DAU+DSUP( 1 / 2 ) have on performance? In Section 5.2, we begin to answer these questions and others by testing our superiority-based algorithms against appropriate benchmarks in an option-trading environment.

5.2 High-Frequency Option Trading

The option-trading environment in which we run our comparative experiments is a commonly used benchmark (see, e.g., [22, 17]). We use an Euler-Maruyama discretization scheme [23] at high resolution to simulate high-frequency trading. Returns are averaged over 10 seeds and 10 different dynamics models (corresponding to data from different stocks). Additionally, following [22], we use disjoint datasets to estimate the dynamics parameters for simulation during training and evaluation. 11

First, we consider the risk-neutral setting. Here we compare QR-DQN, DAU, and three algorithms based on the q -rescaled superiority distribution with q = 1 , 1 / 2 : DSUP( 1 ), DAU+DSUP( 1 / 2 ), and DSUP( 1 / 2 ). Figure 5.2 summarizes their performance at a sample of frequencies (Hz).

We see that DSUP( 1 / 2 ) is not only the most consistent performer, but outperforms every competitor at all but the two lowest frequencies. Even then, its performance is very close to the best performer. We also see that DAU+DSUP( 1 / 2 )'s preservation of both action gaps and W p action gaps does not lead to the strongest performance. In particular, its performance is inconsistent and sometimes poor. We believe this is because the tested frequencies are low enough that DSUP( 1 / 2 ) maintains large enough action gaps to learn performant policies, but high enough that the variances of the distributions underlying A π h cause estimation difficulty. Indeed, the three methods that estimate A π h (explicitly in DAU and DAU+DSUP( 1 / 2 ) or implicitly in DSUP( 1 )) exhibit almost identical behavior.

Our results highlight a dichotomy in existing (ours included) methods for value-based, high-frequency, risk-neutral control. They can either maintain O (1) expected return estimates or O (1) return variance estimates, but not both. We observe better performance in estimating small means from O (1) variance distributions than in estimating O (1) means from receipricolly large variance distributions.

To qualitatively illustrate the appeal of the 1 / 2 -rescaled superiority, Figure 5.3 presents examples of learned action-conditioned distributions used by DSUP( 1 / 2 ) and QR-DQN agents to make decisions.

In this environment, action 1 taken in the start state terminates the episode, yielding the smallest return, 0 , making this action inferior to its alternative action, 1 . We see that DSUP( 1 / 2 ) infers this fact. QR-DQN, on the other hand, has difficulty distinguishing these actions. This is because the 1 / 2 -rescaled superiority preserves W p action gaps, while ζ π h does not.

Second, we consider a risk-sensitive set- ting. Here we compare QR-DQN and DSUP( 1 / 2 ) using α -CVaR for greedy action selection. We do this because Theorem 4.8 does not hold with ϑ π h ; q , and preserving means is less critical in risksensitive control than it is in risk-neutral control. Figure 5.4 depicts our results at ω = 35Hz (see Appendix D for results across a range of ω ).

Again, we see that DSUP( 1 / 2 ) is conclusively the best performer.

6 Related Work

Notions of action gap and ranking have long been of interest in RL (see, e.g., [11]). Action gaps are related to sample complexity in RL-indeed, instance-dependent sample complexity rates are inversely proportional to the divergence between action-conditioned return distributions ([13, 19, 37]). Bellemare et al. [5] argue for the consideration of alternatives to the Bellman operator that explicitly devalue suboptimal actions, and they show that Baird's AL [2] operator falls within this class of operators. On the other hand, Schaul et al. [31] implicitly question Bellemare et al.'s position. They demonstrate that stochastic gradient updates in deep value-based RL algorithms induce frequent changes in relative action values, which in turn is a mechanism for exploration.

The advantage function is commonplace in RL (see, e.g., [39, 32, 27, 35, 24]). In [24], Mésnard et al. employ a distributional critic that is closely related to our (unscaled) distributional superiority. Their choice of critic stems from a desire to minimize variance. We note that the distributional superiority is a posteriori characterized as a minimal variance coupled difference representation of action-conditioned return distributions and policy-induced return distributions.

Lastly, DRL in continuous-time MDPs is in its infancy. There are only three works to mention. Wiltzer et al. [40, 41] give a characterization of return distributions for policy evaluation, and Halperin [14] studies algorithms for control. That said, neither work considers distributional notions of action gaps or advantages. Moreover, Halperin does not consider any of the challenges of estimating the influence of actions in high decision frequency settings.

7 Conclusion

We establish that DRL agents are sensitive to decision frequency through analysis and simulation. In experiments, DSUP( 1 / 2 ) learns well-performing policies across a range of high decision frequencies, unlike prior approaches. DSUP( 1 ) and DAU+DSUP( 1 / 2 ) are less robust. Given our analysis, the performance of DSUP( 1 ) is expected. Building an alternate algorithm to DAU+DSUP( 1 / 2 ) that is both tailored to risk-neutral control and robust to h is an important avenue for future work.

Acknowledgments and Disclosure of Funding

The authors are very grateful to Yunhao Tang for fruitful correspondence about distributional analogues to the advantage. Additionally, we thank Mark Rowland, Jesse Farebrother, Tyler Kastner, Pierluca D'Oro, Nate Rahn, and Arnav Jain for helpful discussions. HW was supported by the Fonds de Recherche du Québec and the National Sciences and Engineering Research Council of Canada (NSERC). MGB was supported by the Canada CIFAR AI Chair program and NSERC. This work was supported in part by DARPA HR0011-23-9-0050 to PS. YJ was supported by in part by NSF Grant 2243869. This research was enabled in part by support provided by Calcul Québec, the Digital Research Alliance of Canada ( alliancecan.ca ), and the compute resources provided by Mila ( mila.quebec ).

References

Carlo Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance , 26(7):1505-1518, July 2002.

L. Baird. Reinforcement Learning Through Gradient Descent . PhD thesis, May 1999.

Marc G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature , 588(7836):77-82, December 2020.

Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning . The MIT Press, May 2023.

Marc G Bellemare, Georg Ostrovski, Arthur Guez, Philip Thomas, and Rémi Munos. Increasing the action gap: New operators for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30, 2016.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

Gerard Brunick and Steven Shreve. Mimicking an Itô process by a solution of a stochastic differential equation. The Annals of Applied Probability , 23(4):1584-1628, August 2013.

Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. In International Conference on Learning Representations , 2018.

Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning , pages 1096-1105. PMLR, 2018.

Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional Reinforcement Learning with Quantile Regression. In AAAI , 2017.

Amir-massoud Farahmand. Action-gap phenomenon in reinforcement learning. Advances in Neural Information Processing Systems , 24, 2011.

Wendell H Fleming and Halil Mete Soner. Controlled Markov processes and viscosity solutions , volume 25. Springer Science & Business Media, 2006.

Todd L. Graves and Tze Leung Lai. Asymptotically Efficient Adaptive Choice of Control Laws in Controlled Markov Chains. SIAM Journal on Control and Optimization , 35(3):715-743, May 1997.

Igor Halperin. Distributional offline continuous-time reinforcement learning with neural physicsinformed pdes (sciphy rl for doctr-l). Neural Computing and Applications , 36(9):4643-4659, 2024.

Yanwei Jia and Xun Yu Zhou. Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms. Journal of Machine Learning Research , 23(275):1-50, 2022.

Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. Journal of Machine Learning Research , 24(161):1-61, 2023.

Tyler Kastner, Murat A Erdogdu, and Amir-massoud Farahmand. Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2014.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Leemon C. Baird. Advantage Updating. Technical report, Defense Technical Information Center, Fort Belvoir, V A, November 1993.

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR) , 2016.

Shiau Hong Lim and Ilyas Malik. Distributional Reinforcement Learning for Risk-Sensitive Policies. In NeurIPS , October 2022.

Gisiro Maruyama. Continuous markov processes and stochastic equations. Rendiconti del Circolo Matematico di Palermo , 4:48-90, 1955.

Thomas Mesnard, Wenqi Chen, Alaa Saade, Yunhao Tang, Mark Rowland, Theophane Weber, Clare Lyle, Audrunas Gruslys, Michal Valko, Will Dabney, et al. Quantile credit assignment. In International Conference on Machine Learning , pages 24517-24531. PMLR, 2023.

Rémi Munos and Paul Bourgine. Reinforcement Learning for Continuous Stochastic Control Problems. In Advances in Neural Information Processing Systems (NeurIPS) , 1997.

Bernt Oksendal. Stochastic differential equations: an introduction with applications . Springer Science & Business Media, 2013.

Hsiao-Ru Pan, Nico Gürtler, Alexander Neitz, and Bernhard Schölkopf. Direct Advantage Estimation. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Simon Ramstedt and Christopher J. Pal. Real-time reinforcement learning. In Advances in Neural Information Processing Systems , 2019.

R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. Journal of Banking & Finance , 26(7):1443-1471, 2002.

Mark Rowland, Rémi Munos, Mohammad Gheshlaghi Azar, Yunhao Tang, Georg Ostrovski, Anna Harutyunyan, Karl Tuyls, Marc G. Bellemare, and Will Dabney. An Analysis of Quantile Temporal-Difference Learning. Journal of Machine Learning Research (JMLR) , 25:1-47, 2023.

Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn. Advances in Neural Information Processing Systems , 35:2537-2549, 2022.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. CoRR , abs/1707.06347, 2017.

Daniel W. Stroock and S. R. Srinivasa Varadhan. Multidimensional Diffusion Processes . Springer, 2006.

Corentin Tallec, Léonard Blier, and Yann Ollivier. Making Deep Q-learning methods robust to time discretization. In International Conference on Machine Learning (ICML) , 2019.

Yunhao Tang, Rémi Munos, Mark Rowland, and Michal Valko. VA-learning as a more efficient alternative to Q-learning. In International Conference on Machine Learning (ICML) , 2023.

Cédric Villani et al. Optimal transport: old and new , volume 338. Springer, 2009.

Andrew J. Wagenmaker and Dylan J. Foster. Instance-Optimality in Interactive Decision Making: Toward a Non-Asymptotic Theory. In Proceedings of Thirty Sixth Conference on Learning Theory , pages 1322-1472. PMLR, July 2023.

Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research , 21(198):1-34, 2020.

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling Network Architectures for Deep Reinforcement Learning. In International Conference on Machine Learning (ICML) , 2016.

Harley Wiltzer. On the Evolution of Return Distributions in Continuous-Time Reinforcement Learning . McGill University (Canada), 2021.

Harley Wiltzer, David Meger, and Marc G. Bellemare. Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning. In International Conference on Machine Learning (ICML) , 2022.

Hanyang Zhao, Wenpin Tang, and David Yao. Policy optimization for continuous reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

A Formalism of Continuous-Time RL Controlled Markov Processes

Expected-value RL is a data-driven approach to solving the (classic) optimal control problem: find an action (control) process ( A s ) s ≥ t and an associated state process (then determined by the environment) ( X s ) s ≥ t with X t = x , for a given t ≥ 0 , that maximize the expected return earned by following the state-action process ( X s , A s ) s ≥ t . In particular, RL agents search the space of state-action processes via policies π : T × X → P ( A ) . Policies prescribe the conditional probabilities of the laws of state-action processes. Indeed, ( X s , A s ) s ≥ t is the state-action process of an agent following π if and only if, for each s ≥ t , the set { π ( · | s, x ) } x ∈ X is the set of conditional probabilities of law (( X s , A s )) with respect to law ( X s ) .

Continuous-time RL is a data-driven approach to stochastic optimal control. Whence, environmental dynamics are assumed to arise from an action-parameterized family of SDEs determined by a drift b : T × X × A → R n and diffusion σ : T × X × A → R n × n . 12 Thus, the goal of expected-value RL (and stochastic optimal control) is to find an expected-return maximizing state-action process among state-action processes ( X s , A s ) s ≥ t that satisfy

We note that the MDPs defined in Section 2 have equivalent formulations in terms of transition kernel, exactly as they formulated in discrete-time RL. We refer the reader to [33] for an in-depth discussion regarding this fact.

A.1 Justification of Random Returns

Given the above formalism, the 'true' distribution of returns of an agent following a policy π is the law of

where ( X s , A s ) s ≥ t is a state-action process associated to π and solves (A.1). That said, by the definition of π ,

where b π and σ π are exactly the coefficients defined in (2.3). Hence, by [7], provided that b π and σ π are regular enough to guarantee that (2.2) is well-posed in law, the processes X π = ( X π s ) s ≥ t and X = ( X s ) s ≥ t are equal in law. 13 Here ( X π s ) s ≥ t satisfies (2.2) with X π t = x . Consequently,

This, formally, justifies G π and Z π h , as defined in (2.4) and (2.5).

A.2 On Assumption 2.3

Here we provide some conditions under which Assumption 2.3 is established. These conditions are presented as additional assumptions. Assumptions A.1 and A.2 are common in stochastic control theory (see, e.g., [12]) and SDE theory in general (see, e.g., [26]). Assumption A.3 is ubiquitous in the continuous-time RL literature [15, 16, 42]. Notably, these conditions together guarantee the existence of transition probabilities for policy-induced state processes arising from (2.2).

Assumption A.1. The coefficients b and σ are uniformly bounded: a finite, positive constant C A. 1 exists such that

Assumption A.2. The matrix σσ ⊤ is uniformly elliptic: a positive, finite constant λ A. 2 exists such that

A consequence of Assumption A.2 is

In other words, σ π is also uniformly elliptic.

Assumption A.3. A finite, positive constant C A. 3 exists for which

where TV is the total variation metric on P ( A ) .

Observe if π satisfies Assumption A.3, then π | h,a,t also satisfies Assumption A.3. Indeed,

since

and π | h,a,t ( · | s, x ) = π ( · | s, x ) for all s ∈ T \ [ t, t + h ) .

Proposition A.4. If Assumptions 2.2, A.1, and A.2 hold and π satisfies Assumption A.3, then Assumption 2.3 holds.

Proof. Observe that

by Assumptions 2.2, A.1, and A.3. Here we have also used Kantorovich duality to computed TV and invoke Assumption A.3.

Let λ be an eigenvalue of σ π ( t, x ) -σ π ( t, y ) with with unit eigenvector v . Observe that

Hence, by (A.2),

In turn, by Assumptions 2.2, A.1, and A.3, as done to prove that b π was Lipschitz above,

Assumption 2.3 follows, since λ was an arbitrary eigenvalue and all norms on finite dimensional spaces are equivalent.

We conclude this section with one final fact: under Assumption 2.2, the policy-averaged coefficient (2.3) have linear growth. Indeed,

and

A.3 Action-Independent Rewards

In this paper, we assume that the rewards do not depend on actions. This is a theoretical limitation of not just our work, but continuous-time DRL in general (see [41, 14]). In the following sections, we discuss the nature of this theoretical limitation. However, many MDPs have action-independent reward functions. For example, MDPs encoding goal-reaching problems, tracking problems, and commodity-trading problems all have action-independent rewards.

A.3.1 Continuous-Time, Expected Return

In continuous-time, expected-value RL, when the reward function r depends on actions, the standard approach to analysis involves considering the averaged reward function r π : T × X → R given by

In continuous-time RL specifically, the averaged reward r π is justified exactly as the coefficients b π and σ π are justified. However, the 'true' return distribution is not equal to the law of

To see this, it suffices to consider an MDP with a single state x . In this case, the expression in (A.3) is deterministic. However, if π is nondeterministic and r is dependent on actions, then the 'true' return distribution is nondeterministic. Hence, the law of the expression in (A.3) cannot be the 'true' return distribution associated to π .

A.3.2 Discrete-Time, Random Return

In discrete-time RL, the distribution of returns given an action-dependent reward is analyzed through the state-action process induced by a policy. This processes is defined by extending the actionparameterized family of transition probability kernels on X , which define the dynamics of a given MDP, to a single transition probability kernel on X × A . In the time-homogeneous setting, for instance, with transition kernels { P (d y | x, a ) } a ∈ A , this amounts to constructing

provided that map y ↦→ π ( E | y ) is measurable for all y ∈ X . In continuous-time environments, such a constructing has yet to be discovered.

We note that trying to analogously extend the action-parameterized family of transition semigroups on X , which define the dynamics of a given time-homogeneous MDP in continuous time, to a single transition semigroup on X × A by defining

̸

where P t (d y | x ) is a transition semigroup, may fail to satisfy the Chapman-Kolmogorov identity. Indeed, suppose A has two elements and X = R . Let π (d a | x ) be the uniform measure on A for all x ∈ X . If P t (d y | x, a δ ) = δ x + t (d y ) and P t (d y | x, a g ) = (2 π ) -1 / 2 exp( -| y -x | 2 / 2 t ) d y , then Chapman-Kolmogorov identity fails, for example, on any tuple ( s, t, x, a δ , E × F ) where E ⊂ X is open, x + t + s / ∈ E , and π ( F ) = 0 . On one hand,

On the other hand,

So

̸

At present, the question of how to generate a well-defined (even in law) state-action process in any continuous-time MDP framework given a stochastic policy is generally open. Of course, if π is deterministic, then the state-action process is ( X s , π ( s, X s )) s ≥ t . If b and σ are Lipschitz in state and action, uniformly in time, and π is Lipschitz in state, uniformly in time, then (A.1) with A s = π ( s, X s ) is well-posed.

B Proofs

B.1 The Distributional Action Gap

In this section, we prove the statements made in Section 3.

Before proving any of the statements made in Section 3, we recall an identity that relates the W p distance between two probability measures µ and ν and the absolute central p th moments of the differences of random variables distributed according to µ and ν :

This identity will be used a number of times, including in the proof of Section 3's first result, which we restate here for the readers convenience.

Proposition 3.4. For all ( t, x ) ∈ T × X , we have that distgap p ( ζ π h , t, x ) ≥ gap ( Q π h , t, x ) .

Proof. Let ( Z 1 , Z 2 ) be any random vector with such that law ( Z i ) = ζ π h ( t, x, a i ) for i = 1 , 2 . Then, by Jensen's inequality,

Hence, since ( Z 1 , Z 2 ) was arbitrary, by (B.1),

̸

Finally, taking the minimum over pairs of actions ( a 1 , a 2 ) such that a 1 = a 2 concludes the proof.

Now we move on to the proofs of Theorems 3.5 and 3.7. We defer the proof of Theorem 3.6 until after the proof of Theorem 3.7 as the proofs of Theorems 3.5 and 3.7 are similar. For clarity's sake, we first prove a collection of lemmas.

Lemma B.1. Under Assumption 2.2, let ( X a s ) s ≥ t be the unique strong solution to (2.2) with X a t = x P -a.s. Then, for all q ≥ 1 and for all s ≥ t ,

where C q = 4 2 q -1

Proof. Let C q = 4 2 q -1 . By Jensen's inequality and Itô's isometry, observe that

Thus, the lemma follows after taking expectation and applying Gronwall's inequality.

Lemma B.2. Under Assumptions 2.2 and 2.3, let ( X · s ) s ≥ t with · ∈ { π, π | h,a,t } be the unique strong solution to (2.2) with X · t = x P -a.s. Then, for all s ≤ t + h ,

where C is some finite positive constant depending on q , C 2 . 2 , and C 2 . 3 .

Proof. Let C q = 8 2 q -1 . Note that X π | h,a,t s ′ = X a s ′ P -a.s. for all s ′ ≤ t + h , by the definition of π | h,a,t and the uniqueness of strong solutions to (2.1). So, by Jensen's inequality and Itô's isometry, observe that

Thus, after taking expectation, we deduce that

And so, the lemma follows after applying Lemma B.1 and Gronwall's inequality.

Lemma B.3. Under Assumptions 2.2 and 2.3, let ( X · s ) s ≥ t with · ∈ { π, π | h,a,t } be the unique strong solution to (2.2) with X · t = x P -a.s. Then, for all s > t + h ,

where C is some finite positive constant depending on q , C 2 . 2 , and C 2 . 3 .

Proof. Let C q = 3 2 q -1 . Observe that

So, as π | h,a,t ( · | s ′ , y ) = π ( · | s ′ , y ) for all ( s ′ , y ) ∈ T \ [ t, t + h ) × X , by Jensen's inequality and Itô's isometry,

Thus, after taking expectation, applying Gronwall's inequality, and considering Lemma B.2, the lemma follows.

One consequence of Lemmas B.2 and B.3 is

for all s ∈ T and for all p ∈ [1 , ∞ ) . And so, if f is bounded, then f ( X π | h,a,t T ) -f ( X π T ) is bounded and converges to zero P -a.s. as h converges to zero. The dominated convergence theorem implies that

Similarly, we see that the functions g h : T → [0 , ∞ ) defined by

are uniformly bounded (in h ) and converge to zero as h ↓ 0 for every s ∈ T . Hence, by the dominated convergence theorem, again,

where Γ(d s ) := ( γ s -t /C T,t,γ ) d s for C T,t,γ := ( γ T -t -1) / log γ . We now prove Theorem 3.5.

Theorem 3.5. If r and f are bounded, then lim h ↓ 0 W p ( ζ π h ( t, x, a ) , η π ( t, x )) = 0 , for all ( t, x, a ) ∈ T × X × A ; hence, lim h ↓ 0 distgap p ( ζ π h , t, x ) = 0 .

Proof. Observe that

̸

Thus, as claimed, it suffices to show that

for all ( t, x, a ) ∈ T × X × A . By (B.1), it suffices to show that

for all ( t, x, a ) ∈ T × X × A .

Since ( s, ω ) ↦→ X · s ( ω ) is measurable for · ∈ { π, π | h,a,t } , by Jensen's inequality and Fubini's theorem, we see that

In turn, (B.4) follows from (B.2) and (B.3).

If T < ∞ and h < 1 , the inequalities in the statements of Lemmas B.2 and B.3 yield the following inequality: for all p ∈ [1 , ∞ ) ,

for some finite positive constant C (B.6) depending on p , | x | , t , T , C 2 . 2 , and C 2 . 3 . With this inequality in hand, we now restate and prove Theorem 3.7.

Theorem 3.7. If r is Lipschitz in state, uniformly in time, f is Lipschitz, and T < ∞ , then W p ( ζ π h ( t, x, a ) , η π ( t, x )) ≲ h 1 / 2 , for all ( t, x, a ) ∈ T × X × A ; hence, distgap p ( ζ π h , t, x ) ≲ h 1 / 2 .

Proof. Arguing as in the proof of Theorem 3.5, we see that

with

This is (B.5). In turn, by (B.6) and that r is Lipschitz in space, uniformly in time and f is Lipschitz, we deduce that

as desired, where C r and C f are the Lipschitz constants of r and f respectively.

Recall r : T × X → R is Lipschitz in state, uniformly in time if a finite positive constant C r exists such that

As promised, we now prove Theorem 3.6.

Theorem 3.6. MDPs and policies exist in and under which, for all ( t, x, a ) ∈ T × X × A , we have that W p ( ζ π h ( t, x, a ) , η π ( t, x )) ≳ h 1 / 2 and distgap p ( ζ π h , t, x ) ≳ h 1 / 2 .

̸

̸

Proof. Let X = R and A = { 0 , 1 } ; set b ≡ 0 and σ = 1 { a =1 } ; for all ( s, y ) ∈ T × X , let r ( s, y ) = y ; and set f ≡ 0 . In words, our action space has two elements, and when action 1 is executed, the system follows Brownian dynamics, and otherwise, the state is fixed. Now consider the policy π which always selects the action 0 : π ( · | s, y ) = δ 0 , for all ( s, y ) ∈ T × X .

Case 1: γ = 1 and T < ∞ . Observe that

where ( ˜ B s ) s ≥ 0 is a Brownian motion. Hence, Z π h ( t, x, 1) -Z π h ( t, x, 0) is equal in law to the sum of two zero mean Gaussian random variables with variances σ 2 1 = h 3 / 3 and σ 2 2 = ( T -t -h ) 2 h respectively. And so, it is also Gaussian, and its variance is σ 2 h = σ 2 1 + σ 2 2 + 2 cσ 1 σ 2 for some -1 ≤ c ≤ 1 . In particular, σ p h ≳ h p/ 2 . Recall that central absolute p th moment of a Gaussian random variable is proportional to its standard deviation to the power p , for all p ≥ 1 . Therefore, by (B.1), we deduce that distgap p ( ζ π h , t, x ) ≳ h 1 / 2 , for h < 1 , as desired.

Case 2: T ∈ [0 , ∞ ] and γ ∈ (0 , 1) . Observe that

for some Brownian motion ( ˜ B s ) s ≥ 0 . We claim that N ( h ) is a mean zero Gaussian random variable with variance σ 2 h ≈ h 3 / 3 . Then the concluding argument in the proof of Case 1 also concludes the proof of this case.

To prove our claim, first note that

As

we see that N ( h ) has the claimed statistics. Second, observe that N ( h ) = lim n →∞ N n ( h ) where

(This is simply a Riemann sum approximation of the integral that defines N ( h ) .) As the sum of any finite number of Gaussian random variables is a Gaussian random variable, N n ( h ) is Gaussian. Furthermore, as the limit of a sequence of Gaussian random variables whose sequences of means and variances converge (to finite values) is Gaussian, N ( h ) is Gaussian, which proves our claim.

B.2 Distributional Superiority

In this section, we prove the statements and claims made in Section 4.

First, we prove that the mean of every ψ π h ( t, x, a ) ∈ D ( ζ π h ( t, x, a ) , η π ( t, x )) is Q π h ( t, x, a ) -V π ( t, x ) .

Lemma B.4. If ψ π h ( t, x, a ) ∈ D ( ζ π h ( t, x, a ) , η π ( t, x )) , then its mean is Q π h ( t, x, a ) -V π ( t, x ) .

Proof. If κ π h ( t, x, a ) be such that ∆ # κ π h ( t, x, a ) = ψ π h ( t, x, a ) , then

as desired.

Second, we prove Theorem 4.3.

Theorem 4.3. Let κ ∈ C ( µ, µ ) for some µ ∈ P ( R ) . The push-forward of κ by ∆ is the delta at zero, ∆ # κ = δ 0 , if and only if κ is a W p -optimal coupling, for some p ∈ [1 , ∞ ) . Moreover, there is only one such coupling. It is given by κ µ := ( id , id ) # µ or, equivalently, κ µ := ( F -1 µ , F -1 µ ) # U (0 , 1) . Here U (0 , 1) is the uniform distribution on [0 , 1] .

First, we establish that there is only one W p optimal coupling between a given µ ∈ P ( R ) and itself, for every p ∈ [1 , ∞ ) .

Lemma B.5. Let µ ∈ P ( R ) . There is only one W p optimal coupling between µ and itself, for every p ∈ [1 , ∞ ) . It is κ µ := ( id , id ) # µ ∈ C ( µ, µ ) .

Proof. Let κ ∈ C ( µ, µ ) , and suppose there exists ϵ > 0 for which

Then

Hence, κ , as considered, is not optimal. Since ϵ was arbitrary, it follows that a W p optimal coupling is concentrated on { z = w } . Therefore, every optimal coupling is of the form ( id , id ) # ν for some ν ∈ P ( R ) . As the marginals of such a coupling are ν and ν , we deduce that ν = µ , as desired.

Proof of Theorem 4.3. Let κ ∈ C ( µ, µ ) be such that ∆ # κ = δ 0 . Observe that

where, again, κ µ = ( id , id ) # µ . Hence, κ = κ µ , by Lemma B.5. On the other hand, since κ µ is concentrated on { z = w } , for any bounded, continuous function g , we see

In turn, ∆ # κ µ = δ 0 , as desired.

Remark B.6. Under the hypotheses of Theorem 3.7, we see that the 1 / 2 -rescaled superiority distributions at any ( t, x, a ) for h ∈ (0 , 1] is a family of probability measures with uniformly bounded second moment. Hence, this family is tight. So, up to subsequences, these rescalings converges to limiting probability measure as h ↓ 0 . An interesting open question, is whether or not these subsequential limits are the same.

Third, we prove Theorems 4.5 and 4.6. The proofs of these theorems are a consequence of the following expression for the W p distance between µ and ν when µ, ν ∈ P ( R ) :

Theorem 4.5. MDPs and policies exist satisfying Assumptions 2.2 and 2.3 in and under which, for all ( t, x ) ∈ T × X , we have that distgap p ( ψ π h ; q , t, x ) ≳ h 1 / 2 -q .

̸

Proof. Recall that F -1 ψ π h ; q ( t,x,a ) = h -q ( F -1 ζ π h ( t,x,a ) -F -1 η π ( t,x ) ) , for all a ∈ A . Hence, for every a 1 = a 2 ,

Thus, taking the p th root of both sides of the above equality, we deduce that

The example presented in Theorem 3.6 is such that W p ( ζ π h ( t, x, a 1 ) , ζ π h ( t, x, a 2 )) ≳ h 1 / 2 . Whence, W p ( ψ π h ; q ( t, x, a 1 ) , ψ π h ; q ( t, x, a 2 )) ≳ h 1 / 2 -q .

Theorem 4.6. Under Assumptions 2.2 and 2.3, if r is Lipschitz in state, uniformly in time, f is Lipschitz, and T < ∞ , then distgap p ( ψ π h ; q , t, x ) ≲ h 1 / 2 -q , for all ( t, x ) ∈ T × X .

The proof of Theorem 4.6 is almost identical to the proof of Theorem 4.5.

Proof. Arguing as in the proof of Theorem 4.5, we see that

Finally, we prove Theorem 4.8

Theorem 4.8. Let ρ β be a distortion risk measure, q ≥ 0 , and h > 0 . If ρ β ( η π ( t, x )) < ∞ , then arg max a ∈ A ρ β ( ψ π h ; q ( t, x, a )) = arg max a ∈ A ρ β ( ζ π h ( t, x, a )) .

Proof. Observe that

Since ρ β ( η π ( t, x )) is independent of a and h > 0 ,

C Algorithms and Pseudocode

Here we discuss methods for policy optimization via distributional superiority. In practice, computers operate at a finite frequency-as such, all policies we consider here will be assumed to apply each action for h units of time, as in the settings of [34, 16].

Before describing the superiority learning algorithms, we first remark on the form of exploration policies used in our approaches. We consider policies of the form

The first argument to such a policy is a vector of 'action-values'. In our case, given a superiority distribution ψ π h ; q ( t, x, · ) , the action values may be ( ρ β ( ψ π h ; q ( t, x, a ))) a ∈ A for a distortion risk measure ρ β . This generalizes the notion of Q -values for distributional learning.

The second argument represents a noise variable, in order to support stochastic policies. As input to our algorithms, we require a probability measure P act on processes ( ϵ t ) t ≥ 0 . This framework generalizes common exploration methods in deep RL. For example, to recover ϵ -greedy policies, P act represent a 2-dimensional white noise, and

Alternatively, one might choose to correlate the action noise. Approaches such as DDPG [21] and DAU present such examples, where action noise evolves over time as an Ornstein-Uhlenbeck process. This can be implemented in the framework above by choosing P act as the distribution of an | A | -dimensional Ornstein-Uhlenbeck process, and defining

In our experiments, we found that ϵ -greedy exploration was sufficient for approximate policy optimization. To keep our implementation closest to the QR-DQN baseline, therefore, our implementations use the definition of π explore from equation (C.1).

The remainder of this section details the implementation of DSUP( q ) and DAU+DSUP( q ) as introduced in Section 4.2. Additionally, we provide source code for our implementations at https://github.com/harwiltz/distributional-superiority .

C.1 Distributional Superiority

Generally, our goal is to learn a ρ β -greedy policy (see Definition 4.7). Since all policies apply actions for h units of time, in order to satisfy Axiom 2, we want

where π : x ↦→ arg max a ρ β ( S π h ( t, x, a )) is the ρ β -greedy policy. As such, following the DAU algorithm of [34], our algorithms will model quantile functions ϕ ( t, x, a ) ∈ R m that aim to satisfy

Our primary algorithmic contribution integrates such a model, with proper superiority rescaling, into the QR-DQN framework of [10] for estimating action-conditioned return distributions. It is outlined in Algorithm 2.

To deal with the increased time-resolution of transitions, [34] modified the step size by a factor of h . More recently, [3] found that subsampling transitions before storing them in the replay buffer was most effective in their high-decision-frequency domain. Thus, we opt for such a strategy here. Rather than only storing every h -1 transitions, we randomly select transitions to store according to independent Bernoulli ( h ) draws in order to avoid the possibility of only capturing cyclic phenomena in the replay buffer. We found that this strategy worked similarly to that of [34], but is far less computationally expensive. Likewise, as h decreases, we extend the number of training interactions by a factor of h -1 . This corresponds to training for a constant amount of time units across decision frequencies, and likewise, a constant number of gradient updates across decision frequencies.

C.2 Two-Timescale Advantage-Shifted Distributional Superiority

An astute reader might recognize that while ψ π h ; q may have nonzero distributional action gap, since h q is asymptotically larger than h for q < 1 , the works of [34, 16] would suggest that the expected action gap under ψ π h ; q should vanish. To account for this, we propose shifting the rescaled superiority quantiles by the advantage function, as estimated e.g. in DAU [34]. It is clear that such a procedure cannot cause the distributional action gap to vanish. The resulting procedure is depicted in Algorithm 3, with the modifications relative to Algorithm 2 highlighted in blue. In practice, we employ a shared feature extractor in the representations of A h and ϕ to reap the representation learning benefits of DRL [4] when approximating the advantage.

C.3 Influence of the Rescaling Factor

The algorithms 2 and 3 are parameterized by a rescaling factor q ∈ (0 , 1] , which is meant to compensate for the collapse of the distributional action gap. Larger values of q correspond to larger compensation. In this work, we argue that the distributional action gap collapses at rate h 1 / 2 , leading to a natural choice of q = 1 / 2 , which theoretically preserves constant order action gaps with respect to the decision frequency. We also test the approach with q = 1 , which corresponds to the well-known scaling rate for preserving expected value action gaps.

For any q > 1 / 2 , the distributional action gap theoretically grows without bound as h ↓ 0 , leading to distributional estimates with arbitrarily large variance. On the other hand, for any q < 1 / 2 , the distributional action gap decays to 0 as h ↓ 0 , which makes it difficult to identify the best actions in the presence of approximation error.

D Additional Experimental Results

Figure D.1 include results comparing the performance of DSUP( 1 / 2 ) and QR-DQN for risk-sensitive option trading across a variety of decision frequencies.

ω

= 20Hz

ω

= 25Hz

ω

= 30Hz

ω

= 35Hz

E Simulation Details

Here we collect further information about the setup for the simulations described in Section 5.2.

E.1 Option Trading Environment

The environment used for the high-frequency option-trading setup is identical to that of Lim and Malik [22]. The environment emulates policies that decide when to exercise American call options. The state space is modeled as X = R and with T = [0 , T ] . Notably, existing works such as [22] and [17] describe the state space by X = R 2 , where one dimension represents time-in our setup, we generally condition policies and returns on time, so we indeed model policies and returns as functions on R 2 . The state X t represents the price of the option at time t , and evolves according to a geometric Brownian motion.

There are two actions in the environment. Action 0 'holds' the option, while action 1 represents 'execute'. Upon taking action 1 (or equivalently, once the time reaches T ), the option is executed , and the agent receives a reward f ( x ) = max(0 , 1 -x ) and the episode terminates; here x represents the price at the time of execution. No rewards are incurred otherwise.

Following the setup of [22], the dynamics of the prices are simulated based on data collected between years 2016 and 2019 from 10 commodities on the DOW market. Lim and Malik proposed a method for estimating the most likely parameters of geometric Brownian motion to fit the data for each commodity, which is then used to simulate many environment rollouts for training and evaluation. This is particularly convenient for our setup, where we additionally scale the decision frequency, corresponding to finer time discretizations of the Euler-Maruyama scheme for the estimated geometric Brownian motion. Like [22], separate dynamics parameters are estimated (for each commodity) between training and evaluation: the dynamics used for training are estimated on prefixes of the data, and those for testing (post-training) are estimated on suffixes of the data. Results are reported on the testing dynamics, averaged over the 10 commodities.

As is standard [22, 17], we simulate the environment with T = 100 and X 0 = 1 . The simulations from [22] correspond to h = 1 in this setting. In our high-frequency simulations, we discretize the dynamics with timestep h < 1 .

E.2 Hyperparameters

In Table 1, we list the hyperparameters used in the simulations for the tested algorithms. We note that although the original DAU implementation scaled the learning rate with h , we take an alternative approach by updating every h -1 environment steps akin to [3]. This is discussed in more detail in Appendix C.

E.3 Compute Resources

Our implementations are written in Jax [6] and executed with a single NVidia V100 GPU. At highest decision frequencies, experiments took longer to execute, averaging out at a maximum of roughly four hours.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We believe our abstract and introduction outline and faithfully summarize the content of our work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have included discussions of the limitations and scope of our results throughout our work, rather than within a specific 'Limitations' section. See, for example, Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided complete statements, proofs, and references to used results in our proofs of our theoretical results. Please see the Appendix for restatements and proofs. Statements can be found in the main body of our work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We believe we have included enough information that others can faithfully reproduce our main experimental results. Please see Sections 4 and 5 and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided data, code, and experimental details. We believe we have included enough information that others can faithfully reproduce our main experimental results. Please see Section 5 and the Appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided these details in Section 5 and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: When appropriate, we have included error bars and information regarding the statistical significance of our experiments. Please see Section 5 and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have included these details in Section 5 and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and believe our work conforms to it in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We believe our work is foundational research without a direct path to negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe our work poses no risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Data for the option-trading environment was downloaded from an open source Github repository, and we cited the work that introduced the dataset [22].

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our work does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work neither involves crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work neither involves crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the scenario of continuous RL with high decision frequency. They show that it is hard for distributional RL agents to accurately estimate action values when the decision frequency increases, just like in the case of ordinary RL agents. They propose a distributional analogy of the action gap in ordinary RL called superiority. Based on this notion they try to mitigate this issue by the rescaling technique and propose a superiority-based algorithm. They also validate the proposed methods by numerical simulations.\n\nSTRENGTHS:\n* The paper is overall well-written and easy to follow.\n\n* The idea of superiority distributions is interesting and intuitive, with solid theoretical justifications.\n\n* The work is quite complete. Besides the discussion about the action gap in distributional RL and the notion of superiority distributions, the author also proposed new algorithms and performed extensive empirical studies.\n\nTherefore, I choose to give a positive rating of this work. However, given that I am unfamiliar with continuous RL, I choose to assign a low confidence score to my review.\n\nWEAKNESSES:\n* It seems that the basic idea of the superiority distribution is that for two distributions $\\eta_1,\\eta_2$, we define an object $\\Delta$ that can be viewed as the difference between $\\eta_1$ and $\\eta_2$. And then given a risk-sensitive objective $\\phi$, one may choose actions according to rescaled $\\phi(\\Delta/h^q)$, where $h$ denotes the decision frequency. But why not simply use the criterion $(\\phi(\\eta_1)-\\phi(\\eta_2))/h^q$ for decision making? I think the authors should discuss this issue in later versions of this paper.\n\n* I suggest the author add some explanations about the problem setting to the intro section, which would make the paper more readable for those not familiar with the specific field.\n\nQUESTIONS:\n* I note that the superior distributions are actually rescaled by $h^{-q}$, where $q$ is a tuning parameter. In the empirical studies, $q$ is set to be $1$ or $1/2$. Are there any possible heuristic rules for the choice of $q$?\n\n* It seems that in the simulation studies, the performances of some methods (DAU, DSUP(1), DAU+DSUP(1/2)) are not monotone w.r.t. the decision frequency, but oscillate a lot as the decision frequency increases. Why would this happen? Can the authors give a more detailed explanation?\n\nLIMITATIONS:\nIt seems the authors do not explicitly addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the action gap in distributional RL, where the decisions are made at high frequency. The authors showed that the distributional RL is also sensitive to the decision frequency. In particular, they proved the action-conditioned return distribution collapse with different rates for the statistics. Also, they introduce the generalization of the advantage function in continuous-time MDP, i.e., superiority, based on which a novel algorithm is designed. The authors finally conducted extensive experiments in the high-frequency option-trading domain to demonstrate their theory and the efficacy of their algorithms.\n\nSTRENGTHS:\n* The writing is rigorous with detailed and clear definitions and theorems. \n\n* Algorithmic practice in this setting is creditable. \n\n* Experiments are relatively extensive, considering both illustrated and comparative settings.\n\nWEAKNESSES:\n* The motivation can be strengthened. What is an action gap in continuous-time MDP, and what is an algorithm's outcome when encountering high-frequency decision-making settings? Why do we investigate the distributional RL in this setting? More explanation would be better.\n\n* The clarity can be enhanced. Although I appreciate the rigorous writing, it would be better to emphasize the main conclusions of this paper. For instance, the statistics in distributional RL also decrease at different rates. How do readers learn from these observations? Some rigorous theorems may be put in the appendix and replaced with more straightforward explanations.\n\nQUESTIONS:\n1. While the authors stated that DAU+DSUP(1/2) is inferior to DSUP(1/2), it is indeed a clear drawback of the proposed algorithm as this empirical result is consistent with the theory stated in Section 4.2, and hard to understand to me. Although the authors gave some explanations, they did not rigorously validate their hypothesis. Also, I am not convinced why DSUP(1/2) performs poorly in low-frequency domains. \n\n2. Why only consider 30HZ and DSUP(1/2) in the risk-sensitive setting? My suggestion is to include DAU+DSUP(1/2) as well for an extensive comparison across different frequencies, which could be more convincing. \n\n3. It is suggested that more examples be given to emphasize the significance of studying continuous-time MDP and the motivation to explore distributional RL.\n\n4. It would be better to provide the pointers to the proof of each theorem to allow readers to check the proof instantly.\n\n5. In what sense does the approximation hold in Eq. 4.2?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the issues around continuous-time distributional reinforcement learning. In traditional RL, the advantage becomes less informative as the frequency of actions increases, vanishing at the limit and making it impossible to distinguish between actions. This work extends this result to distributional RL in several ways: it first proposes a framework based on Wasserstein distances between return distributions, then proves in this framework that a similar problem appears in DLR, and gives tight bounds on the asymptotic convergence rate of the distgap (an action gap analogue for continuous DLR), establishing that the rate differs from the traditional RL case.\n\nAfter that, the paper constructs a notion of superiority - a DLR analogue of advantage, and considers its different variants (transformations) that ensure that action gaps are preserved in the limit. Finally, two families of algorithms are constructed, and shown to outperform the baseline QR-DQN on a benchmark of high-frequency options trading task.\n\nSTRENGTHS:\nThe paper investigates a very interesting question, and gives a theoretically satisfactory answers. The formal analysis gives rise to a practical algorithm, which is empirically shown to outperform the QR-DQN baseline. The presentation is very readable while being maximally mathematically precise, which I really appreciated.\n\nWEAKNESSES:\nThe theoretical part requires quite a lot of background from the reader. While authors give a short intro to SDEs in the appendix, it is, in my opinion, nowhere near the depth required to understand the paper. While it's not really possible to present a comprehensive intro to stochastic analysis, the paper would benefit, in my opinion, from a slight expansion of the relevant appendix. Also in terms of presentation, I found the explanation of the algorithmic part of the paper really dense and confusing, and I cannot say I understood it particularly well from the main text.\n\nQUESTIONS:\n1. The Wasserstein distances considered are for $p \\in [1, \\infty)$. What breaks if we put $p = \\infty$?\n2. Why is the behavior of $\\vartheta$ mean non-monotone wrt frequency in the example considered in sec 5.1?\n3. Apart form distributional RL, could a similar method be applied to continuous-time max-ent RL?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}