Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning

Authors: harley  wiltzer
marc g bellemare
david  meger
patrick  shafto
yash  jhaveri
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:58:11

Abstract: When decisions are made at high frequency, traditional reinforcement learning (RL) methods struggle to accurately estimate action values. In turn, their performance is inconsistent and often poor. Whether the performance of distributional RL (DRL) agents suffers similarly, however, is unknown. In this work, we establish that DRL agents *are* sensitive to the decision frequency. We prove that action-conditioned return distributions collapse to their underlying policy's return distribution as the decision frequency increases. We quantify the rate of collapse of these return distributions and exhibit that their statistics collapse at different rates. Moreover, we define distributional perspectives on action gaps and advantages. In particular, we introduce the *superiority* as a probabilistic generalization of the advantage---the core object of approaches to mitigating performance issues in high-frequency value-based RL. In addition, we build a superiority-based DRL algorithm. Through simulations in an option-trading domain, we validate that proper modeling of the superiority distribution produces improved controllers at high decision frequencies.

Main paper: 
 introduction: 
 In many real-time deployments of reinforcement learning (RL)-quantitative finance, robotics, and autonomous driving, for instance-the state of the environment evolves continuously in time, but policies make decisions at discrete timesteps (h units of time apart) [28] . In such systems, the performance of value-based agents is sensitive to the frequency ω := 1 /h with which actions are taken. In particular, action values become indistinguishable as the time between actions decreases. In turn, in high-frequency settings, Baird demonstrated that action value estimates are susceptible to noise and approximation error [20] . Moreover, Tallec et al. exhibited that the performance of popular deep Q-learning agents is inconsistent and often poor [34] .In order to remedy this sensitivity, Baird proposed the advantage function and advantage-based variants of Q-learning, Advantage Updating (AU) [20] and Advantage Learning (AL) [2] . Unlike action values, advantages (appropriately rescaled) do not become indistinguishable as decision frequency increases. As a result, Baird, in [20, 2] , demonstrated that advantage-based agents can learn faster and be more resilient to noise than their action value-based counterparts. Furthermore, Tallec et al. , in [34] , exhibited that their extension of AU, Deep Advantage Updating (DAU), works efficiently over a wide range of timesteps and environments, unlike standard deep Q-learning approaches.While advantage-based approaches to RL have demonstrated robustness to decision frequency, in this work, we establish that they are nevertheless sensitive to the frequency with which actions are taken. This discovery arises as we answer the question: to what extent is the performance of distributional RL (DRL) agents sensitive to decision frequency? To this end, we build theory within the formalism of continuous-time RL where environmental dynamics are governed by SDEs, as in [25] . Additionally, we validate our theory empirically through simulations. Specifically, we make the following four contributions: Distributional Action Gap. First, we extend notions of action gap to the realm of DRL. Precisely, we consider the minimal distance between pairs of action-conditioned distributions under metrics on the space of probability measures on R. We observe that some metrics are viable for this extension, while others are not. This formalism sets the stage for analyzing the influence of individual actions as well as decision frequency on, for example, an agent's return distributions.Collapse of Distributional Control at High Frequency. Second, we establish tight bounds on the distributional action gaps of h-dependent action-conditioned return distributions-return distributions induced by applying a specific initial action for h units of time. We prove that these distributional action gaps not only collapse, as h tends to zero, but do so at a slower rate than action-value gaps. On one hand, therefore, distributional Q-learning algorithms are susceptible to the same failures as Qlearning in continuous-time RL. On the other hand, however, remedies to these failures transliterated to distributional Q-learning algorithms are unlikely to succeed, because the means of these return distributions collapse faster than their other statistics.Distributional Superiority. Third, we propose an axiomatic construction of a distributional analogue of the advantage, which we call the superiority. Leveraging our analysis of h-dependent actionconditioned returns and their distributional action gaps, we present a frequency-scaled superiority distribution that enables greedy action selection at any fixed decision frequency.A Distributional Action Gap-Preserving Algorithm. Fourth, we propose an algorithm that learns the superiority distribution from data. Empirically, we demonstrate that our algorithm maintains the ability to perform policy optimization at high frequencies more reliably than existing methods. 
setting: 
 Notation: Spaces will either be subsets of Euclidean space or discrete. Measurability, in the former case, will be with respect to the Borel sigma algebra; in the latter case, it will be with respect to the power set. The set of probability measures over a space Y will be denoted by P(Y). Functions on spaces are assumed to be measurable. For f : Y → Z and µ ∈ P(Y), the push forward of µ through/by f , f # µ ∈ P(Z), is defined by f # µ := µ • f -1 . For a random variable X, defined implicitly on some probability space (Ω, F, P), we write law(X) := X # P to denote the law of X; the notation X = law Y is shorthand for law(X) = law(Y ). For any µ ∈ P(R), the quantile function of µ, F -1 µ , is defined by F -1 µ (τ ) := inf z {F µ (z) ≥ τ }, where F µ (z) is the CDF of µ.Here we give a brief introduction to the technical aspects of continuous-time RL, à la [25] . We provide additional exposition and references in Appendix A. For any reader looking to defer some of this technical introduction, we summarize the core objects of interest at the end of Section 2.1.1.Continuous-time Markov Decision Processes (MDPs) are defined by three spaces and four measurable functions: a time interval T := [0, T ] with T ∈ (0, ∞) or T := [0, T ) with T = ∞, a state space X ⊂ R n , an action space A, a drift b : T × X × A → R n , a diffusion σ : T × X × A → R n×n , a reward r : T × X → R, and a terminal reward f : X → R. 3 The pair (b, σ) govern the environment's dynamics by a family of SDEs parameterized by a ∈ A, dX a t = b(t, X a t , a) dt + σ(t, X a t , a) dB t .(2.1)Here (B t ) t≥0 is an n-dimensional Brownian motion. In turn, any solution to (2.1) collects the state paths of an agent that chooses action a at every time, regardless of the state they are in.As is done in discrete-time RL, an agent might consider the induced Markov Reward Process (MRP) derived from a policy π : T × X → P(A). 4 The dynamics of a policy-induced MRP (with policy π) are governed by the SDE dX π t = b π (t, X π t ) dt + σ π (t, X π t ) dB t .(2.2)Here, following [38, 15, 16] , the policy-averaged coefficients b π and σ π are defined by b π (t, x) := A b(t, x, a) π(da | t, x) and σ π (t, x) := A σσ ⊤ (t, x, a) π(da | t, x)1 /2 . (2.3)Thus, solutions to (2.2) collect the paths of an agent following policy π.A class of policies central to our study is those that fix an action a from some time t for a given persistence horizon h.Definition 2.1. Given h > 0 and a ∈ A, a policy π is said to be (h, a)-persistent at time t ∈ T if π(• | s, y) = δ a for all (s, y) ∈ [t, t + h) × X.In particular, given a policy π, we will consider (h, a)-persistent modifications of π: for t ∈ T,π| h,a,t (• | s, y) := δ a if s ∈ [t, t + h) π(• | s, y) if s / ∈ [t, t + h).These policies will help us understand the influence of taking actions relative to others as well as to those taken by π. We assume h is small enough so that t + h ∈ T.In order to guarantee the global-in-time existence and uniqueness of solutions to our SDEs (2.1) and (2.2), we make two sets of assumptions. sup t |b π (t, x) -b π (t, y)| + sup t |σ π (t, x) -σ π (t, y)| ≤ C 2.3 |x -y| ∀x, y ∈ X.These assumptions are standard in the analysis of continuous-time RL, optimal control, and SDEs [12, 26, 38, 16, 42] . Since π is a function of state, we note that Assumption 2.3 is not a direct consequence of Assumption 2.2. The coefficients b π and σ π satisfy the conditions of Assumption 2.3 provided b and σ satisfy some (also standard) additional regularity conditions and π satisfies some regularity conditions. These technical details are discussed in Appendix A.2.In summary, in continuous-time RL, there are three stochastic processes of interest: (X • s ) s≥t with • ∈ {a, π, π| h,a,t }, all beginning at some time t. These processes collect the state paths of an agent in one of three scenarios: 1. choosing action a at every state and time; 2. following a policy π; or 3. choosing a at every state and time for the first h units of time and following π thereafter.Given a policy-induced state process (X π s ) s≥t , the (discounted) random return G π (t, x) earned by π starting from state x ∈ X at time t ∈ T is defined [41] byEQUATIONwhere f ≡ 0 when T = [0, ∞). We distinguish returns earned by (h, a)-persistent modifications of policies. We call these h-dependent action-conditioned returns and denote them by Z π h (t, x, a). Given π, they are defined byEQUATIONValue-based approaches in RL estimate either the value function 5 As distributional approaches in RL estimate the laws of returns, following [41] , we define η π (t, x) := law(G π (t, x)) and ζ π h (t, x, a) := law(Z π h (t, x, a)). It is important to note that only the laws of random returns (and not their representations as random variables) are observable and modeled in practice.V π (t, x) := E[G π (t, x)] or the h-dependent action-value function Q π h (t, x, a) := E[Z π h (t, x, a)].The failure of action-value-based RL in continuous-time stems from the collapse of action values at a given state to the value of that state. Precisely, Tallec et al. and Jia and Zhou established thatEQUATIONwhere H π ∈ R is independent of h (see [34] and [16] respectively). 6 In a discrete action space, given a state x and time t, a concise way to capture the asymptotic information of (2.6) is by considering the action gap [11, 5] of the associated h-dependent action valuesgap(Q π h , t, x) := min a1̸ =a2 |Q π h (t, x, a 1 ) -Q π h (t, x, a 2 )|.Anticipating (2.6), which implies that gap(Q π h , t, x) = O(h), Baird proposed AU wherein he estimated the rescaled advantage function A π h in place of Q π h :A π h (t, x, a) := Q π h (t, x, a) -V π (t, x) h ∀(t, x, a) ∈ T × X × A. (2.7)Note that gap(A π h , t, x) = O(1). Tallec et al. [34] and Jia and Zhou [16] , following Baird, also estimated A π h to ameliorate Q-learning in continuous time.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have included discussions of the limitations and scope of our results throughout our work, rather than within a specific "Limitations" section. See, for example, Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
the distributional action gap: 
 In this section, we define a distributional notion of action gap; we prove that h-dependent actionconditioned return distributions collapse to their underlying policy's return distribution as h vanishes; and we quantify the rate of collapse of these return distributions. d(µ(t, x, a 1 ), µ(t, x, a 2 )).While R has a canonical metric, induced by | • |, the space P(R) does not. So a choice must be made, and some metrics are unsuitable. For example, in deterministic MDPs with deterministic policies, return distributions are identified by expected returns:ζ π h (t, x, a) is the delta at Q π h (t, x, a), for all (t, x, a) ∈ T × X × A. Thus, distgap d (ζ π h , t, x) should vanish as h decreases to zero if gap(Q π h , t, x) vanishes as h decreases to zero. With the total variation metric d = TV, for instance, this is not the case, making TV unsuitable. Indeed, suppose we have a deterministic MDP with A = {a 1 , a 2 } and such that ζ π h (t, x, a 1 ) = δ h and ζ π h (t, x, a 2 ) = δ 0 , for some state x and time t (see, e.g., [34] ). Thendistgap TV (ζ π h , t, x) = 1, for all h > 0, yet gap(Q π h , t, x) = h.The W p distances from the theory of Optimal Transportation (see [36] ), however, are suitable. They are defined via couplings of distributions.Definition 3.2. Let µ, ν ∈ P(R). A κ ∈ P(R 2) is a coupling of µ and ν if its first and second marginals are µ and ν respectively. We denote the set of these couplings by C (µ, ν).Definition 3.3. Let µ, ν ∈ P(R) 7 and p ∈ [1, ∞). The W p distance between µ and ν is W p (µ, ν) := inf κ∈C (µ,ν) R 2|z -w| p κ(dzdw) Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: We have provided complete statements, proofs, and references to used results in our proofs of our theoretical results. Please see the Appendix for restatements and proofs.Statements can be found in the main body of our work.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
distributional superiority: 
 In this section, we introduce a probabilistic generalization of the advantage. We define this random variable-which we call the superiority and denote by S π h -via a pair of axioms. A natural construction of the superiority at (t, x, a) is given by Z π h (t, x, a) -G π (t, x). The law of this difference, however, depends on the joint law of (Z π h (t, x, a), G π (t, x)), which is unobservable in practice and ill-defined (cf. Section 2.1.2). Yet, the set of all possible laws of this difference is easily characterized; it is the set of coupled difference representations of ζ π h (t, x, a) and η π (t, x). Definition 4.1. Let µ, ν ∈ P(R). A coupled difference representation (CDR) ψ ∈ P(R) of µ and ν takes the form ψ = ∆ # κ where κ ∈ C (µ, ν) and ∆ : R 2 → R is given by ∆(z, w) := z -w. The set of all coupled difference representations of µ and ν will be denoted by D(µ, ν).Our first axiom places the superiority's law in this set, D(ζ π h (t, x, a), η π (t, x)). Axiom 1. The law of S π h (t, x, a) is a coupled difference representation of ζ π h (t, x, a) and η π (t, x). Our second axiom encodes a type of consistency for deterministic policy behavior. Axiom 2. S π h (t, x, a) is deterministic whenever π is (h, a)-persistent at time t. To see how Axiom 2 encodes a notion of deterministic consistency, first consider its discrete-time analogue: the superiority at (x, a) for a policy π is deterministic if π at x deterministically chooses a. In this situation, our a-following agent makes the same choices as a π-following agent-both take action a in state x initially and then follow π thereafter-, and we posit that the superiority should not be random. The continuous-time analogue of the situation just described occurs precisely when a policy π is (h, a)-persistent at starting time t. Given a starting time t and state x, an agent that chooses action a between t and t + h and then follows π is following the (h, a)-persistent modification of π at time t. By definition, they make the same choices as a π-following agent when π is (h, a)-persistent starting at time t. Axiom ). This variance is 0 only when π's return is deterministic. Hence, ι π h (t, x, a) may be nontrivial even when conditioning on a reflects the policy's behavior exactly. We posit, via Axiom 2, that this should be prohibited.In fact, Axiom 2 determines a single coupling, if we want a consistent choice across all time-stateaction triplets and all MDPs.Theorem 4.3. Let κ ∈ C (µ, µ) for some µ ∈ P(R). The push-forward of κ by ∆ is the delta at zero, ∆ # κ = δ 0 , if and only if κ is a W p -optimal coupling, for some p ∈ [1, ∞). Moreover, there is only one such coupling. It is given by κ µ := (id, id) # µ or, equivalently,κ µ := (F -1 µ , F -1 µ ) # U(0, 1). Here U(0, 1) is the uniform distribution on [0, 1].The second definition of κ µ corresponds, more generally, to the W p -optimal coupling, for all p ≥ 1, of µ and ν given by κ µ,ν := (F -1 µ , F -1 ν ) # U(0, 1). As ∆ # κ µ,ν 's quanitle function is F -1 µ -F -1ν , we have in hand everything we need to define the superiority distribution (via its quantile function).Definition 4.4. The superiority distribution ψ π h at (t, x, a) ∈ T × X × A is ψ π h (t, x, a) := (F -1 ζ π h (t,x,a) -F -1 η π (t,x) ) # U(0, 1).As ψ π h (t, x, a) has the smallest possible central absolute pth moments among all CDRs of ζ π h (t, x, a) and η π (t, x), heuristically, it captures more of the individual features of both return distributions than other such CDRs (like ι π h (t, x, a) in Example 4.2). We illustrate this by example in From Section 3, we know that the W p distance between ζ π h (t, x, a) and η π (t, x), for every p ≥ 1, vanishes as h vanishes. Moreover, we know the rate at which this distance disappears. As a result, by construction, the central absolute pth moments of ψ π h (t, x, a) collapse as h collapses, and, for a large class of MDPs, we understand the rate at which these moments collapse. More generally and precisely, if we consider the q-rescaled superiority distribution defined by ψ π h;q := (h -q id) # ψ π h , we see that Theorems 3.6 and 3.7 translate to the follow statements on W p action gaps of ψ π h;q : Theorem 4.5. MDPs and policies exist satisfying Assumptions 2.2 and 2.3 in and under which, for all (t, x) ∈ T × X, we have that distgap p (ψ π h;q , t, x) ≳ h 1 /2-q . Theorem 4.6. Under Assumptions 2.2 and 2.3, if r is Lipschitz in state, uniformly in time, f is Lipschitz, and T < ∞, then distgap p (ψ π h;q , t, x) ≲ h 1 /2-q , for all (t, x) ∈ T × X. These two theorems tell us how to preserves the W p action gaps of q-rescaled superiority distributions (as a function of h). They identify q = 1 /2. For q < 1 /2, W p action gaps vanish as h vanishes. Whereas for q > 1 /2, W p action gaps blow up as h vanishes. These behaviors are undesirable. When q < 1 /2, the influence of an action on an agent's superiority becomes indistinguishable from any other action. For q > 1 /2, ever larger sample sizes are need to obtain any statistical estimate of an agent's superiority with the same level of accuracy. These scenarios are untenable.Another consideration regarding rescalings of ψ π h is whether they upend rankings of actions determined by some given measure of utility. This would be counterproductive. In DRL, agents often use distortion risk measures [1] to rank actions ( [10, 9, 22, 4, 17] ). Definition 4.7. Given β ∈ P([0, 1]), the distortion risk measure ρ β : P(R) → R is defined by ρ β (µ) := ⟨β, F -1 µ ⟩; on µ ∈ P(R), its value is given by the integral of F -1 µ with respect to β. A family of ρ β is the α-conditional value-at-risk measures (α-CVaR) [29] , where β α = U(0, α) for α ∈ (0, 1]; α = 1 is the expected-value utility measure. Crucially, ψ π h;q preserves ρ β -valued utility. Theorem 4.8. Let ρ β be a distortion risk measure, q ≥ 0, andh > 0. If ρ β (η π (t, x)) < ∞, then arg max a∈A ρ β (ψ π h;q (t, x, a)) = arg max a∈A ρ β (ζ π h (t, x, a)). In turn, the 1 /2-rescaled superiority distribution is not only W p action gap preserving but matches ζ π h in its greedy choice of action as measured by a distortion risk measure.We now turn to building DRL algorithms based on our theory. Our algorithms leverage the quantile TD-learning framework [10] to learn ρ β -greedy policies, for a given distortion risk measure ρ β , just as DAU [34] leverages the Q-learning framework to learn greedy policies. Full pseudocode and implementation details are given in Appendix C.At the heart of our algorithms is an equality of quantile functions, which holds by construction,EQUATIONIndeed, given η and ψ h;q , as models of η π and ψ π h;q respectively, equation (4.1) justifies the application of quantile TD-learning to ζ h , as a model for ζ π h , defined via the quantile functionF -1 ζ h (t,x,a) := F -1 η (t,x) + h q F -1 ψ h;q (t,x,a) . (4.2)That said, we cannot realize quantile TD-learning without defining predictions and bootstrap targets in terms of m-quantile representations9 [10, 4] of ζ h , via those of η and ψ h;q .While we may freely parameterize the m-quantile representation of η with a neural network (with interface) θ : T × X → R m , we have to be careful when parameterizing the m-quantile representation of ψ h;q . Given a neural network ϕ : T × X × A → R m , we setEQUATIONThis ensures we identify a ρ β -greedy policy; it is 0 at the ρ β -greedy action a ⋆ (cf. [34, Eq. 27] ).With appropriate parameterized m-quantile representations of η and ψ h;q in hand, we derive our predictions and bootstrap targets. By (4.2), recalling θ and (4.3), we compute our predictions viaEQUATIONBy Wiltzer [40] , as 10 for all p ∈ N. So upon getting a sample state/realization x t+h of X a t+h , as in [30] , we compute our bootstrap targets viaX π| h,a,t t = x and X π| h,a,t t+h = X a t+h , observe that Z π h (t, x, a) = law h 0 γ s r(t + s, X π| h,a,t t+s ) ds + γ h G π (t + h, X π| h,a,t t+h ) = law hr(t, x) + γ h G π (t + h, X a t+h ) + Y h , where E[|Y h | p ] = o(h)EQUATIONIn summary, the predictions (4.4) and the bootstrap targets (4.5) together characterize a family of QR-DQN-based algorithms called DSUP(q), whose core update is outlined in Algorithm 1.Algorithm 1 DSUP(q) Update Require: q (rescale factor), α (step size), D (replay buffer), ρ β (distortion risk measure)EQUATION)] ϕ ← ϕ -α∇ ϕ ℓ(ϕ, θ) and θ ← θ -α∇ θ ℓ(ϕ, θ) ▷ Gradient updatesOne theoretical drawback of DSUP(q) for mean-return control is that the mean of the q-rescaled superiority distribution is O(1) only when q = 1, by (2.6) and (2.7). Thus, we propose modeling A π h simultaneously. This yields a novel form of a two-timescale approach to value-based RL (see, e.g., [8] ). In particular, we estimate ϑ π h;q defined byF -1 ϑ π h;q (t,x,a) := F -1 ψ π h;q (t,x,a) + (1 -h 1-q )A π h (t, x, a).We call ϑ π h;q the advantage-shifted q-rescaled superiority. Note that its mean is A π h , which is O(1). To realize this, we approximate A π h using DAU and employ parameter sharing between the approximators of A π h and ψ π h;q . We call this family of algorithms DAU+DSUP(q). We note that A π h is used only for increasing action gaps; it does not change the training loss for η and ψ h;q .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes]Justification: We believe we have included enough information that others can faithfully reproduce our main experimental results. Please see Sections 4 and 5 and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
simulations: 
 The empirical work herein is two-fold in nature: illustrative and comparative. First, we simulate an example that illustrates Theorems 3.6/4.5 and Theorem 3.7/4.6 and their consequences. Second, in an option-trading environment, we compare the performance of ψ π h -based agent(s) against QR-DQN [10] and DAU [34] in the risk-neutral setting and against QR-DQN in a risk-sensitive setting.Consider an MDP with time horizon 10, a two element action space, 0 and 1-when action 1 is executed, the system follows 1-dimensional Brownian dynamics with a constant drift of 10, and otherwise, the state is fixed-, a reward that equals the agent's signed distance to 0, and a trivial terminal reward. We estimate four distributions at (t, x, a) = (0, 0, 1) for the policy that always selects 0. Figure 5 .1 shows these estimated distributions for a sample of frequencies (kHz), ω = 1 /h.ζ π h = ψ π h ψ π h;1 ω ψ π h; 1 /2 ϑ π h; 1 /2 Figure 5.1: Monte-Carlo estimates of ψ π h;q , for q = 0, 1, 1 /2, and ϑ π h; 1 /2 as a function of ω = 1 /h.First (from the left), we see that ψ π h collapses to δ 0 , as h tends to 0. Thus, accurate action ranking distributional or otherwise becomes impossible in the vanishing h limit. Second, we see that rescaling by h produces distributions with O(1) mean but infinite non-mean statistics in the vanishing h limit. Here the O(1) means are imperceptible in face of the large variances. So while this rescaling permits ranking actions by action values, it does so at the expense of producing high-variance distributions.Third, we see that rescaling by h 1 /2 yields distributions with O(1) non-mean statistics but vanishingly small means, O(h 1 /2 ). Hence, this rescaling permits ranking actions by non-mean statistics, even if action values again becomes indistinguishable in the vanishing h limit. That said, the vanishing rate of the means here is slower than when no rescaling is considered, O(h). Fourth, we see that rescaling by h 1 /2 and then shifting it by (1 -h 1 /2 )A π h produces distributions with O(1) mean and non-mean statistics. In turn, this two-timescale approach permits ranking actions by either action values or non-mean statistics (but not both by Theorem 4.8). However, the mean estimates here are inaccurate and imprecise-rather than uniformly being 100, they oscillate substantially.In risk-neutral control, we are left with a number of questions. What effect do the high variance distributions in DAU/DSUP(1) have on performance? What effect do the O(h 1 /2 ) means have on the performance of DSUP( 1 /2)? What effect does the instability of the mean estimates in DAU+DSUP( 1 /2) have on performance? In Section 5.2, we begin to answer these questions and others by testing our superiority-based algorithms against appropriate benchmarks in an option-trading environment.The option-trading environment in which we run our comparative experiments is a commonly used benchmark (see, e.g., [22, 17] ). We use an Euler-Maruyama discretization scheme [23] at high resolution to simulate high-frequency trading. Returns are averaged over 10 seeds and 10 different dynamics models (corresponding to data from different stocks). Additionally, following [22] , we use disjoint datasets to estimate the dynamics parameters for simulation during training and evaluation. 11First, we consider the risk-neutral setting. Here we compare QR-DQN, DAU, and three algorithms based on the q-rescaled superiority distribution with q = 1, 1 /2: DSUP(1), DAU+DSUP( 1 /2), and DSUP( 1 /2). We see that DSUP( 1 /2) is not only the most consistent performer, but outperforms every competitor at all but the two lowest frequencies. Even then, its performance is very close to the best performer. We also see that DAU+DSUP( 1 /2)'s preservation of both action gaps and W p action gaps does not lead to the strongest performance. In particular, its performance is inconsistent and sometimes poor. We believe this is because the tested frequencies are low enough that DSUP( 1 /2) maintains large enough action gaps to learn performant policies, but high enough that the variances of the distributions underlying A π h cause estimation difficulty. Indeed, the three methods that estimate A π h (explicitly in DAU and DAU+DSUP( 1 /2) or implicitly in DSUP(1)) exhibit almost identical behavior.Our results highlight a dichotomy in existing (ours included) methods for value-based, high-frequency, risk-neutral control. They can either maintain O(1) expected return estimates or O(1) return variance estimates, but not both. We observe better performance in estimating small means from O(1) variance distributions than in estimating O(1) means from receipricolly large variance distributions.To qualitatively illustrate the appeal of the 1 /2-rescaled superiority, Figure 5 .3 presents examples of learned action-conditioned distributions used by DSUP( 1 /2) and QR-DQN agents to make decisions. In this environment, action 1 taken in the start state terminates the episode, yielding the smallest return, 0, making this action inferior to its alternative action, 1. We see that DSUP( 1 /2) infers this fact. QR-DQN, on the other hand, has difficulty distinguishing these actions. This is because the 1 /2-rescaled superiority preserves W p action gaps, while ζ π h does not. Second, we consider a risk-sensitive setting. Here we compare QR-DQN and DSUP( 1 /2) using α-CVaR for greedy action selection. We do this because Theorem 4.8 does not hold with ϑ π h;q , and preserving means is less critical in risksensitive control than it is in risk-neutral control. Again, we see that DSUP( 1 /2) is conclusively the best performer. 
related work: 
 Notions of action gap and ranking have long been of interest in RL (see, e.g., [11] ). Action gaps are related to sample complexity in RL-indeed, instance-dependent sample complexity rates are inversely proportional to the divergence between action-conditioned return distributions ( [13, 19, 37] ). Bellemare et al. [5] argue for the consideration of alternatives to the Bellman operator that explicitly devalue suboptimal actions, and they show that Baird's AL [2] operator falls within this class of operators. On the other hand, Schaul et al. [31] implicitly question Bellemare et al.'s position. They demonstrate that stochastic gradient updates in deep value-based RL algorithms induce frequent changes in relative action values, which in turn is a mechanism for exploration.The advantage function is commonplace in RL (see, e.g., [39, 32, 27, 35, 24] ). In [24] , Mésnard et al. employ a distributional critic that is closely related to our (unscaled) distributional superiority. Their choice of critic stems from a desire to minimize variance. We note that the distributional superiority is a posteriori characterized as a minimal variance coupled difference representation of action-conditioned return distributions and policy-induced return distributions.Lastly, DRL in continuous-time MDPs is in its infancy. There are only three works to mention. Wiltzer et al. [40, 41] give a characterization of return distributions for policy evaluation, and Halperin [14] studies algorithms for control. That said, neither work considers distributional notions of action gaps or advantages. Moreover, Halperin does not consider any of the challenges of estimating the influence of actions in high decision frequency settings. 
conclusion: 
 We establish that DRL agents are sensitive to decision frequency through analysis and simulation. In experiments, DSUP( 1 /2) learns well-performing policies across a range of high decision frequencies, unlike prior approaches. DSUP (1) and DAU+DSUP( 1 /2) are less robust. Given our analysis, the performance of DSUP(1) is expected. Building an alternate algorithm to DAU+DSUP( 1 /2) that is both tailored to risk-neutral control and robust to h is an important avenue for future work.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: When appropriate, we have included error bars and information regarding the statistical significance of our experiments. Please see Section 5 and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have included these details in Section 5 and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We have reviewed the NeurIPS Code of Ethics and believe our work conforms to it in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We believe our work is foundational research without a direct path to negative societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our work neither involves crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our work neither involves crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Expected-value RL is a data-driven approach to solving the (classic) optimal control problem: find an action (control) process (A s ) s≥t and an associated state process (then determined by the environment) (X s ) s≥t with X t = x, for a given t ≥ 0, that maximize the expected return earned by following the state-action process (X s , A s ) s≥t . In particular, RL agents search the space of state-action processes via policies π : T × X → P(A). Policies prescribe the conditional probabilities of the laws of state-action processes. Indeed, (X s , A s ) s≥t is the state-action process of an agent following π if and only if, for each s ≥ t, the set {π(• | s, x)} x∈X is the set of conditional probabilities of law((X s , A s )) with respect to law(X s ).Continuous-time RL is a data-driven approach to stochastic optimal control. Whence, environmental dynamics are assumed to arise from an action-parameterized family of SDEs determined by a drift b : 12 Thus, the goal of expected-value RL (and stochastic optimal control) is to find an expected-return maximizing state-action process among state-action processes (X s , A s ) s≥t that satisfyT × X × A → R n and diffusion σ : T × X × A → R n×n .dX s = b(s, X s , A s ) ds + σ(s, X s , A s ) dB s with X t = x. (A.1)We note that the MDPs defined in Section 2 have equivalent formulations in terms of transition kernel, exactly as they formulated in discrete-time RL. We refer the reader to [33] for an in-depth discussion regarding this fact.Given the above formalism, the "true" distribution of returns of an agent following a policy π is the law ofT t γ s-t r(s, X s ) ds + γ T -t f (X T )where (X s , A s ) s≥t is a state-action process associated to π and solves (A.1). That said, by thedefinition of π, b π (s, X s ) = E[b(s, X s , A s ) | X s ] and σ π (σ π ) ⊤ (s, X s ) = E[σσ ⊤ (s, X s , A s ) | X s ],where b π and σ π are exactly the coefficients defined in (2.3). Hence, by [7] , provided that b π and σ π are regular enough to guarantee that (2.2) is well-posed in law, the processes X π = (X π s ) s≥t and X = (X s ) s≥t are equal in law. 13 Here (X π s ) s≥t satisfies (2.2) with X π t = x. Consequently,G π (t, x) = law T t γ s-t r(s, X s ) ds + γ T -t f (X T ).This, formally, justifies G π and Z π h , as defined in (2.4) and (2.5).Here we provide some conditions under which Assumption 2.3 is established. These conditions are presented as additional assumptions. Assumptions A.1 and A.2 are common in stochastic control theory (see, e.g., [12] ) and SDE theory in general (see, e.g., [26] ). Assumption A.3 is ubiquitous in the continuous-time RL literature [15, 16, 42] . Notably, these conditions together guarantee the existence of transition probabilities for policy-induced state processes arising from (2.2). Assumption A.2. The matrix σσ ⊤ is uniformly elliptic: a positive, finite constant λ A.2 exists such that inft,x,a inf |v|=1 v ⊤ σσ ⊤ (t, x, a)v ≥ λ 2 A.2 I.12 Without loss of generality, as the the laws of our stochastic processes are the objects of interest, we may assume that our diffusion matrix is square. Indeed, for any SDE with n × m-dimensional diffusion σ, the matrix (σσ ⊤ ) 1/2 , which is uniquely defined by σσ ⊤ , and not σ is the diffusion matrix that determines the law of the solution to that SDE. 13 The assumptions under which we work guarantee that X π = law X.A.2 is inf t,x inf |v|=1 v ⊤ σ π (t, x)v ≥ λ A.2 I. (A.2)In other words, σ π is also uniformly elliptic.Assumption A.3. A finite, positive constant C A.3 exists for whichsup t TV(π(• | t, x), π(• | t, y)) ≤ C A.3 |x -y| ∀x, y ∈ X,where TV is the total variation metric on P(A).Observe if π satisfies Assumption A.3, then π| h,a,t also satisfies Assumption A.3. Indeed,TV(π| h,a,t (• | s, x), π| h,a,t (• | s, y)) ≤ TV(π(• | s, x), π(• | s, y)) since sup s∈[t,t+h) TV(π| h,a,t (• | s, x), π| h,a,t (• | s, y)) = 0 and π| h,a,t (• | s, x) = π(• | s, x) for all s ∈ T \ [t, t + h).Proposition A. Proof. Observe that Let λ be an eigenvalue of σ π (t, x) -σ π (t, y) with with unit eigenvector v. Observe thatEQUATIONv ⊤ (σ π (t, x) 2 -σ π (t, y) 2 )v = v ⊤ (σ π (t, x) -σ π (t, y))σ π (t, x) + σ π (t, y)(σ π (t, x) -σ π (t, y))v = λv ⊤ (σ π (t, x) + σ π (t, x))vHence, by (A.2),|λ| = |v ⊤ (σ π (t, x) 2 -σ π (t, y) 2 )v| v ⊤ (σ π (t, x) + σ π (t, x))v ≤ 1 2λ A.2 |v ⊤ (σ π (t, x) 2 -σ π (t, y) 2 )v|.In turn, by Assumptions 2.2, A.1, and A.3, as done to prove that b π was Lipschitz above,|λ| ≤ 1 λ A.2 (C 2.2 + C A.1 C A.3 )|x -y|.Assumption 2.3 follows, since λ was an arbitrary eigenvalue and all norms on finite dimensional spaces are equivalent.We conclude this section with one final fact: under Assumption 2.2, the policy-averaged coefficient (2.3) have linear growth. Indeed,|b π (t, x)| ≤ |b(t, x, a)| π(da | t, x) ≤ C 2.2 (1 + |x|)and|σ π (t, x)| 2 ≤ |σ(t, x, a)| 2 π(da | t, x) ≤ C 2 2.2 (1 + |x|) 2 .In this paper, we assume that the rewards do not depend on actions. This is a theoretical limitation of not just our work, but continuous-time DRL in general (see [41, 14] ). In the following sections, we discuss the nature of this theoretical limitation. However, many MDPs have action-independent reward functions. For example, MDPs encoding goal-reaching problems, tracking problems, and commodity-trading problems all have action-independent rewards.In continuous-time, expected-value RL, when the reward function r depends on actions, the standard approach to analysis involves considering the averaged reward function r π : T × X → R given byr π (t, x) := A r(t, x, a) π(da | t, x).In continuous-time RL specifically, the averaged reward r π is justified exactly as the coefficients b π and σ π are justified. However, the "true" return distribution is not equal to the law ofT t γ s-t r π (s, X π s ) ds + γ T -t f (X π T ). (A.3)To see this, it suffices to consider an MDP with a single state x. In this case, the expression in (A.3) is deterministic. However, if π is nondeterministic and r is dependent on actions, then the "true" return distribution is nondeterministic. Hence, the law of the expression in (A.3) cannot be the "true" return distribution associated to π.In discrete-time RL, the distribution of returns given an action-dependent reward is analyzed through the state-action process induced by a policy. This processes is defined by extending the actionparameterized family of transition probability kernels on X, which define the dynamics of a given MDP, to a single transition probability kernel on X×A. In the time-homogeneous setting, for instance, with transition kernels {P (dy | x, a)} a∈A , this amounts to constructingP π (dydb | x, a) := π(db | y) ⊗ P (dy | x, a),provided that map y → π(E | y) is measurable for all y ∈ X. In continuous-time environments, such a constructing has yet to be discovered.We note that trying to analogously extend the action-parameterized family of transition semigroups on X, which define the dynamics of a given time-homogeneous MDP in continuous time, to a single transition semigroup on X × A by definingP π t (dydb | x, a) := π(db | y) ⊗ P t (dy | x, a), where P t (dy | x) is a transition semigroup, may fail to satisfy the Chapman-Kolmogorov identity. Indeed, suppose A has two elements and X = R. Let π(da | x) be the uniform measure on A for all x ∈ X. If P t (dy | x, a δ ) = δ x+t (dy) and P t (dy | x, a g ) = (2π) -1 /2 exp(-|y -x| 2 /2t) dy, then Chapman-Kolmogorov identity fails, for example, on any tuple (s, t, x, a δ , E × F ) where E ⊂ X is open, x + t + s / ∈ E, and π(F ) ̸ = 0. On one hand,P π t+s (E × F | x, a δ ) = π(F )δ x+t+s (E) = 0. On the other hand, P π s (E × F | y, a) P π t (dyda | x, a δ ) = π(F ) 2 (P s (E | x + t, a g ) + δ x+t+s (E)) > 0. So P π t+s (E × F | x, a δ ) ̸ = P π s (E × F | y, a) P π t (dyda | x, a δ ).At present, the question of how to generate a well-defined (even in law) state-action process in any continuous-time MDP framework given a stochastic policy is generally open. Of course, if π is deterministic, then the state-action process is (X s , π(s, X s )) s≥t . If b and σ are Lipschitz in state and action, uniformly in time, and π is Lipschitz in state, uniformly in time, then (A.1) with A s = π(s, X s ) is well-posed.In this section, we prove the statements made in Section 3.Before proving any of the statements made in Section 3, we recall an identity that relates the W p distance between two probability measures µ and ν and the absolute central pth moments of the differences of random variables distributed according to µ and ν:W p (µ, ν) p = inf (X,Y ) {E[|X -Y | p ] : law(X) = µ and law(Y ) = ν}. (B.1)This identity will be used a number of times, including in the proof of Section 3's first result, which we restate here for the readers convenience.Proposition 3.4. For all (t, x) ∈ T × X, we have thatdistgap p (ζ π h , t, x) ≥ gap(Q π h , t, x).Proof. Let (Z 1 , Z 2 ) be any random vector with such that law(Z i ) = ζ π h (t, x, a i ) for i = 1, 2.Then, by Jensen's inequality,E[|Z 1 -Z 2 | p ] 1/p ≥ E[|Z 1 -Z 2 |] ≥ |E[Z 1 -Z 2 ]| = |Q π h (t, x, a 1 ) -Q π h (t, x, a 2 )|. Hence, since (Z 1 , Z 2 ) was arbitrary, by (B.1), W p (ζ π h (t, x, a 1 ), ζ π h (t, x, a 2 )) ≥ |Q π h (t, x, a 1 ) -Q π h (t, x, a 2 )|.Finally, taking the minimum over pairs of actions (a 1 , a 2 ) such that a 1 ̸ = a 2 concludes the proof. Now we move on to the proofs of Theorems 3.5 and 3.7. We defer the proof of Theorem 3.6 until after the proof of Theorem 3.7 as the proofs of Theorems 3.5 and 3.7 are similar. For clarity's sake, we first prove a collection of lemmas.Lemma B.1. Under Assumption 2.2, let (X a s ) s≥t be the unique strong solution to (2.2) with X a t = x P-a.s. Then, for all q ≥ 1 and for all s ≥ t,E[|X a s -x| 2q ] ≤ C q C 2q 2.2 (1 + |x|) 2q ((s -t) q + 1)(s -t) q e CqC 2q2.2 ((s-t) q +1)(s-t) qwhere C q = 4 2q-1 .Proof. Let C q = 4 2q-1 . By Jensen's inequality and Itô's isometry, observe that|X a s -x| 2q ≤ C q (s -t) 2q-1 s t |b(s ′ , x, a)| 2q ds ′ + C q (s -t) q-1 s t |σ(s ′ , x, a)| 2q ds ′ + C q (s -t) 2q-1 s t |b(s ′ , X a s ′ , a) -b(s ′ , x, a)| 2q ds ′ + C q (s -t) q-1 s t |σ(s ′ , X a s ′ , a) -σ(s ′ , x, a)| 2q ds ′ ≤ C q ((s -t) 2q + (s -t) q )C 2q 2.2 (1 + |x|) 2q + C q ((s -t) 2q-1 + (s -t) q-1 )C 2q 2.2 s t |X a s ′ -x| 2q ds ′ .Thus, the lemma follows after taking expectation and applying Gronwall's inequality.Lemma B.2. Under Assumptions 2.2 and 2.3, let (X • s ) s≥t with • ∈ {π, π| h,a,t } be the unique strong solution to (2.2) with X • t = x P-a.s. Then, for all s ≤ t + h,E[|X π| h,a,t s -X π s | 2q ] ≤ C(1 + |x|) 2q((s -t) q + 1)(s -t) q e C((s-t) q +1)(s-t) q where C is some finite positive constant depending on q, C 2.2 , and C 2.3 .Proof. Let C q = 8 2q-1 . Note that X π| h,a,t s ′ = X a s ′ P-a.s. for all s ′ ≤ t + h, by the definition of π| h,a,t and the uniqueness of strong solutions to (2.1). So, by Jensen's inequality and Itô's isometry, observe that|X π| h,a,t s -X π s | 2q ≤ C q ((s -t) 2q-1 + (s -t) q-1 ) s t (C 2q 2.2 + C 2q 2.3 )|X a s ′ -x| 2q ds ′ + s t (C 2q 2.2 + C 2q 2.3 )(1 + |x|) 2q ds ′ + s t C 2q 2.3 |X π| h,a,t s ′ -X π s ′ | 2q ds ′ .Thus, after taking expectation, we deduce thatE[|X π| h,a,t s -X π s | 2 ] ≤ C q ((s -t) 2q + (s -t) q ) (C 2q 2.2 + C 2q 2.3 ) max s ′ ∈[s,t] E[|X a s ′ -x| 2q |] + (C 2q 2.2 + C 2q 2.3 )(1 + |x|) 2q + C q ((s -t) 2q-1 + (s -t) q-1 )C 2q 2.3 s t E[|X π| h,a,t s ′ -X π s ′ | 2q ] ds ′ .And so, the lemma follows after applying Lemma B. π| h,a,t s -X π s | 2q ] ≤ C(1 + |x|) 2q (h q + 1)h q e C((s-t-h) q +1)(s-t-h) q where C is some finite positive constant depending on q, C 2.2 , and C 2.3 .Proof. Let C q = 3 2q-1 . Observe thatX • s = X • t+h + s t+h b • (s ′ , X • s ′ ) ds ′ + s t+h σ • (s ′ , X • s ′ ) dB s ′ So, as π| h,a,t (• | s ′ , y) = π(• | s ′ , y) for all (s ′ , y) ∈ T \ [t, t + h) × X, by Jensen's inequality and Itô's isometry,|X π| h,a,t s -X π s | 2q ≤ C q |X π| h,a,t t+h -X π t+h | 2q + C q (s -t -h) 2q-1 + (s -t -h) q-1 C 2q 2.3 s t+h |X π| h,a,t s ′ -X π s ′ | 2q ds ′ .Thus, after taking expectation, applying Gronwall's inequality, and considering Lemma B.2, the lemma follows.One consequence of Lemmas B.2 and B.3 isE[|X π| h,a,t s -X π s | p ] → 0 as h ↓ 0, for all s ∈ T and for all p ∈ [1, ∞). And so, if f is bounded, then f (X π| h,a,t T ) -f (X π T) is bounded and converges to zero P-a.s. as h converges to zero. The dominated convergence theorem implies thatE[|(f (X π| h,a,t T ) -f (X π T ))| p ] → 0 as h ↓ 0. (B.2) Similarly, we see that the functions g h : T → [0, ∞) defined byg h (s) := E[|r(s, X π| h,a,t s) -r(s, X π s )| p ] are uniformly bounded (in h) and converge to zero as h ↓ 0 for every s ∈ T. Hence, by the dominated convergence theorem, again,T t E[|r(s, X π| h,a,t s ) -r(s, X π s )| p ] Γ(ds) → 0 as h ↓ 0 (B.3)where Γ(ds) := (γ s-t /C T,t,γ ) ds for C T,t,γ := (γ T -t -1)/ log γ. We now prove Theorem 3.5.Theorem 3.5. If r and f are bounded, then lim h↓0 W p (ζ π h (t, x, a), η π (t, x)) = 0, for all (t, x, a) ∈ T × X × A; hence, lim h↓0 distgap p (ζ π h , t, x) = 0.Proof. Observe thatdistgap p (ζ π h , t, x) = min a1̸ =a2 W p (ζ π h (t, x, a 1 ), ζ π h (t, x, a 1 )) ≤ min a1̸ =a2 W p (ζ π h (t, x, a 1 ), η π (t, x)) + min a1̸ =a2 W p (ζ π h (t, x, a 2 ), η π (t, x)) ≤ 2 max a W p (ζ π h (t, x, a), η π (t, x)).Thus, as claimed, it suffices to show thatW p (ζ π h (t, x, a), η π (t, x)) → 0 as h ↓ 0, for all (t, x, a) ∈ T × X × A. By (B.1), it suffices to show that E[|Z π h (t, x, a) -G π (t, x)| p ] → 0 as h ↓ 0, (B.4)for all (t, x, a) ∈ T × X × A.Since (s, ω) → X • s (ω) is measurable for • ∈ {π, π| h,a,t }, by Jensen's inequality and Fubini's theorem, we see that E[|Z π h (t, x, a) -G π (t, x)| p ] ≤ 2 p-1 C p T,t,γ T t E[|r(s, X π| h,a,t s ) -r(s, X π s )| p ] Γ(ds) + 2 p-1 γ p(T -t) E[|(f (X π| h,a,t T ) -f (X π T ))| p ]. (B.p (ζ π h (t, x, a), η π (t, x)) ≲ h 1 /2 , for all (t, x, a) ∈ T × X × A; hence, distgap p (ζ π h , t, x) ≲ h 1 /2 .Proof. Arguing as in the proof of Theorem 3.5, we see thatE[|Z π h (t, x, a) -G π (t, x)| p ] ≤ 2 p-1 C p T,t,γ I + 2 p-1 γ p(T -t) II with I := T t E[|r(s, X π| h,a,t s ) -r(s, X π s )| p ] Γ(ds) and II := E[|(f (X π| h,a,t T ) -f (X π T ))| p ].This is (B.5). In turn, by (B.6) and that r is Lipschitz in space, uniformly in time and f is Lipschitz, we deduce thatI + II ≤ C (B.6) (C p r + C p f )h p/2, as desired, where C r and C f are the Lipschitz constants of r and f respectively.Recall r : T × X → R is Lipschitz in state, uniformly in time if a finite positive constant C r exists such that supt |r(t, x) -r(t, y)| ≤ C r |x -y| ∀x, y ∈ X.As promised, we now prove Theorem 3.6.Theorem 3.6. MDPs and policies exist in and under which, for all (t, x, a) ∈ T × X × A, we have thatW p (ζ π h (t, x, a), η π (t, x)) ≳ h 1 /2 and distgap p (ζ π h , t, x) ≳ h 1 /2 .Proof. Let X = R and A = {0, 1}; set b ≡ 0 and σ = 1 {a=1} ; for all (s, y) ∈ T × X, let r(s, y) = y; and set f ≡ 0. In words, our action space has two elements, and when action 1 is executed, the system follows Brownian dynamics, and otherwise, the state is fixed. Now consider the policy π which always selects the action 0: π(• | s, y) = δ 0 , for all (s, y) ∈ T × X.Case 1: γ = 1 and T < ∞. Observe thatZ π h (t, x, 1) -Z π h (t, x, 0) = law h 0 Bs ds + (T -t -h) Bh ,where ( Bs ) s≥0 is a Brownian motion. Hence, Z π h (t, x, 1) -Z π h (t, x, 0) is equal in law to the sum of two zero mean Gaussian random variables with variances σ 2 1 = h 3 /3 and σ 2 2 = (T -t -h) 2 h respectively. And so, it is also Gaussian, and its variance isσ 2 h = σ 2 1 + σ 2 2 + 2cσ 1 σ 2 for some -1 ≤ c ≤ 1. In particular, σ p h ≳ h p/2 .Recall that central absolute pth moment of a Gaussian random variable is proportional to its standard deviation to the power p, for all p ≥ 1. Therefore, by (B.1), we deduce that distgap p (ζ π h , t, x) ≳ h 1 /2 , for h < 1, as desired. Case 2: T ∈ [0, ∞] and γ ∈ (0, 1). Observe thatZ π h (t, x, 1) -Z π h (t, x, 0) = law N (h) + γ T -t -γ h log γ Bh with N (h) := h 0 γ s Bs ds,for some Brownian motion ( Bs ) s≥0 . We claim that N (h) is a mean zero Gaussian random variable with variance σ 2 h ≈ h 3 /3. Then the concluding argument in the proof of Case 1 also concludes the proof of this case.To prove our claim, first note thatE[N (h)] = h 0 γ s E[ Bs ] ds = 0 and E[N (h) 2 ] = h 0 h 0 γ s+s ′ E[ Bs Bs ′ ] dsds ′ .Asγ 2h h 3 3 = γ 2h h 0 h 0 min{s, s ′ } dsds ′ ≤ E[N (h) 2 ] ≤ h 0 h 0 min{s, s ′ } dsds ′ ≤ h 3 3 ,we see that N (h) has the claimed statistics. Second, observe that N (h) = lim n→∞ N n (h) whereN n (h) := lim n→∞ n-1 i=0 γ si Bsi d h,n with d h,n := h n and s i := id h,n .(This is simply a Riemann sum approximation of the integral that defines N (h).) As the sum of any finite number of Gaussian random variables is a Gaussian random variable, N n (h) is Gaussian. Furthermore, as the limit of a sequence of Gaussian random variables whose sequences of means and variances converge (to finite values) is Gaussian, N (h) is Gaussian, which proves our claim.In this section, we prove the statements and claims made in Section 4.First, we prove that the mean of everyψ π h (t, x, a) ∈ D(ζ π h (t, x, a), η π (t, x)) is Q π h (t, x, a)-V π (t, x). Lemma B.4. If ψ π h (t, x, a) ∈ D(ζ π h (t, x, a), η π (t, x)), then its mean is Q π h (t, x, a) -V π (t, x). Proof. If κ π h (t, x, a) be such that ∆ # κ π h (t, x, a) = ψ π h (t, x, a), then r ψ π h (t, x, a)(dr) = (z -w) κ π h (t, x, a)(dzdw) = z ζ π h (t, x, a)(dz) -w η π (t, x)(dw) = Q π h (t, x, a) -V π (t, x), as desired.Second, we prove Theorem 4.3.Theorem 4.3. Let κ ∈ C (µ, µ) for some µ ∈ P(R). The push-forward of κ by ∆ is the delta at zero, ∆ # κ = δ 0 , if and only if κ is a W p -optimal coupling, for some p ∈ [1, ∞). Moreover, there is only one such coupling. It is given by κ µ := (id, id) # µ or, equivalently, κ µ := (F -1 µ , F -1 µ ) # U(0, 1). Here U(0, 1) is the uniform distribution on [0, 1].First, we establish that there is only one W p optimal coupling between a given µ ∈ P(R) and itself, for every p ∈ [1, ∞).Lemma B.5. Let µ ∈ P(R). There is only one W p optimal coupling between µ and itself, for everyp ∈ [1, ∞). It is κ µ := (id, id) # µ ∈ C (µ, µ).Proof. Let κ ∈ C (µ, µ), and suppose there exists ϵ > 0 for whichc ϵ := κ({|z -w| ≥ ϵ : z, w ∈ R}) > 0. Then |z -w| p κ(dzdw) ≥ {|z-w|≥ϵ} |z -w| p κ(dzdw) ≥ |ϵ| p c ϵ > 0 = |z -w| p κ µ (dzdw).Hence, κ, as considered, is not optimal. Since ϵ was arbitrary, it follows that a W p optimal coupling is concentrated on {z = w}. Therefore, every optimal coupling is of the form (id, id) # ν for some ν ∈ P(R). As the marginals of such a coupling are ν and ν, we deduce that ν = µ, as desired.Proof of Theorem 4.3. Let κ ∈ C (µ, µ) be such that ∆ # κ = δ 0 . Observe that |z -w| p κ(dzdw) = r 2 δ 0 (dr) = 0 = |z -w| p κ µ (dzdw),where, again, κ µ = (id, id) # µ. Hence, κ = κ µ , by Lemma B.5. On the other hand, since κ µ is concentrated on {z = w}, for any bounded, continuous function g, we seeg(r) ∆ # κ µ (dr) = g(z -w) κ µ (dzdw) = {z=w} g(z -w) κ µ (dzdw) = g(0).In turn, ∆ # κ µ = δ 0 , as desired.Remark B.6. Under the hypotheses of Theorem 3.7, we see that the 1 /2-rescaled superiority distributions at any (t, x, a) for h ∈ (0, 1] is a family of probability measures with uniformly bounded second moment. Hence, this family is tight. So, up to subsequences, these rescalings converges to limiting probability measure as h ↓ 0. An interesting open question, is whether or not these subsequential limits are the same.Third, we prove Theorems 4.5 and 4.6. The proofs of these theorems are a consequence of the following expression for the W p distance between µ and ν when µ, ν ∈ P(R):W p (µ, ν) p = 1 0 |F -1 µ (τ ) -F -1 ν (τ )| p dτ.Theorem 4.5. MDPs and policies exist satisfying Assumptions 2.2 and 2.3 in and under which, for all (t, x) ∈ T × X, we have that distgap p (ψ π h;q , t, x) ≳ h 1 /2-q .Proof. Recall that F -1ψ π h;q (t,x,a) = h -q (F -1 ζ π h (t,x,a) -F -1 η π (t,x)), for all a ∈ A. Hence, for every a 1 ̸ = a 2 , W p (ψ π h;q (t, x, a 1 ), ψ π h;q (t, x, a 2 )) p = 1 0 |F -1 ψ π h;q (t,x,a1) (τ ) -F -1 ψ π h;q (t,x,a2) (τ )| p dτ = h -qp 1 0 |F -1 ζ π h (t,x,a1) (τ ) -F -1 ζ π h (t,x,a2) (τ )| p = h -qp W p (ζ π h (t, x, a 1 ), ζ π h (t, x, a 2 )) p .Thus, taking the pth root of both sides of the above equality, we deduce thatW p (ψ π h;q (t, x, a 1 ), ψ π h;q (t, x, a 2 )) = h -q W p (ζ π h (t, x, a 1 ), ζ π h (t, x, a 2 )).The example presented in Theorem 3.6 is such that W p (ζ π h (t, x, a 1 ), ζ π h (t, x, a 2 )) ≳ h 1 /2 . Whence, W p (ψ π h;q (t, x, a 1 ), ψ π h;q (t, x, a 2 )) ≳ h 1 /2-q .Theorem 4.6. Under Assumptions 2.2 and 2.3, if r is Lipschitz in state, uniformly in time, f is Lipschitz, and T < ∞, then distgap p (ψ π h;q , t, x) ≲ h 1 /2-q , for all (t, x) ∈ T × X. The proof of Theorem 4.6 is almost identical to the proof of Theorem 4.5.Proof. Arguing as in the proof of Theorem 4.5, we see thatW p (ψ π h;q (t, x, a 1 ), ψ π h;q (t, x, a 2 )) = h -q W p (ζ π h (t, x, a 1 ), ζ π h (t, x, a 2 )).By Theorem 3.7, it follows that W p (ψ π h;q (t, x, a 1 ),ψ π h;q (t, x, a 2 )) ≲ h 1 /2-q .Finally, we prove Theorem 4.8Theorem 4.8. Let ρ β be a distortion risk measure, q ≥ 0, and h > 0.If ρ β (η π (t, x)) < ∞, then arg max a∈A ρ β (ψ π h;q (t, x, a)) = arg max a∈A ρ β (ζ π h (t, x, a)).Proof. Observe thatρ β (ψ π h;q (t, x, a)) = 1 0 F -1 ψ π h;q (t,x,a) (τ ) β(dτ ) = 1 0 h -q F -1 ψ π h (t,x,a) (τ ) β(dτ ) = h -q 1 0 (F -1 ζ π h (t,x,a) (τ ) -F -1 η π (t,x) (τ )) β(dτ ) = h -q ρ β (ζ π h (t, x, a)) -h -q ρ β (η π (t, x)) Since ρ β (η π (t, x)) is independent of a and h > 0, arg max a ρ β (ψ π h;q (t, x, a)) = arg max a ρ β (ζ π h (t, x, a)).Here we discuss methods for policy optimization via distributional superiority. In practice, computers operate at a finite frequency-as such, all policies we consider here will be assumed to apply each action for h units of time, as in the settings of [34, 16] .Before describing the superiority learning algorithms, we first remark on the form of exploration policies used in our approaches. We consider policies of the formπ explore : R A × R n → A.The first argument to such a policy is a vector of "action-values". In our case, given a superiority distribution ψ π h;q (t, x, •), the action values may be (ρ β (ψ π h;q (t, x, a))) a∈A for a distortion risk measure ρ β . This generalizes the notion of Q-values for distributional learning.The second argument represents a noise variable, in order to support stochastic policies. As input to our algorithms, we require a probability measure P act on processes (ϵ t ) t≥0 . This framework generalizes common exploration methods in deep RL. For example, to recover ϵ-greedy policies, P act represent a 2-dimensional white noise, andπ explore (v, ξ) = arg max a v a if ξ 1 ≤ ϵ a ⌈ξ2|A|⌉ otherwise. (C.1)Alternatively, one might choose to correlate the action noise. Approaches such as DDPG [21] and DAU present such examples, where action noise evolves over time as an Ornstein-Uhlenbeck process. This can be implemented in the framework above by choosing P act as the distribution of an |A|-dimensional Ornstein-Uhlenbeck process, and definingπ explore (v, ξ) = arg max a∈A (v + ξ) . (C.2)In our experiments, we found that ϵ-greedy exploration was sufficient for approximate policy optimization. To keep our implementation closest to the QR-DQN baseline, therefore, our implementations use the definition of π explore from equationThe remainder of this section details the implementation of DSUP(q) and DAU+DSUP(q) as introduced in Section 4.2. Additionally, we provide source code for our implementations at https://github.com/harwiltz/distributional-superiority.Generally, our goal is to learn a ρ β -greedy policy (see Definition 4.7). Since all policies apply actions for h units of time, in order to satisfy Axiom 2, we want S π h (t, x, π(x)) ≡ 0, where π : x → arg max a ρ β (S π h (t, x, a)) is the ρ β -greedy policy. As such, following the DAU algorithm of [34] , our algorithms will model quantile functions ϕ(t, x, a) ∈ R m that aim to satisfyF -1 ψ π h;q (t,x,a;h) ≈ ϕ(t, x, a) -ϕ(t, x, a ⋆ ) for some a ⋆ ∈ arg max a ρ β (ϕ(t, x, a)).Our primary algorithmic contribution integrates such a model, with proper superiority rescaling, into the QR-DQN framework of [10] for estimating action-conditioned return distributions. It is outlined in Algorithm 2.To deal with the increased time-resolution of transitions, [34] modified the step size by a factor of h. More recently, [3] found that subsampling transitions before storing them in the replay buffer was most effective in their high-decision-frequency domain. Thus, we opt for such a strategy here. Rather than only storing every h -1 transitions, we randomly select transitions to store according to independent Bernoulli(h) draws in order to avoid the possibility of only capturing cyclic phenomena in the replay buffer. We found that this strategy worked similarly to that of [34] , but is far less computationally expensive. Likewise, as h decreases, we extend the number of training interactions by a factor of h -1 . This corresponds to training for a constant amount of time units across decision frequencies, and likewise, a constant number of gradient updates across decision frequencies.An astute reader might recognize that while ψ π h;q may have nonzero distributional action gap, since h q is asymptotically larger than h for q < 1, the works of [34, 16] would suggest that the expected action gap under ψ π h;q should vanish. To account for this, we propose shifting the rescaled superiority quantiles by the advantage function, as estimated e.g. in DAU [34] . It is clear that such a procedure cannot cause the distributional action gap to vanish. The resulting procedure is depicted in Algorithm 3, with the modifications relative to Algorithm 2 highlighted in blue. In practice, we employ a shared feature extractor in the representations of A h and ϕ to reap the representation learning benefits of DRL [4] when approximating the advantage.The algorithms 2 and 3 are parameterized by a rescaling factor q ∈ (0, 1], which is meant to compensate for the collapse of the distributional action gap. Larger values of q correspond to larger compensation. In this work, we argue that the distributional action gap collapses at rate h 1 /2 , leading to a natural choice of q = 1 /2, which theoretically preserves constant order action gaps with respect to the decision frequency. We also test the approach with q = 1, which corresponds to the well-known scaling rate for preserving expected value action gaps.For any q > 1 /2, the distributional action gap theoretically grows without bound as h ↓ 0, leading to distributional estimates with arbitrarily large variance. On the other hand, for any q < 1 /2, the distributional action gap decays to 0 as h ↓ 0, which makes it difficult to identify the best actions in the presence of approximation error.Algorithm 2 Distributional Superiority Quantile Regression Require: m (number of quantiles) q factor) h (decision frequency), µ state Require: explore policy) and ρ (distortion measure) P (action distribution) θ(t, ϕ(t, a) R , quantile for (t, ψ h;q (t, x, Require: x) R , quantile for π x). Require: n θ (target period), (step size), N (batch size) D ← ∅ reset ← True for i ∈ N do // Gather data with exploratory policy (ξ) s≥0 ∼ P act ▷ Sample from stochastic process for exploration noisefor j ∈ {0, . . . , ⌊h -1 ⌋} do if reset is True then t ← 0 and x 0 ∼ µ reset ← False end if a ⋆ ← arg max a ρ β 1 m m n=1 δ ϕ(t,xt,a)n F -1 ψ h;q (a) ← ϕ(t, x t , a) -ϕ(t, x t , a ⋆ ) for each a ∈ A v a ← ρ β 1 m m n=1 δ F -1 ψ h;q (a)n for each a ∈ A a t ∼ π explore ((v a ) a∈A , ξ jh )Perform action a t for h units of time, observe (r t , x t+h , done t+h ). Y j ∼ Bernoulli(h) ▷ Subsample transitions for replay if Y j = 1 ∨ done j+1 = True then D ← D ∪ {(t,EQUATION)] end for ℓ(ϕ, θ) ← 1 N N k=1 ℓ k (ϕ, θ) ϕ ← ϕ -α∇ ϕ ℓ(ϕ, θ) and θ ← θ -α∇ θ ℓ(ϕ, θ) ▷ Gradient updates if i | n θ then θ ← θ end if end forAlgorithm 3 Advantage-Shifted Distributional Superiority Quantile Regression Require: m of and q (rescale Require: -1 (decision frequency), (initial distribution) Require: π (exploration and β risk Require: act noise Require: x), x, ∈ m parameterized representations for π (t, x), ψ π (t, x, a). Require: θ(t, x) ∈ R m , target quantile representations for η π (t, x). Require: A h (t, x, a) (parameterized representation of advantage function) Require: n θ (target update period), α (step size), N (batch size) D ← ∅ reset ← True for i ∈ N do // Gather data with exploratory policy (ξ) s≥0 ∼ P act ▷ Sample from stochastic process for exploration noise for j ∈ {0, . . . , ⌊h -1 ⌋} do if reset is True then t ← 0 andx 0 ∼ µ reset ← False end if a ⋆ ← arg max a ρ β 1 m m n=1 δ ϕ(t,xt,a)n+(1-h 1-q ) A h (t,xt,a) F -1 ψ h;q (a) ← ϕ(t, x t , a) -ϕ(t, x t , a ⋆ ) for each a ∈ A A h (t, x t , •) ← A h (t, x t , •) -A h (t, x t , a ⋆ ) ▷ see [34, DAU] v a ← ρ β 1 m m n=1 δ F -1 ψ h;q (a)n+(1-h 1-q ) A h (t,xt,a)for each a ∈ A a t ∼ π explore ((v a ) a∈A , ξ jh ) Perform action a t for h units of time, observe (r t , x t+h , done t+h ). Y j ∼ Bernoulli(h) ▷ Subsample transitions for replay if Y j = 1 ∨ done j+1 = True then D ← D ∪ {(t,EQUATION)] end for ℓ(ϕ, θ) ← 1 N N k=1 ℓ k (ϕ, θ) and ℓ adv ( A h ) ← 1 2N N k=1 ℓ adv k ( A h ) ϕ ← ϕ -α∇ ϕ ℓ(ϕ, θ) and θ ← θ -α∇ θ ℓ(ϕ, θ) and A h ← A h -α∇ℓ adv ( A h ) if i | n θ then θ ← θ end if end forHere we collect further information about the setup for the simulations described in Section 5.2.The environment used for the high-frequency option-trading setup is identical to that of Lim and Malik [22] . The environment emulates policies that decide when to exercise American call options. The state space is modeled as X = R and with T = [0, T ]. Notably, existing works such as [22] and [17] describe the state space by X = R 2 , where one dimension represents time-in our setup, we generally condition policies and returns on time, so we indeed model policies and returns as functions on R 2 . The state X t represents the price of the option at time t, and evolves according to a geometric Brownian motion.There are two actions in the environment. Action 0 "holds" the option, while action 1 represents "execute". Upon taking action 1 (or equivalently, once the time reaches T ), the option is executed, and the agent receives a reward f (x) = max(0, 1 -x) and the episode terminates; here x represents the price at the time of execution. No rewards are incurred otherwise.Following the setup of [22] , the dynamics of the prices are simulated based on data collected between years 2016 and 2019 from 10 commodities on the DOW market. Lim and Malik proposed a method for estimating the most likely parameters of geometric Brownian motion to fit the data for each commodity, which is then used to simulate many environment rollouts for training and evaluation. This is particularly convenient for our setup, where we additionally scale the decision frequency, corresponding to finer time discretizations of the Euler-Maruyama scheme for the estimated geometric Brownian motion. Like [22] , separate dynamics parameters are estimated (for each commodity) between training and evaluation: the dynamics used for training are estimated on prefixes of the data, and those for testing (post-training) are estimated on suffixes of the data. Results are reported on the testing dynamics, averaged over the 10 commodities.As is standard [22, 17] , we simulate the environment with T = 100 and X 0 = 1. The simulations from [22] correspond to h = 1 in this setting. In our high-frequency simulations, we discretize the dynamics with timestep h < 1.In Table 1 , we list the hyperparameters used in the simulations for the tested algorithms. We note that although the original DAU implementation scaled the learning rate with h, we take an alternative approach by updating every h -1 environment steps akin to This is discussed more in Appendix C. Justification: We believe our abstract and introduction outline and faithfully summarize the content of our work.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.We work with action-independent rewards. This is the norm in continuous-time DRL.We assume that (t, x) → π(E | t, x) is measurable for every E ⊂ A and every (t, x) ∈ T × X.In[38] and[16], the authors established V π and Q π h as continuous-time RL analogues of the classic value and action-value functions. We note, however, that this was done without explicitly defining G π and Z π h .The function H π is the Hamiltonian of the classic stochastic optimal control problem defined by our MDP.Here we assume that µ and ν have finite absolute pth moments.Here ζ π h (t, x, a) = η π (t, x); so the mean of ∆ # κ π h (t, x, a) is 0.An m-quantile representation of a given distribution in P(R) is a vector in R m whose ith component encodes the i-1/2 m -quantile of said distribution.This holds if r is continuous, a standard assumption in continuous-time control (see, e.g.,[25,34,16]).Our code is available at https://github.com/harwiltz/distributional-superiority. 


Decision: This paper investigates the challenges arising from dealing with continuous-time control in distributional RL. In particular, whenever decision are made at a high frequency, it comes harder to compare the relative advatage of actions, thus making it difficult or impossible to distinguish between them. This paper introduces a framework exploiting Wasserstein distances between return distributions to provide bounds on the convergence rate of the gap between actions and generalizes the notion of advantage functions to continuous-time MDPs. 

All reviewers had an overall positive view of this work, highlighting the importance of the problem being tackled by the authors and the thoroughness of the presented theory. ZgER praised the relatively extensive experiments conducted by the authors. Reviewer ZgER suggested that the empirical analyses presented in this work be strengthened by testing the proposed idea across additional decision frequencies, which the authors did. Reviewers brought up a few (relatively minor) points of contention, and the discussion phase was fruitful: the authors did address the most important concerns mentioned in the reviews and pointed out concrete ways in which their work can and will be improved in terms of presentation and clarity.

The most common and pressing concern regarding this paper was its clarity. Although reviewers agreed that the writing is rigorous, the consensus is that the paper is not necessarily accessible to a broad audience due to the highly dense text and the authors not always discussing intuitions that underlie the ideas being investigated and proposed. This will most likely limit how accessible the contributions introduced in this paper are to the NeurIPS community. The reviewers strongly encouraged the authors to improve their document by adding further intuition and discussion and by improving clarity to ensure that the insights introduced in this work become more apparent to a broader audience of readers who may not be, a priori, experts on all theoretical concepts exploited in this paper.

Reviews: review: paper_summary: paper_summary: The paper investigates the issues around continuous-time distributional reinforcement learning. In traditional RL, the advantage becomes less informative as the frequency of actions increases, vanishing at the limit and making it impossible to distinguish between actions. This work extends this result to distributional RL in several ways: it first proposes a framework based on Wasserstein distances between return distributions, then proves in this framework that a similar problem appears in DLR, and gives tight bounds on the asymptotic convergence rate of the distgap (an action gap analogue for continuous DLR), establishing that the rate differs from the traditional RL case.

After that, the paper constructs a notion of superiority - a DLR analogue of advantage, and considers its different variants (transformations) that ensure that action gaps are preserved in the limit. Finally, two families of algorithms are constructed, and shown to outperform the baseline QR-DQN on a benchmark of high-frequency options trading task.
strength_weakness: strength_weakness: The theoretical part requires quite a lot of background from the reader. While authors give a short intro to SDEs in the appendix, it is, in my opinion, nowhere near the depth required to understand the paper. While it's not really possible to present a comprehensive intro to stochastic analysis, the paper would benefit, in my opinion, from a slight expansion of the relevant appendix. Also in terms of presentation, I found the explanation of the algorithmic part of the paper really dense and confusing, and I cannot say I understood it particularly well from the main text.
questions: questions: 1. The Wasserstein distances considered are for $p \in [1, \infty)$. What breaks if we put $p = \infty$?
2. Why is the behavior of $\vartheta$ mean non-monotone wrt frequency in the example considered in sec 5.1?
3. Apart form distributional RL, could a similar method be applied to continuous-time max-ent RL?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper investigates the action gap in distributional RL, where the decisions are made at high frequency. The authors showed that the distributional RL is also sensitive to the decision frequency. In particular, they proved the action-conditioned return distribution collapse with different rates for the statistics. Also, they introduce the generalization of the advantage function in continuous-time MDP, i.e., superiority, based on which a novel algorithm is designed. The authors finally conducted extensive experiments in the high-frequency option-trading domain to demonstrate their theory and the efficacy of their algorithms.
strength_weakness: strength_weakness: * The motivation can be strengthened. What is an action gap in continuous-time MDP, and what is an algorithm's outcome when encountering high-frequency decision-making settings? Why do we investigate the distributional RL in this setting? More explanation would be better.

* The clarity can be enhanced. Although I appreciate the rigorous writing, it would be better to emphasize the main conclusions of this paper. For instance, the statistics in distributional RL also decrease at different rates. How do readers learn from these observations? Some rigorous theorems may be put in the appendix and replaced with more straightforward explanations.
questions: questions: 1. While the authors stated that DAU+DSUP(1/2) is inferior to DSUP(1/2), it is indeed a clear drawback of the proposed algorithm as this empirical result is consistent with the theory stated in Section 4.2, and hard to understand to me. Although the authors gave some explanations, they did not rigorously validate their hypothesis. Also, I am not convinced why DSUP(1/2) performs poorly in low-frequency domains. 

2. Why only consider 30HZ and DSUP(1/2) in the risk-sensitive setting? My suggestion is to include DAU+DSUP(1/2) as well for an extensive comparison across different frequencies, which could be more convincing. 

3. It is suggested that more examples be given to emphasize the significance of studying continuous-time MDP and the motivation to explore distributional RL.

4. It would be better to provide the pointers to the proof of each theorem to allow readers to check the proof instantly.

5. In what sense does the approximation hold in Eq. 4.2?
limitations: limitations: yes
paper_summary: paper_summary: This paper focuses on the scenario of continuous RL with high decision frequency. They show that it is hard for distributional RL agents to accurately estimate action values when the decision frequency increases, just like in the case of ordinary RL agents. They propose a distributional analogy of the action gap in ordinary RL called superiority. Based on this notion they try to mitigate this issue by the rescaling technique and propose a superiority-based algorithm. They also validate the proposed methods by numerical simulations.
strength_weakness: strength_weakness: * It seems that the basic idea of the superiority distribution is that for two distributions $\eta_1,\eta_2$, we define an object $\Delta$ that can be viewed as the difference between $\eta_1$ and $\eta_2$. And then given a risk-sensitive objective $\phi$, one may choose actions according to rescaled $\phi(\Delta/h^q)$, where $h$ denotes the decision frequency. But why not simply use the criterion $(\phi(\eta_1)-\phi(\eta_2))/h^q$ for decision making? I think the authors should discuss this issue in later versions of this paper.

* I suggest the author add some explanations about the problem setting to the intro section, which would make the paper more readable for those not familiar with the specific field.
questions: questions: * I note that the superior distributions are actually rescaled by $h^{-q}$, where $q$ is a tuning parameter. In the empirical studies, $q$ is set to be $1$ or $1/2$. Are there any possible heuristic rules for the choice of $q$?

* It seems that in the simulation studies, the performances of some methods (DAU, DSUP(1), DAU+DSUP(1/2)) are not monotone w.r.t. the decision frequency, but oscillate a lot as the decision frequency increases. Why would this happen? Can the authors give a more detailed explanation?
limitations: limitations: It seems the authors do not explicitly addressed the limitations of their work.

Comments: title: Response
comment: Thanks for the response. All of my major concerns are properly addressed. So I decide to keep my positive rating of this paper.
Thanks for clearing that up! Indeed we did misinterpret the original question.

The issue with your suggestion is that it still involves learning $\eta_1$ itself. This is what we want to avoid: as $h$ decreases, the distributions $\{\eta_1(x, a)\}_{a\in\mathcal{A}}$ will be all roughly equal in the $W_p$ metrics (Theorems 3.5 and 3.7) – the distributional action gap is small. Therefore, $\arg\max_a(\phi(\eta_1(x, a)) - \phi(\eta_2(x)))$ is going to be highly corrupted by approximation error.

Your suggestion is actually equivalent to the QR-DQN baseline in Figure 5.4 (and Figure 5.2 for the risk-neutral case). Note that, in your example, $\eta_2$ is a function of state only, so it doesn’t affect the ranking of actions. Likewise, scaling $\phi(\eta_1(x, a))$ or $(\phi(\eta_1(x, a)) - \phi(\eta_2(x)))$ by $h^{-q}$ will not change the ranking of actions. Estimating $\eta_1$ and acting according to $\phi(\eta_1(x, a))$ (which is the same as acting according $\phi(\eta_1(x, a)) - \phi(\eta_2(x))$) is precisely what the QR-DQN baseline is doing in Figure 5.4, and we see that this struggles (again, due to the vanishing distributional action gap).

Our method circumvents this by learning the rescaled superiority directly. That is, rather than modeling $\eta_1$ and $\eta_2$ and constructing $\Delta/h^{q}$ from those, we directly model $\Delta/h^{q}$. For instance, see equation 4.1 or Algorithm 1: we never model $\zeta(x, a)$ (which is $\eta_1$ in your example), we only model the rescaled superiority $\psi_{h;1/2}(x, a)$. This preserves the distributional action gap as shown by Theorem 4.8 (and Figure 5.3 for an empirical demonstration), preserves the correct ranking of actions as shown by Theorem 4.10, and performs much better empirically as a consequence, as shown in Figures 5.2 and 5.4.
I want to thank the authors for the detailed response. However, I think the authors may have misunderstood my point on the difference between considering $\phi(\eta_1) - \phi(\eta_2)$ and $\phi(\Delta/h^q)$. Here the idea is not to directly estimate $\phi(\eta_1) - \phi(\eta_2)$ but to first estimate $\eta_1$ and $\eta_2$ (with existing distributional RL techniques) and then make decisions according to $\phi(\eta_1) - \phi(\eta_2)$. I think this method is conceptually simpler than the methodology proposed in the paper. I hope the authors can provide reasons why they choose to adopt the (seemingly more complexi) proposed methodology.
I thank the authors for their thorough response. Re: specific additional suggestions: I agree with the other reviewers, that adding heuristic, natural language explanations would benefit the paper. At the same time, given the space constraints, I understand it might be difficult to maintain the level of the mathematical rigour, which I value more highly.

When reading the paper, I looked at the Algorithm 1 in the Appendix C, but could not follow the details - I would recommend either splitting it into sub-procedures, or otherwise giving a higher-level abstraction description, because right now it is a page-long wall-of-text, which makes comprehending it a quite daunting task.

I decided to keep my original (positive) rating.
Thanks for the authors' response. Although the paper can be further enhanced by improving the motivation and strengthening the experiments, I believe it is an interesting supplement to distributional RL literature in terms of continuous time and has great potential. Thus, I keep the positive assessment.
Thank you again for your time and thoughts. We’re keen to discuss our work with you, especially in light of your insightful comments. If you have any remaining questions, we would be happy to address them; please let us know.