Title: Statistical Efficiency of Distributional Temporal Difference Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Distributional Reinforcement Learning

Distributional Temporal Difference Learning

Sample Complexity
Abstract: Distributional reinforcement learning (DRL) has achieved empirical success in various domains.
One of the core tasks in the field of DRL is distributional policy evaluation, which involves estimating the return distribution $\eta^\pi$ for a given policy $\pi$.
The distributional temporal difference learning has been accordingly proposed, which
is an extension of the temporal difference learning (TD) in the classic RL area.
In the tabular case,  Rowland et al. [2018] and Rowland et al. [2023] proved the asymptotic convergence of two instances of distributional TD, namely categorical temporal difference learning (CTD) and quantile temporal difference learning (QTD), respectively.
In this paper, we go a step further and analyze the finite-sample performance of distributional TD.
To facilitate theoretical analysis, we propose a non-parametric distributional TD learning (NTD).
For a $\gamma$-discounted infinite-horizon tabular Markov decision process,
we show that for NTD we need $\widetilde O\left(\frac{1}{\varepsilon^{2p}(1-\gamma)^{2p+1}}\right)$ iterations to achieve an $\varepsilon$-optimal estimator with high probability, when the estimation error is measured by the $p$-Wasserstein distance.
This sample complexity bound is minimax optimal (up to logarithmic factors) in the case of the $1$-Wasserstein distance.
To achieve this, we establish a novel Freedman's inequality in Hilbert spaces, which would be of independent interest.
In addition, we revisit CTD, showing that the same non-asymptotic convergence bounds hold for CTD in the case of the $p$-Wasserstein distance.

Statistical Efficiency of Distributional Temporal Difference Learning

Abstract

Distributional reinforcement learning (DRL) has achieved empirical success in various domains. One core task in the field of DRL is distributional policy evaluation, which involves estimating the return distribution η π for a given policy π . The distributional temporal difference learning has been accordingly proposed, which is an extension of the temporal difference learning (TD) in the classic RL area. In the tabular case, Rowland et al. [2018] and Rowland et al. [2024a] proved the asymptotic convergence of two instances of distributional TD, namely categorical temporal difference learning (CTD) and quantile temporal difference learning (QTD), respectively. In this paper, we go a step further and analyze the finitesample performance of distributional TD. To facilitate theoretical analysis, we propose non-parametric distributional temporal difference learning (NTD). For a γ -discounted infinite-horizon tabular Markov decision process, we show that for NTD we need ˜ O ( 1 ε 2 p (1 -γ ) 2 p +1 ) iterations to achieve an ε -optimal estimator with high probability, when the estimation error is measured by the p -Wasserstein distance. This sample complexity bound is minimax optimal up to logarithmic factors in the case of the 1 -Wasserstein distance. To achieve this, we establish a novel Freedman's inequality in Hilbert spaces, which would be of independent interest. In addition, we revisit CTD, showing that the same non-asymptotic convergence bounds hold for CTD in the case of the p -Wasserstein distance for p ≥ 1 .

1 Introduction

In high-stake applications of reinforcement learning (RL), such as healthcare [Lavori and Dawson, 2004, Böck et al., 2022] and finance[Ghysels et al., 2005], only considering the mean of returns is insufficient. It is necessary to take risk and uncertainties into consideration. Distributional reinforcement learning (DRL) Morimura et al. [2010], Bellemare et al. [2017, 2023] addresses such issues by modeling the complete distribution of returns instead of their expectations.

In the field of DRL, one of the most fundamental tasks is to estimate the return distribution η π for a given policy π , which is referred to as distributional policy evaluation. Distributional temporal difference learning (TD) is probably the most widely-used approach for solving the distributional policy evaluation problem. A key aspect of implementing a distributional TD algorithm is how to represent the return distribution, an infinite-dimensional object, via a computationally feasible finite-dimensional parametrization. This has led to the development of two special instances of distributional TD: categorical temporal difference learning (CTD) [Bellemare et al., 2017] and quantile temporal difference learning (QTD) [Dabney et al., 2018]. These algorithms provide computationally tractable parametrizations and updating schemes of the return distribution.

Previous theoretical works have primarily focused on the asymptotic behaviors of distributional TD. In particular, Rowland et al. [2018] and Rowland et al. [2024a] showed the asymptotic convergences of CTD and QTD in the tabular case, respectively. A natural question arises: can we depict the statistical efficiency of distributional TD by non-asymptotic results similar to the classic TD algorithm [Li et al., 2024] ?

1.1 Contributions

In this paper, we manage to answer the above question affirmatively in the synchronous setting

[Kakade, 2003, Kearns et al., 2002]. Firstly, we introduce non-parametric distributional temporal difference learning (NTD) in Section 3, which is not practical but aids theoretical understanding. We show that ˜ O ( 1 ε 2 p (1 -γ ) 2 p +1 ) 4 iterations are sufficient to yield an estimator ˆ η π , such that the p -Wasserstein metric between ˆ η π and η π is less than ε with high probability (Theorem 4.1). This bound is minimax optimal (Theorem B.1) in the 1 -Wasserstein metric case, if we neglect all logarithmic terms. Next, we revisit the more practical CTD, and show that, in terms of the p -Wasserstein metric, CTD and NTD have the same non-asymptotic convergence bounds (Theorem 4.2). It is worth pointing out that to attain such tight bounds in Theorem 4.1, we establish a Freedman's inequality in Hilbert spaces (Theorem A.2). We would believe it is of independent interest beyond the current work.

1.2 Related Work

Non-asymptotic results of DRL. Recently, there has been an emergence of work focusing on finite-sample/iteration results of the distributional policy evaluations.

Wu et al. [2023] studied the offline distributional policy evaluation problem. They solved the problem via fitted likelihood estimation (FLE) inspired by the classic offline policy evaluation algorithm fitted Q evaluation (FQE), and provided a generalization bound in the p -Wasserstein metric case.

Zhang et al. [2023] proposed to solve distributional policy evaluation by the model-based approach and derived corresponding sample complexity bounds, namely O ( 1 ε 2 p (1 -γ ) 2 p +2 ) in the

˜ p -Wasserstein metric case, and ˜ O ( 1 ε 2 (1 -γ ) 4 ) in both the Kolmogorov-Smirnov metric and total variation metric under different conditions. Rowland et al. [2024b] proposed direct categorical fixedpoint computation (DCFP), a model-based version of CTD, in which they constructed the estimator by solving a linear system directly instead of performing an iterative algorithm. They showed that the sample complexity of DCFP is ˜ O ( 1 ε 2 (1 -γ ) 3 ) in the 1 -Wasserstein metric case by introducing the novel stochastic categorical CDF Bellman operator and equation. Their result matches the minimax lower bound (up to logarithmic factors) ˜ Ω ( 1 ε 2 (1 -γ ) 3 ) proposed in [Zhang et al., 2023], which implies that learning the full return distribution can be as sample-efficient as learning just its expectation. It's worth noting that the algorithms analyzed in both [Zhang et al., 2023] and [Rowland et al., 2024b] are model-based, hence they are less similar to practical algorithms. While distributional TD analyzed in this paper, as a model-free method, is more practical, and also involves a more complicated theoretical analysis.

Böck and Heitzinger [2022] also considered model-free method. They proposed speedy categorical policy evaluation (SCPE), which can be regarded as CTD with an additional acceleration term. They showed that the sample complexity of SCPE is ˜ O ( 1 ε 2 (1 -γ ) 4 ) in the 1 -Wasserstein metric case. Compared to [Böck and Heitzinger, 2022], our work shows that even if we do not introduce any acceleration techniques to the original CTD algorithm, it is still possible to attain the near-minimax optimal sample complexity bounds. Thus, we give sharper bounds based on a simpler algorithm.

Table 1 gives more detailed comparisons of sample complexity with the previous work in the 1 -Wasserstein metric. Note that solving distributional policy evaluation can also address the traditional

(

)

policy evaluation task by taking expectation of the return distribution estimator. And the supreme 1 -Wasserstein metric error of the return distribution estimator is not smaller than the /lscript ∞ error of the induced value function estimator (see the proof of Theorem B.1 in Appendix B), we have also listed the sample complexity of the policy evaluation task in Table 1 for comparison.

Freedman's inequality. Freedman's inequality was originally proposed in [Freedman, 1975]. It can be viewed as a Bernstein's inequality for martingales, which is crucial for analyzing stochastic approximation algorithms. Tropp [2011] generalized Freedman's inequality to matrix martingales. And Talebi et al. [2022] established Freedman inequalities for martingales in the setting of noncommutative probability spaces. The closest literature to ours is [Tarres and Yao, 2014] and [Martinez-Taboada and Ramdas, 2024], where they provided a special case of our Theorem A.2 with H = 0 independently. When H = 0 , we can only utilize the deterministic upper bound on the quadratic variation rather than the high-probability upper bound. In certain problems, such as the distributional TD learning we aim to investigate, it is impossible to achieve the optimal upper bound using the H = 0 version. [Martinez-Taboada and Ramdas, 2024] also proposed an empirical Freedman's inequality in (2 , D ) -smooth Banach space, which can be used to construct confidence sets or perform hypothesis testing. To the best of our knowledge, we are the first to present this version (Theorem A.2) of Freedman's inequality in Hilbert spaces 5 .

The remainder of this paper is organized as follows. In Section 2, we introduce some background of DRL and state Freedman's inequality in Hilbert spaces. In Section 3, we revisit distributional TD and propose NTD for further theoretical analysis. In Section 4, we analyze the non-asymptotic convergence bounds of NTD and CTD. Section 5 presents proof outlines of our theoretical results, and Section 6 concludes our work. We put the detailed results with Freedman's inequality in Hilbert spaces in Appendix A, and the minimax lower bound of the distributional policy evaluation task in Appendix B.

2 Background

An infinite-horizon tabular Markov decision process (MDP) is defined by a 5-tuple M = 〈S , A , P R , P, γ 〉 , where S represents a finite state space, A a finite action space, P R the distribution of rewards, P the transition dynamics, i.e. , P R ( ·| s, a ) ∈ ∆([0 , 1]) , P ( ·| s, a ) ∈ ∆( S ) for any state action pair ( s, a ) ∈ S × A , and γ ∈ (0 , 1) a discount factor. Here we use ∆( · ) to represent the set of all probability distributions over some set. Given a policy π : S → ∆( A ) and an initial state s 0 = s ∈ S , a random trajectory { ( s t , a t , t t ) ∞ t =0 } can be sampled from M : a t | s t ∼ π ( · | s t ) , r t | ( s t , a t ) ∼ P R ( · | s t , a t ) , s t +1 | ( s t , a t ) ∼ P ( · | s t , a t ) for any t ∈ N . Given a trajectory, we define the return by G π ( s ) := ∑ ∞ t =0 γ t r t ∈ [ 0 , 1 1 -γ ] . We denote return distribution η π ( s ) as the probability distribution of G π ( s ) , and η π := ( η π ( s )) s ∈S . The expected return V π ( s ) = E G π ( s ) is the value function in the traditional RL setting.

2.1 Distributional Bellman Equation and Operator

Recall that the classic policy evaluation aims at computing the value functions V π . It is known that V π = ( V π ( s )) s ∈S satisfy the Bellman equation. That is, for any s ∈ S ,

The operator T π : R S → R S is called the Bellman operator, and V π is a fixed point of T π .

The task of distribution policy evaluation is finding η π given some fixed policy π . η π satisfies a distributional version of the Bellman equation (1). That is, for any s ∈ S , where b r,γ : R → R is an affine function defined by b r,γ ( x ) = r + γx . And f # µ is the push forward measure of µ through any function f : R → R , so that f # µ ( A ) = µ ( f -1 ( A )) for any Borel set A , where f -1 ( A ) := { x : f ( x ) ∈ A } . The operator T π : ∆ ([ 0 , 1 1 -γ ]) S → ∆ ([ 0 , 1 1 -γ ]) S is known as the distributional Bellman operator, and η π is a fixed point of T π . For notational simplicity, we denote ∆ ([ 0 , 1 1 -γ ]) as P from now on.

2.2 T π as Contraction in P

A key property of the Bellman operator T π is that it is a γ -contraction w.r.t. the supreme norm ( i.e. /lscript ∞ norm). However, before we can properly discuss the contraction properties of T π , we need to specify a metric d on P . And for any metric d on P , we denote ¯ d as the corresponding supreme metric on P S , i.e. , ¯ d ( η, η ′ ) := max s ∈S d ( η ( s ) , η ′ ( s )) for any η, η ′ ∈ P S .

Suppose µ and ν are two probability distributions on R with finite p -moments for p ∈ [1 , ∞ ] . The p -Wasserstein metric between µ and ν is defined as W p ( µ, ν ) := ( inf κ ∈ Γ( µ,ν ) ∫ R 2 | x -y | p κ ( dx, dy ) ) 1 /p . Each element κ ∈ Γ( µ, ν ) is a coupling of µ and ν , i.e. , a joint distribution on R 2 with prescribed marginals µ and ν on each 'axis.' When p = 1 we have W 1 ( µ, ν ) = ∫ R | F µ ( x ) -F ν ( x ) | dx , where F µ and F ν are the cumulative distribution function of µ and ν , respectively. It can be shown that T π is a γ -contraction w.r.t. the supreme p -Wasserstein metric ¯ W p .

Proposition 2.1. [Bellemare et al., 2023, Propositions 4.15] The distributional Bellman operator is a γ -contraction on P S w.r.t. the supreme p -Wasserstein metric for p ∈ [1 , ∞ ] . That is, for any η, η ′ ∈ P S , we have ¯ W p ( T π η, T π η ′ ) ≤ γ ¯ W p ( η, η ′ ) .

Proposition 2.2. [Bellemare et al., 2023, Propositions 4.20] The distributional Bellman operator is a γ 1 p -contraction on P S w.r.t. the supreme /lscript p metric for p ∈ [1 , ∞ ) . That is, for any η, η ′ ∈ P S , we have ¯ /lscript p ( T π η, T π η ′ ) ≤ γ 1 p ¯ /lscript p ( η, η ′ ) .

The /lscript p metric between µ and ν is defined as /lscript p ( µ, ν ) = (∫ R | F µ ( x ) -F ν ( x ) | p dx ) 1 p for p ∈ [1 , ∞ ) , and T π is γ 1 p -contraction w.r.t. the supreme /lscript p metric ¯ /lscript p .

Note that the /lscript 1 metric coincides with the 1 -Wasserstein metric. And the /lscript 2 metric is also called the Cramér metric, which plays an important role in subsequent analysis because the zero-mass signed measure space equipped with this metric ( M , ‖·‖ /lscript 2 ) (defined in Section 5.1) is a Hilbert space 6 . Thereby, we can apply Freedman's inequality in Hilbert spaces.

2.3 Freedman's Inequality in Hilbert Spaces

Just as Freedman's inequality is essential for the theory of TD (Theorem 1 in [Li et al., 2024]), a Hilbert space version of Freedman's inequality is indispensable for deriving the minimax nonasymptotic convergence bound for distributional TD. At the moment, we state a Hilbert space version of the original Freedman's inequality (Theorem 1.6 in [Freedman, 1975]), and more detailed results can be found in Appendix A.

Let X be a Hilbert space, { X i } n i =1 be an X -valued martingale difference sequence adapted to the filtration {F i } n i =1 , Y i := ∑ i j =1 X j be the corresponding martingale, and W i := ∑ i j =1 σ 2 j be the corresponding quadratic variation process. Here σ 2 j := E j -1 ‖ X j ‖ 2 , and E i [ · ] := E [ ·|F i ] denotes the conditional expectation.

Theorem 2.1 (Freedman's inequality in Hilbert spaces) . Suppose max i ∈ [ n ] ‖ X i ‖ ≤ b for some constant b > 0 . Then, for any ε and σ > 0 , the following inequality holds

3 Distributional Temporal Difference Learning

If the MDP M = 〈S , A , P R , P, γ 〉 is known, and because V π is the fixed point of the contraction T π , V π can be evaluated via the famous dynamic programming (DP) algorithm. To be concrete, for any initialization V (0) ∈ R S , if we define the iteration sequence V ( k +1) = T π ( V ( k ) ) for k ∈ N , we have lim k →∞ ∥ ∥ V ( k ) -V π ∥ ∥ ∞ = 0 by the contraction mapping theorem (Proposition 4.7 in [Bellemare et al., 2023]).

In most application scenarios, the transition dynamic P and reward distribution P R are unknown, and instead we can only get samples of P and P R in a streaming manner. In this paper, we assume a generative model [Kakade, 2003, Kearns et al., 2002] is accessible, which generates independent samples for all states in each iteration, i.e. , in the t -th iteration, we collect sample a t ( s ) ∼ π ( ·| s ) , s t ( s ) ∼ P ( ·| s, a t ( s )) , r t ( s ) ∼ P R ( ·| s, a t ( s )) for each s ∈ S . Similar to TD [Sutton, 1988] in classic RL, distributional TD also employs the stochastic approximation (SA) [Robbins and Monro, 1951] technique to address the aforementioned problem and can be viewed as an approximate version of distributional DP.

Similarly, the distributional dynamic programming algorithm defines the iteration sequence as η ( k +1) = T π η ( k ) for any initialization η (0) . In the same way, we have lim k →∞ ¯ W p ( η ( k ) , η π ) = 0 for p ∈ [1 , ∞ ] and lim k →∞ ¯ /lscript p ( η ( k ) , η π ) = 0 for p ∈ [1 , ∞ ) .

Non-parametric Distributional TD Wefirst introduce non-parametric distributional temporal difference learning (NTD), which is helpful in the theoretical understanding of distributional TD. In the setting of NTD, we assume the return distributions can be precisely updated without any parametrization. For any initialization η π 0 ∈ P S , the updating scheme is given by for any t ≥ 1 . Here α t is the step size. The empirical Bellman operator at the t -th iteration T π t is defined as

which is an unbiased estimator of ( T π η ) ( s ) . It is evident that NTD is a SA modification of distributional DP. Consequently, we can analyze NTD using the techniques from the SA area.

Categorical Distributional TD Now, we revisit the more practical CTD. In this case, the updates in CTD is computationally tractable, due to the following categorical parametrization of probability distributions:

where K ∈ N , and 0 ≤ x 0 < · · · < x K ≤ 1 1 -γ are fixed points of the support. For simplicity, we assume { x k } K k =0 are equally-spaced, i.e. , x k = k K (1 -γ ) . We denote the gap between two points by ι K = 1 K (1 -γ ) . When updating the return distributions, we need to evaluate the /lscript 2 -projection of P K , Π K : P → P K , Π K µ := argmin ˆ µ ∈ P K /lscript 2 ( µ, ˆ µ ) . It can be shown (Proposition 5.14 in [Bellemare et al., 2023]) that the projection is uniquely given by

( x ) + := max { x, 0 } for any x ∈ R . It is known that Π K is non-expansive w.r.t. the Cramér metric (Lemma 5.23 in [Bellemare et al., 2023]), i.e. , /lscript 2 (Π K µ, Π K ν ) ≤ /lscript 2 ( µ, ν ) for any µ, ν ∈ P . For any η ∈ P S , s ∈ S , we slightly abuse the notation and define (Π K η ) ( s ) := Π K η ( s ) . Π K is still non-expansive w.r.t. ¯ /lscript 2 . Hence T π,K := Π K T π is a √ γ -contraction w.r.t. ¯ /lscript 2 , we denote its unique fixed point as η π,K ∈ P S K . The approximation error induced by categorical parametrization is given by (Proposition 3 in Rowland et al. [2018])

Now, we are ready to give the updating scheme of CTD, given any initialization η π 0 ∈ P S K ,

for any t ≥ 1 . We can find that the only difference between CTD and NTD lies in the additional application of the projection operator Π K at each iteration in CTD.

4 Statistical Analysis

In this section, we state our main results. For both NTD and CTD, we give the non-asymptotic convergence rates of ¯ W p ( η π T , η π ) and ¯ /lscript 2 ( η π T , η π ) , respectively.

4.1 Non-asymptotic Analysis of NTD

We first provide a non-asymptotic convergence rate of ¯ W 1 ( η π T , η π ) for NTD, which is minimax optimal (Theorem B.1) up to logarithmic factors.

Theorem 4.1 (Sample complexity of NTD in the 1 -Wasserstein metric) . Given any δ ∈ (0 , 1) and ε ∈ (0 , 1) , let the initialization be η π 0 ∈ P S , the total update number T satisfy

for some large universal constant C 1 > 1 , i.e. , T = ˜ O ( 1 ε 2 (1 -γ ) 3 ) , and the step size α t satisfy for some small universal constants c 2 > c 3 > 0 . Then, with probability at least 1 -δ , the last iterate estimator satisfies ¯ W 1 ( η π T , η π ) ≤ ε .

Because ¯ W 1 ( η π T , η π ) ≤ 1 1 -γ always holds, we can translate the high probability bound to a mean error bound, that is, if we take δ ≤ ε (1 -γ ) . In the subsequent discussion, we will not state the mean error bound conclusions for the sake of brevity.

The key idea of our proof is to first expand the error term ¯ W 1 ( η π T , η π ) over the time steps. Then it can be decomposed into an initial error term and a martingale term. The initial error term becomes smaller as the iteration goes due to the contraction properties of T π . To control the martingale term, we first use the basic inequality (Lemma E.1) W 1 ( µ, ν ) ≤ 1 √ 1 -γ /lscript 2 ( µ, ν ) , which allows us to analyze this error term in the Hilbert space ( M , ‖·‖ /lscript 2 ) defined in Section 5.1. Consequently, we can bound it using Freedman's inequality in the Hilbert space (Theorem A.2). A more detailed outline of proof can be found in Section 5.2.

Combining Theorem 4.1 with the basic inequality ¯ W p ( η, η ′ ) ≤ 1 (1 -γ ) 1 -1 p ¯ W 1 p 1 ( η, η ′ ) for any η, η ′ ∈

P S (Lemma E.1), we can derive that T = ˜ O ( 1 ε 2 p (1 -γ ) 2 p +1 ) iterations are sufficient to ensure

¯ W p ( η π T , η π ) ≤ ε . As pointed out in the example after Corollary 3.1 in [Zhang et al., 2023], when p > 1 , the slow rate in terms of ε is inevitable without additional regularity conditions.

Although the 1 -Wasserstein metric cannot bound the Cramér metric properly, by making slight modifications to the proof we have the following non-asymptotic convergence rate of ¯ /lscript 2 ( η π T , η π ) . See Appendix C.5 for our proof.

Corollary 4.1 (Sample complexity of NTD in the Cramér metric) . Given any δ ∈ (0 , 1) and ε ∈ (0 , 1) , let the initial value η π 0 ∈ P S , the total update number T satisfy for some large universal constant C 1 > 1 , i.e. , T = O ( 1 ε 2 (1 -γ ) 5 / 2 ) , and the step size α t satisfy

for some small universal constants c 2 > c 3 > 0 . Then, with probability at least 1 -δ , the last iterate estimator satisfies ¯ /lscript 2 ( η π T , η π ) ≤ ε .

4.2 Non-asymptotic Analysis of CTD

We first state a parallel result to Theorem 4.1.

Theorem 4.2 (Sample complexity of CTD in the 1 -Wasserstein metric) . Given any δ ∈ (0 , 1) and ε ∈ (0 , 1) , suppose K > 4 1 -γ , the initial value η π 0 ∈ P S K , the total update number T satisfies for some large universal constant C 1 > 1 , i.e. , T = ˜ O ( 1 ε 2 (1 -γ ) 3 ) , and the step size α t satisfies

for some small universal constants c 2 > c 3 > 0 . Then, with probability at least 1 -δ , the last iterate estimator satisfies ¯ W 1 ( η π T , η π,K ) ≤ ε 2 . Furthermore, according to the upper bound (3) of the approximation error ¯ W 1 ( η π,K , η π ) , if we take K > 4 ε 2 (1 -γ ) 3 , we have ¯ W 1 ( η π T , η π ) ≤ ε .

Note that the order (modulo logarithmic factors) of sample complexity of CTD is better than the previous results of SCPE [Böck and Heitzinger, 2022], and we do not need the additional term introduced in the updating scheme of SCPE.

The proof of this theorem is almost the same as that of Theorem 4.1, we outline the proof in Section 5.2. The ¯ W 1 metric result can be translated into sample complexity bound ˜ O ( 1 ε 2 p (1 -γ ) 2 p +1 ) in the ¯ W p metric. We comment that this theoretical result matches the sample complexity bound in the model-based setting [Rowland et al., 2024b].

As in the NTD setting, we have the following non-asymptotic convergence rate of ¯ /lscript 2 ( η π T , η π ) as a corollary of Theorem 4.2. See Appendix C.5 for the proof.

Corollary 4.2 (Sample complexity of CTD in the Cramér metric) . For any given δ ∈ (0 , 1) and ε ∈ (0 , 1) , suppose K > 4 1 -γ , the initialization is η π 0 ∈ P S K , the total update number T satisfies for some large universal constant C 1 > 1 , i.e. , T = O ( 1 ε 2 (1 -γ ) 5 / 2 ) , and the step size α t satisfies

for some small universal constants c 2 > c 3 > 0 . Then, with probability at least 1 -δ , the last iterate estimator satisfies ¯ /lscript 2 ( η π T , η π,K ) ≤ ε 2 . Furthermore, according to the upper bound (3) of the approximation error ¯ /lscript 2 ( η π,K , η π ) , if we take K > 4 ε 2 (1 -γ ) 2 , we have ¯ /lscript 2 ( η π T , η π ) ≤ ε .

5 Proof Outlines

In this section, we will outline the proofs of our main theoretical results (Theorem 4.1, Corollary 4.1, Theorem 4.2, and Corollary 4.2). Before diving into the details of the proofs, we first define some notation.

5.1 Zero-mass Signed Measure Space

To analyze the distance between the estimator and the ground-truth η π , we will work with the zeromass signed measure space M defined as follows where | µ | is the total variation measure of µ , and supp ( µ ) is the support of µ . See [Bogachev, 2007] for more details about signed measures.

For any µ ∈ M , we define its cumulative function as F µ ( x ) := µ [0 , x ) . We can check that F µ is linear w.r.t. µ , that is, F αµ + βν = αF µ + βF ν for any α, β ∈ R , µ, ν ∈ M .

To analyze the Cramér metric case, we define the following Cramér inner product on M :

It is easy to verify that 〈· , ·〉 /lscript 2 is indeed an inner product on M . The corresponding norm, called the Cramér norm, is given by ‖ µ ‖ /lscript 2 = √ 〈 µ, µ 〉 /lscript 2 = √ ∫ 1 1 -γ 0 ( F µ ( x )) 2 dx . We have ν 1 -ν 2 ∈ M and ‖ ν 1 -ν 2 ‖ /lscript 2 = /lscript 2 ( ν 1 , ν 2 ) for any ν 1 , ν 2 ∈ P .

We can extend the distributional Bellman operator T π and the Cramér projection operator Π K naturally to M S . Here, the product space M S is also a Banach space, and we use the supreme norm: ‖ η ‖ ¯ /lscript 2 := max s ∈S ‖ η ( s ) ‖ /lscript 2 , and ‖ η ‖ ¯ W 1 := max s ∈S ‖ η ( s ) ‖ W 1 for any η ∈ M S . We denote by I the identity operator in M S .

The W 1 norm on M is defined as ‖ µ ‖ W 1 := ∫ 1 1 -γ 0 | F µ ( x ) | dx . Wehave ‖ ν 1 -ν 2 ‖ W 1 = W 1 ( ν 1 , ν 2 ) for any ν 1 , ν 2 ∈ P .

When the norm ‖·‖ is applied to A ∈ L ( X ) , where X is any Banach space, and L ( X ) is the space of all bounded linear operators in X , we refer ‖ A ‖ to the operator norm of A , which is defined as ‖ A ‖ := sup η ∈X , ‖ η ‖ =1 ‖ Aη ‖ . With this notation, L ( X ) = { A : A is a linear operator mapping from X to X , and ‖ A ‖ < ∞} .

Proposition 5.1. T π and Π K are linear operators in M S . Furthermore, ‖T π ‖ ¯ /lscript 2 ≤ √ γ , ‖T π ‖ ¯ W 1 ≤ γ , ‖ Π K ‖ ¯ /lscript 2 = 1 , and ‖ Π K ‖ ¯ W 1 ≤ 1 .

The proof of the last inequality can be found in the proof of Lemma C.4, while the remaining results are trivial. We omit the proofs for brevity.

Moreover, we have the following matrix (of operators) representations of T π and Π K : T π ∈ L ( M ) S×S for any η ∈ M S , where T π ( s, s ′ ) ∈ L ( M ) for any ν ∈ M ,

It can be verified that ‖T ( s, s ′ ) ‖ /lscript 2 ≤ √ γ ∑ a ∈A π ( a | s ) P ( s ′ | s, a ) =: √ γP π ( s ′ | s ) . Similarly, ‖T ( s, s ′ ) ‖ W 1 ≤ γP π ( s ′ | s ) , and Π K = diag ( Π K ∣ ∣ M ) s ∈S ∈ L ( M ) S×S . With these representations, Π K T π ∈ L ( M ) S×S can be interpreted as matrix multiplication, where the scalar multiplication is replaced by the composition of operators. It can be verified that (Π K T π ) ( s, s ′ ) = Π K T π ( s, s ′ ) , and ‖ (Π K T π ) ( s, s ′ ) ‖ /lscript 2 ≤ √ γP π ( s ′ | s ) .

Remark 1: In Lemma E.2, we show that both ( M , ‖·‖ /lscript 2 ) and ( M , ‖·‖ W 1 ) are separable. And in Lemma E.3, we show that ( M , ‖·‖ W 1 ) is not complete. To resolve this problem, we will use their completions to replace them without loss of generality, because the completeness property does not affect the separability. For simplicity, we still use M to denote the completion space. According to the BLT theorem [Theorem 5.19 Hunter and Nachtergaele, 2001], any bounded linear operator can be extended to the completion space, and still preserves its operator norm.

5.2 Analysis of Theorems 4.1 and 4.2

For simplicity, we abbreviate both ‖·‖ ¯ /lscript 2 and ‖·‖ /lscript 2 as ‖·‖ in this part. For all t ∈ [ T ] := { 1 , 2 , · · · , T } , we denote T t := T π t , T := T π , η := η π for NTD; T t := Π K T π t , T := Π K T π , η := η π,K for CTD; and η t := η π t , ∆ t := η t -η ∈ M S for both NTD and CTD. According to Lemma E.4, η t ∈ P S for NTD and η t ∈ P S K for CTD. Our goal is to bound the ¯ W 1 norm of the error term ‖ ∆ T ‖ ¯ W 1 . This can be achieved by bounding ‖ ∆ T ‖ , as ‖ ∆ T ‖ ¯ W 1 ≤ 1 √ 1 -γ ‖ ∆ T ‖ .

According to the updating rule, we have the error decomposition

Applying it recursively, we can further decompose the error into two terms

where ∏ t k =1 A k is defined as A t A t -1 · · · A 1 for any operators or matrices { A k } t k =1 throughout the paper. Term (I) is an initial error term that becomes negligible when T is large because T is a contraction. Term (II) can be bounded via Freedman's inequality in the Hilbert space (Theorem A.2). Combining the two upper bound, we can establish a recurrence relation. Solving this relation will lead to the conclusion.

We first establish the conclusion for step sizes that depend on T . Specifically, we consider

where c 5 > c 6 > 0 are small constants satisfying c 5 c 6 ≤ 1 8 , and C 4 > 1 is a large constant depending only on c 5 and c 6 . As shown in Appendix C.1, once we have established the conclusion in this setting, we can recover the original conclusion stated in the theorem.

Now, we introduce the following useful quantities involving step sizes and γ

The following lemma provides useful bounds for β ( t ) k .

Lemma 5.1. Suppose c 5 c 6 ≤ 1 8 . Then, for all t ≥ T c 6 log T , we have that

The proof can be found in Appendix C.2. From now on, we only consider t ≥ T c 6 log T .

The upper bound of term (I) is given by

As for term (II), we have the following upper bound with high probability by utilizing Freedman's inequality (Theorem A.2).

Lemma 5.2. For any δ ∈ (0 , 1) , with probability at least 1 -δ , we have for all t ≥ T c 6 log T , in the NTD case,

The conclusion still holds for the CTD case if we take K ≥ 4 ε 2 (1 -γ ) 2 +1 .

The proof can be found in Appendix C.3. Combining the two results, we find the following recurrence relation in terms of the ¯ W 1 norm holds given the choice of T , with probability at least 1 -δ , for all t ≥ T c 6 log T

In Theorem C.1, we solve the relation and obtain the error bound of the last iterate estimator:

where C 7 > 1 is a large universal constant depending on c 6 . Now, we can obtain the conclusion if taking C 4 ≥ 2 C 2 7 and T ≥ C 4 log 3 T ε 2 (1 -γ ) 3 log |S| T δ .

6 Conclusions

In this paper we have studied the statistical performance of the distributional temporal difference learning (TD) from a non-asymptotic perspective. Specifically, we have considered two instances of distributional TD, namely, the non-parametric distributional TD (NTD) and the categorical distributional TD (CTD). For both NTD and CTD, we have shown that ˜ O ( 1 ε 2 p (1 -γ ) 2 p +1 ) iterations are sufficient to achieve a p -Wasserstein ε -optimal estimator, which is minimax optimal (up to logarithmic factors). We have established a novel Freedman's inequality in Hilbert spaces to prove these theoretical results, which has independent theoretical value beyond the current work. We leave the details to Appendix A.

Acknowledgments and Disclosure of Funding

This work has been supported by the National Key Research and Development Project of China (No. 2022YFA1004002), the National Natural Science Foundation of China (No. 12271011 and No. 12350001), and the MOE Project of Key Research Institute of Humanities and Social Sciences (No.22JJD110001).

References

G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International conference on machine learning , pages 449-458. PMLR, 2017.

G. Bellemare, W. Dabney, and M. Rowland. Distributional Reinforcement Learning . MIT Press, 2023. http://www.distributional-rl.org .

Böck and C. Heitzinger. Speedy categorical distributional reinforcement learning and complexity analysis. SIAM Journal on Mathematics of Data Science , 4(2):675-693, 2022. doi: 10.1137/ 20M1364436. URL https://doi.org/10.1137/20M1364436 .

Böck, J. Malle, D. Pasterk, H. Kukina, R. Hasani, and C. Heitzinger. Superhuman performance on sepsis mimic-iii data by distributional reinforcement learning. PLoS One , 17(11):e0275358, 2022.

I. Bogachev. Measure theory , volume 1. Springer, 2007.

Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

H. de la Pena. A general class of exponential inequalities for martingales and ratios. The Annals of Probability , 27(1):537-564, 1999.

Durrett. Probability: theory and examples , volume 49. Cambridge university press, 2019.

A. Freedman. On tail probabilities for martingales. The Annals of Probability , pages 100-118, 1975.

Gheshlaghi Azar, R. Munos, and H. J. Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning , 91:325-349, 2013.

Ghysels, P. Santa-Clara, and R. Valkanov. There is a risk-return trade-off after all. Journal of financial economics , 76(3):509-548, 2005.

K. Hunter and B. Nachtergaele. Applied analysis . World Scientific Publishing Company, 2001.

M. Kakade. On the Sample Complexity of Reinforcement Learning . PhD thesis, University College London, 2003.

Kearns, Y. Mansour, and A. Y. Ng. A sparse sampling algorithm for near-optimal planning in large markov decision processes. Machine learning , 49:193-208, 2002.

W. Lavori and R. Dawson. Dynamic treatment regimes: practical design considerations. Clinical trials , 1(1):9-20, 2004.

Li, C. Cai, Y . Chen, Y . Wei, and Y . Chi. Is q-learning minimax optimal? a tight sample complexity analysis. Operations Research , 72(1):222-236, 2024.

Martinez-Taboada and A. Ramdas. Empirical bernstein in smooth banach spaces. arXiv preprint arXiv:2409.06060 , 2024.

Morimura, M. Sugiyama, H. Kashima, H. Hachiya, and T. Tanaka. Nonparametric return distribution approximation for reinforcement learning. In Proceedings of the 27th International Conference on Machine Learning (ICML-10) , pages 799-806, 2010.

Pananjady and M. J. Wainwright. Instance-dependent /lscript ∞ -bounds for policy evaluation in tabular reinforcement learning. IEEE Transactions on Information Theory , 67(1):566-585, 2020.

Pinelis. Optimum bounds for the distributions of martingales in banach spaces. The Annals of Probability , pages 1679-1706, 1994.

Pisier. Martingales in Banach Spaces . Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2016. doi: 10.1017/CBO9781316480588.

Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics , pages 400-407, 1951.

Rowland, M. Bellemare, W. Dabney, R. Munos, and Y. W. Teh. An analysis of categorical distributional reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 29-37. PMLR, 2018.

Rowland, R. Munos, M. G. Azar, Y. Tang, G. Ostrovski, A. Harutyunyan, K. Tuyls, M. G. Bellemare, and W. Dabney. An analysis of quantile temporal-difference learning. Journal of Machine Learning Research , 25(163):1-47, 2024a. URL http://jmlr.org/papers/v25/23-0154.html .

Rowland, L. K. Wenliang, R. Munos, C. Lyle, Y. Tang, and W. Dabney. Near-minimax-optimal distributional reinforcement learning with a generative model. arXiv preprint arXiv:2402.07598 , 2024b.

S. Sutton. Learning to predict by the methods of temporal differences. Machine learning , 3:9-44, 1988.

Talebi, G. Sadeghi, and M. Moslehian. Freedman inequality in noncommutative probability spaces. Complex Analysis and Operator Theory , 16(2):22, 2022.

Tarres and Y. Yao. Online learning as stochastic approximation of regularization paths: Optimality and almost-sure convergence. IEEE Transactions on Information Theory , 60(9):5716-5735, 2014.

Tropp. Freedman's inequality for matrix martingales. Electronic Communications in Probability , 16(none):262 -270, 2011. doi: 10.1214/ECP.v16-1624. URL https://doi.org/10.1214/ECP.v16-1624 .

Villani et al. Optimal transport: old and new , volume 338. Springer, 2009.

Wu, M. Uehara, and W. Sun. Distributional offline policy evaluation with predictive error guarantees. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 37685-37712. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/wu23s.html .

Zhang, Y. Peng, J. Liang, W. Yang, and Z. Zhang. Estimation and inference in distributional reinforcement learning. arXiv preprint arXiv:2309.17262 , 2023.

A The Key Lemma: Freedman's Inequality in Hilbert Spaces

Freedman's inequality, proposed in [Freedman, 1975], can be viewed as a Bernstein's inequality for martingales, which is crucial for analyzing stochastic approximation algorithms. Compared to the Azuma-Hoeffding inequality which only utilizes the boundedness of martingale difference sequences, Freedman's inequality incorporates second-order information, namely the quadratic variation (cumulative conditional variance) of martingales. This may leads to a sharper concentration result. It has various generalizations, such as matrix Freedman's inequality [Tropp, 2011]. However, to the best of our knowledge, a Freedman's inequality in Hilbert spaces has not been established yet. Just as Freedman's inequality is essential for the theory of TD (Theorem 1 in [Li et al., 2024]), it is indispensable for deriving the minimax non-asymptotic convergence bound for distributional TD.

In this section, we will present a Freedman's inequalities in Hilbert spaces. Firstly, we will state a Hilbert space version of the original Freedman's inequality (Theorem 1.6 in [Freedman, 1975]). After that, we state a generalization of a more powerful version (Theorem 6 in [Li et al., 2024]) to Hilbert spaces. We will provide self-contained proofs in Appendix A.1, primarily inspired by Theorem 3.2 in [Pinelis, 1994]. The necessary knowledge of martingale theory for the proofs can be found in any standard textbook, such as [Durrett, 2019].

Let X be a Hilbert space, { X i } n i =1 be an X -valued martingale difference sequence adapted to the filtration {F i } n i =1 , Y i := ∑ i j =1 X j be the corresponding martingale, W i := ∑ i j =1 σ 2 j be the corresponding quadratic variation process. Here σ 2 j := E j -1 ‖ X j ‖ 2 , and E i [ · ] := E [ ·|F i ] is the conditional expectation.

Theorem A.1 (Freedman's inequality in Hilbert spaces) . Suppose max i ∈ [ n ] ‖ X i ‖ ≤ b for some constant b > 0 . Then, for any ε, σ > 0 , the following inequality holds

Now, we are ready to state the generalization of Theorem 6 in [Li et al., 2024] to Hilbert spaces, which is used in our non-asymptotic analysis.

Theorem A.2 (Freedman's inequality in Hilbert spaces with bounded quadratic variation) . Suppose max i ∈ [ n ] ‖ X i ‖ ≤ b and W n ≤ σ 2 for some constants b, σ > 0 almost surely. Then, for any δ ∈ (0 , 1) , and any positive integer H ≥ 1 , with probability at least 1 -δ , for all k ∈ [ n ] , the following inequality holds

The proof can be found in Appendix A.2.

Remark 2: Theorem 2.1 can be straightforwardly extended to the case where ( ‖ X i ‖ ) n i =1 satisfies the Bernstein condition (Theorem 1.2A in [de la Pena, 1999]), thereby relaxing the boundedness assumption on ‖ X i ‖ . Namely, E i -1 ‖ X i ‖ k ≤ 1 2 k ! σ 2 i b k -2 for some b > 0 , and for all i ∈ [ n ] , k ∈ { 2 , 3 , · · ·} . In this case, Freedman's inequality still holds, albeit with a worse constant.

The proof only requires making appropriate modifications after the fifth line of Equation (12). Note that Bernstein condition holds if max i ∈ [ n ] ‖ X i ‖ ≤ b .

A.1 Proof of Theorem 2.1

Proof. For any λ > 0 , t ∈ [0 , 1] and j ∈ [ n ] , let φ ( t ) = φ j,λ ( t ) := E j -1 cosh ( λ ‖ Y j -1 + tX j ‖ ) = E j -1 cosh ( λu ( t )) , where u ( t ) := ‖ Y j -1 + tX j ‖ . We aim to use the Newton-Leibniz formula to establish the relationship between φ (1) = E j -1 cosh ( λ ‖ Y j ‖ ) and φ (0) = cosh ( λ ‖ Y j -1 ‖ ) . This will allow us to construct a positive supermartingale ( B i ) n i =0 . By utilizing the positive supermartingale and optional stopping theorem, we can derive the desired concentration inequality.

Firstly, we calculate the derivative of φ .

By utilizing Newton-Leibniz formula, we have

Now, we calculate the second order derivate of φ .

where in the third line, we used u ′′ ( t ) = ‖ X j ‖ 2 u ( t ) - 〈 Y j -1 + tX j ,X j 〉 u ( t ) 2 u 2 ( t ) ≥ 0 by Cauchy-Schwarz inequality, and h ( x ) = x cosh( x ) -sinh( x ) ≥ 0 for any x ≥ 0 , the inequality holds because h (0) = 0 and h ′ ( x ) = x sinh( x ) ≥ 0 for any x ≥ 0 . In the fourth line, we used ( u 2 ) ′′ ( t ) = 2 ( ( u ′ ( t )) 2 + u ′′ ( t ) u ( t ) ) . In the fifth line, we used

And in the last line, we used

this holds since

Hence, we can derive the following inequality for all j ∈ [ n ]

which holds for any λ ∈ (0 , 3 b ) . In the fifth line, we used Taylor expansion e x = ∑ ∞ k =0 x k k ! . In the sixth line, we used ( k +2)! ≥ 2(3 k ) and ‖ X j ‖ ≤ b . In the seventh line, we used Taylor expansion 1 1 -x = ∑ ∞ k =0 x k for x ∈ ( -1 , 1) .

Let B 0 := 1 , B i := exp { -λ 2 W i 2(1 -λb/ 3) } cosh ( λ ‖ Y i ‖ ) , then

i.e. , ( B i ) n i =0 is positive supermartingale. By optional stopping theorem (Theorem 4.8.4 in [Durrett, 2019]), for any stopping time τ , we have E [ B τ ] ≤ E [ B 0 ] = 1 .

Let τ := inf { k ∈ [ n ] : ‖ Y k ‖ ≥ ε } be a stopping time, and inf ∅ := ∞ . Define an event

then on A , we have τ < ∞ , ‖ Y τ ‖ ≥ ε and W τ ≤ σ 2 , noting that W k is non-decreasing with k . Our goal is to provide an upper bound for P ( A ) .

where in the second line, we used Cauchy-Schwarz inequality. In the third line, we used E [ B τ ] ≤ 1 . In the fourth line, we used ‖ Y τ ‖ ≥ ε and W τ ≤ σ 2 on A , and cosh( x ) is increasing when x ≥ 0 . In the last line, we used cosh( x ) ≥ 1 2 e x .

we can choose λ /star = ε σ 2 + εb/ 3 ∈ (0 , 3 b ) , then

which is the desired conclusion.

A.2 Proof of Theorem A.2

Proof. According to Theorem 2.1, for any ε, ˜ σ > 0 , we have

We can check that when ε = √ 4˜ σ 2 log 2 δ + 4 3 b log 2 δ , the upper bound on RHS is less than δ . Hence,

Hence for any λ ∈ (0 , 3 b )

For each k ∈ [ n ] , define the events

By the definition, we only need to show P ( ⋃ k ∈ [ n ] H ( k ) H ) ≤ δ . Since W k ≤ W n ≤ σ 2 almost surely, we can find that H ( k ) H ⊆ ⋃ h ∈ [ H ] B ( k ) h,H (we will justify this later). Then ⋃ k ∈ [ n ] H ( k ) H ⊆ ⋃ h ∈ [ H ] ⋃ k ∈ [ n ] B ( k ) h,H . By the inequality (19) with ˜ σ 2 = σ 2 2 h -1 and setting δ as δ H , we have P ( ⋃ k ∈ [ n ] B ( k ) h,H ) ≤ δ H for all h ∈ [ H ] . By the union bound, we can arrive at the conclusion:

To justify H ( k ) H ⊆ ⋃ h ∈ [ H ] B ( k ) h,H , we can consider the decomposition where

The decomposition holds because W k ≤ W n ≤ σ 2 almost surely. We only need to show that for each h ∈ [ H ] ,

On the event H ( k ) H ∩ C ( k ) h,H , we have

hence H ( k ) H ∩ C ( k ) h,H ⊆ B ( k ) h,H

B Minimax Lower Bound of Distributional Policy Evaluation

In this section, we still consider infinite-horizon tabular MDP defined in Section 2, and assume a generative model is accessible. For any positive integer D , we define M ( D ) as the set of all MDPs with state space size |S| = D . For any MDP M and policy π , we denote V π M as the corresponding value function, and η π M as the corresponding return distribution.

Now, we can state the minimax lower bound of the distributional policy evaluation task in the 1 -Wasserstein metric.

Theorem B.1 (Minimax lower bound of distributional policy evaluation in the 1 -Wasserstein metric) . For any positive integer D ≥ 3 , and sample size T ≥ C 1 -γ log D 2 , the following result holds

Here, c, C > 0 are universal constants, and the infimum ˆ η ∈ P D ranges over all measurable functions of T samples from the generative model.

The theorem states that for any algorithm, there exist corresponding MDP M and policy π , such that to ensure E [ ¯ W 1 (ˆ η, η π M ) ] ≤ ε for some ε > 0 , at least ˜ Ω ( 1 ε 2 (1 -γ ) 3 ) samples are required.

Proof of Theorem B.1. For any η ∈ P D , we define V ( η ) ∈ R D as the entry-wise expectation of η . It is easy to check that V ( η π M ) = V π M . And recall the dual representation of 1 -Wasserstein metric (Corollary 5.16 in [Villani et al., 2009])

we have ¯ W 1 (ˆ η, η π M ) ≥ ‖V (ˆ η ) -V π M ‖ ∞ . Hence inf ˆ η sup M ∈ M ( D ) sup π E [ ¯ W 1 (ˆ η, η π M ) ] ≥ inf ˆ η sup M ∈ M ( D ) sup π E [ ‖V (ˆ η ) -V π M ‖ ∞ ] M ∈ ( D ) [∥ ∥ ∥ ∥ ∥ ∥ ∞ ] ≥ c (1 γ ) 3 / 2 √ log D 2 T , where the second inequality holds because V (ˆ η ) ∈ R D is also a measurable function of T samples from the generative model, and the infimum ˆ V ∈ R D ranges over all measurable functions of T samples from the generative model. And the last inequality is due to Theorem 2(b) in [Pananjady and Wainwright, 2020].

C Omitted Proofs in Section 5

C.1 Remove the Dependence on T for Step Sizes

We have shown that the conclusion holds for

where c 5 c 6 ≤ 1 8 , c 5 > c 6 > 0 and C 4 > 0 .

Then for some c 2 > c 3 > 0 to be determined, now we assume

Next, we will show that if we consider the result of the T 2 -th iteration with this step size scheme as the initialization of a new iteration process, then the step sizes in the subsequent T 2 iterations lie in the previously established range. If this is done, the conclusion still holds if we choose T ≥ 2 C 4 log 3 T ε 2 (1 -γ ) 3 log |S| T δ , since the initialization η π T/ 2 ∈ P S (or P S K in the case of CTD) is independent of the samples obtained for T 2 < t ≤ T .

For any T 2 < t ≤ T , we denote τ := t -T 2 , we can see that there exist c 2 > c 3 > 0 , such that the last inequality in both of the following lines hold simultaneously, which is desired.

and

C.2 Range of Step Size

Proof of Lemma 5.1.

For any 0 ≤ k ≤ t 2 , where in the last inequality, we used c 5 c 6 ≤ 1 8 .

C.3 Concentration of the Martingale Term

Proof of Lemma 5.2. We will show that the inequality holds for each t ≥ T c 6 log T and then apply the union bound. For any s ∈ S , we denote where we omit the superscript ( t ) for brevity, then LHS in the lemma equals ∥ ∥ ∥ ∑ t k =1 ζ k ∥ ∥ ∥ for each t . Let F k denote the σ -field that contains all information up to time step k , then { ζ k ( s ) } t k =1 is a {F k } t k =1 -martingale difference sequence:

the first equality holds because a Bochner integral can be exchanged with a bounded linear operator (see Pisier [2016] for more details about Bochner integral), and the second equality holds due to the definition of the empirical distributional Bellman operator.

We hope to use Freedman's inequality (Theorem A.2) to bound this martingale. To this end, we need to give a deterministic upper bound of the martingale difference sequence, and an upper bound of its quadratic variation.

And for any t 2 < k ≤ t ,

Deterministic upper bound of max k ∈ [ t ] ‖ ζ k ( s ) ‖ . The norm of the martingale difference ‖ ζ k ( s ) ‖ can be bounded as follow

Hence, max k ∈ [ t ] ‖ ζ k ( s ) ‖ ≤ max k ∈ [ t ] β ( t ) k √ 1 -γ ≤ 1 √ 1 -γ max { 1 T 2 , 2 log 3 T (1 - √ γ ) T } ≤ 4 log 3 T (1 -γ ) 3 / 2 T =: b .

Upper bound of quadratic variation. Now, let's calculate the quadratic variation.

For any ξ ∈ M S , we define its one-step update Cramér variation as σ ( ξ ) := Var ( ( ̂ T - T ) ξ ) ∈ R S , where T is a random operator and has the same distribution as T 1 .

We first introduce some notations. For any k ∈ N , we denote Var ( ξ ) := ( E [ ‖ ξ ( s ) ‖ 2 ]) s ∈S ∈ R S , Var k ( ξ ) := ( E k [ ‖ ξ ( s ) ‖ 2 ]) s ∈S ∈ R S for any random element ξ in M S .

̂ For any x , y ∈ R S , we say x ≤ y if x ( s ) ≤ y ( s ) for all s ∈ S . In this part, ‖ x ‖ := ‖ x ‖ ∞ = max s ∈S | x ( s ) | , √ x := ( √ x ( s ) ) s ∈S . And for any U ∈ R S×S , ‖ U ‖ := ‖ U ‖ ∞ = sup x ∈ R S , ‖ x ‖ =1 ‖ Ux ‖ = max s ∈S ∑ s ′ ∈S | U ( s, s ′ ) | .

We denote I ∈ R S×S as the identity matrix, 1 ∈ R S as the all-ones vector, and P := P π ∈ R S×S , i.e. , P ( s, s ′ ) := P π ( s ′ | s ) = ∑ a ∈A π ( a | s ) P ( s ′ | s, a ) .

For any { x k } n k =1 ⊂ R S , we denote max k ∈ [ n ] x k as ( max k ∈ [ n ] x k ( s ) ) s ∈S .

With these notations, the quadratic variation is W t := ∑ t k =1 Var k -1 ( ζ k ) . To bound the quadratic variation W t , we need to bound Var k -1 ( ζ k ) .

Lemma C.1.

Hence, the quadratic variation W t can be bounded as follow

where in the fourth line, we used and

In the last line, we used the fact that max k : t/ 2 ≤ k<t σ ( η k -1 ) ≥ 0 and the following lemma:

Lemma C.2. For any t ∈ N , ( α i ) i ∈ [ t ] ∈ [0 , 1] t , the following inequality holds entry-wise:

According to (39), we have the following deterministic upper bound for ‖ W t ‖ = max s ∈S W t ( s ) ,

Let H = ⌈ 2 log 2 1 1 -γ ⌉ , we have

By applying Freedman's inequality (Theorem A.2) and utilizing the union bound over s ∈ S , we obtain with probability at least 1 -δ , for all t ∈ [ T ] and s ∈ S

where we used log 8 |S| T log 1 1 -γ δ ≤ 2 log |S| T δ in the second line, which holds due to the choice of T . The following lemmas are required for deriving the upper bound, which hold for both cases of NTD and CTD.

Lemma C.3. For any t ∈ [ T ] ,

Lemma C.4.

Combining the upper bound with the two lemmas, we get the desired conclusion

where in the last line, we used that, excluding the constant term, the first term is larger than the second term, given the choice of T ≥ C 4 log 3 T ε 2 (1 -γ ) 3 log |S| T δ .

C.4 Solve the Recurrence Relation

Theorem C.1. Suppose for all t ≥ T c 6 log T ,

Then there exists some large universal constant C 7 > 0 , such that

Proof. For any k ≥ 0 , we denote for 0 ≤ k ≤ log 2 ( c 6 log T ) . We can see that ‖ ∆ T ‖ ¯ W 1 ≤ u k for any valid k . Hence, it suffices to show the upper bound holds for u k for any valid k . It can be verified that u 0 ≤ 1 1 -γ , and for k ≥ 0

We first show that once u k ≤ 1 , the subsequent values of u k + l will also remain upper bounded by 1 . Namely, if u k ≤ 1 for some k ≥ 1 , then if T ≥ 2450 log 3 T log |S| T δ (1 -γ ) 3 .

Let τ := inf { k : u k ≤ 1 } , then for any k > τ , we have

For k ≤ τ , we have u k ≥ 1 and thereby i.e. ,

Apply it recursively, we have

i.e. ,

To sum up, for any k ≥ 0 , u k +1 is always less than the sum of the upper bounds in cases of k > τ and k ≤ τ ,

Note that, a 2 ( 1 -1 / 2 k ) ≤ max { a, √ a } , and if we take k ≥ c 8 log log 1 1 -γ for any constant c 8 , we have 1 (1 -γ ) 1 / 2 k = O (1) . We can take the constant c 8 small enough such that c 8 log log 1 1 -γ < log 2 ( c 6 log T ) (this can be done and c 8 is universal since 1 1 -γ = o ( T ) ), and thereby we can find a valid k /star ≥ c 8 log log 1 1 -γ +1 . Then which is the desired conclusion, and C 7 is some large universal constant related to c 8 .

C.5 Analysis of Corollaries 4.1 and 4.2

The difference in the proof compared to Section 5.2 arises in Lemma 5.2 when we control term (II). Now we further bound the result in Lemma C.3 by the Cramér norm of the error term,

In the same way, we can derive the following recurrence relation: with probability at least 1 -δ , for all t ≥ T c 6 log T

By repeating the reasoning of Theorem C.1, we can obtain the desired conclusion,

which is less than ε if we take C 4 ≥ 2 C 2 7 and T ≥ C 4 log 3 T ε 2 (1 -γ ) 5 / 2 log |S| T δ . Here, C 7 > 1 is a large universal constant depending on c 6 .

C.6 Proof of Lemma C.1

Proof. We first introduce some notations. For any matrix of operators U ∈ L ( M ) S×S , we denote U ( s ) = ( U ( s, s ′ )) s ′ ∈S ∈ L ( M ) S as the s -row of U . And for any ξ ∈ M S , we define the vector inner product operation U ( s ) ξ := ∑ s ′ ∈S U ( s, s ′ ) ξ ( s ′ ) ∈ M . We need the following lemma, which holds for both cases of NTD and CTD.

Lemma C.5. For any ν ∈ M , n ∈ N , ( α i ) i ∈ [ n ] ∈ [0 , 1] n , let U n = ∏ n i =1 [(1 -α i ) I + α i T ] , U n = ∏ n i =1 [ (1 -α i ) I + α i √ γ P ] , u n = ∏ n i =1 [ (1 -α i ) + α i √ γ ] then for any s, s ′ ∈ S , we have

Utilizing this lemma, we get the following result. Recall that ̂ T is a random operator and has the same distribution as T 1 . Then, for any non-random ξ ∈ M S ,

where we used different rows of ̂ T are independent, and ̂ T ( s ′ ) ξ is an unbiased estimator of T ( s ′ ) ξ ∈ M . Hence, Var ( U n ( ̂ T - T ) ξ ) ≤ u n U n σ ( ξ ) . Now, we are ready to bound Var k -1 ( ζ k )

C.7 Proof of Lemma C.2

Proof.

C.8 Proof of Lemma C.3

where the inequality holds entry-wise since we can verify that all entries of ( I - √ γ P ) -1 = ∑ ∞ k =0 ( √ γ P ) k and (1 -α i ) I + α i √ γ P are non-negative.

Proof.

In the case of CTD, if we can show Π K is non-expansive w.r.t. 1 -Wasserstein metric, the conclusion still holds. For any x, y ∈ [ 0 , 1 1 -γ ] such that x < y , we denote x ∈ [ x k , x k +1 ) and y ∈ [ x l , x l +1 ) , then k ≤ l , by the definition of Π K , we have

If k = l , we can check that W 1 (Π K δ x , Π K δ y ) = ι K y -x ι K = y -x . If k < l , we have W 1 (Π K δ x , Π K δ y ) ≤ W 1 (Π K δ x , x k +1 ) + W 1 ( x k +1 , x l ) + W 1 ( x l , Π K δ y ) = ( x k +1 -x ) + ( x l -x k +1 ) + ( y -x x l ) = y -x . Hence, for any ν 1 , ν 2 ∈ P and for any transport plan κ ∈ Γ( ν 1 , ν 2 ) , the previous results tell us the cost of the transport plan Π K κ ∈ Γ(Π K ν 1 , Π K ν 2 ) induced by Π K is no greater than the cost of κ . Consequently, W 1 (Π K ν 1 , Π K ν 2 ) ≤ W 1 ( ν 1 , ν 2 ) , i.e. , Π K is nonexpansive w.r.t. 1 -Wasserstein metric, which is desired.

C.9 Proof of Lemma C.4

In the case of CTD, by Corollary 5.12 in [Rowland et al., 2024b], we have given K > 4 1 -γ .

C.10 Proof of Lemma C.5

Proof. We proof this result by induction. For n = 0 , we have U 0 = I , U 0 = I , u 0 = 1 , thereby the inequality holds trivially. Suppose the inequality holds true for n -1 . To prove that the inequality holds for n , it is sufficient to show that, for any µ ∈ M ,

where δ s,s ′ = 1 if s = s ′ , and 0 otherwise.

LHS can be bounded as follow

where we used Cauchy-Schwarz inequality. We need to give an upper bound for ‖T ( s, s ′ ) µ ‖ 2 .

Note that (Π K T π ) ( s, s ′ ) = Π K ( T π ( s, s ′ )) and ‖ Π K ‖ = 1 , we only need to consider the case of NTD, by the definition of T ( s, s ′ ) , we have where we used Jensen's inequality and Fubini's theorem. Substitute it back to the upper bound,

which is desired.

D Stochastic Distributional Bellman Equation and Operator

In this section, we use the same notations as in Appendix C and only consider the NTD setting. Inspired by stochastic categorical CDF Bellman operator introduced in [Rowland et al., 2024b], we introduce stochastic distributional Bellman operator T : ∆ ( P S ) → ∆ ( P S ) to derive an upper bound for ∥ ∥ ( I -γ P ) -1 σ ( η ) ∥ ∥ in the case of NTD. For any φ ∈ ∆ ( P S ) , we denote η φ be the random element in P S with law φ .

where ( ̂ T η φ )( ω ) := ( ̂ T )( ω )( η φ )( ω ) ∈ P S for any ω ∈ Ω , Ω is the corresponding probability space, and ̂ T is independent of η φ . In this part, ̂ T does not consist of Π K since we only consider the NTD setting.

We consider the 1 -Wasserstein metric W 1 on ∆ ( P S ) , the space of all probability measures on the space ( P S , ¯ /lscript 2 ) . Since ( P S , ¯ /lscript 2 ) is Polish (complete and separable), the space ( ∆ ( P S ) , W 1 ) is also Polish (Theorem 6.18 in [Villani et al., 2009]).

Proposition D.1. The stochastic distributional Bellman operator T is a √ γ -contraction on ∆ ( P S ) , i.e. , for any φ , φ ′ ∈ ∆ ( P S ) , we have

Proof. Let κ /star ∈ Γ( φ , φ ′ ) be the optimal coupling between φ and φ ′ . The existence of κ /star is guaranteed by Theorem 4.1 in [Villani et al., 2009]. And let the random element ξ = ( ξ 1 , ξ 2 ) in ( P S ) 2 has the law κ /star , where ξ 1 and ξ 2 are both random elements in P S . We denote T κ /star := Law [( ̂ T ξ 1 , ̂ T ξ 2 )] ∈ Γ( T φ , T φ ′ ) .

By the proposition and contraction mapping theorem, there exists a unique fixed point of T , we denote ψ ∈ ∆ ( P S ) as the fixed point. Hence, the stochastic distributional Bellman equation reads ψ = T ψ . (73)

We denote η ψ as the random element in P with law ψ , then ̂ T η ψ and η ψ have the same law. As shown in the following proposition, η ψ can be regarded as a noisy version of η . Proposition D.2.

where the expectation is regarded as the Bochner integral in the space of all finite measures on P S , which is a normed linear space equipped with Cramér metric as its norm.

Proof.

where we used ̂ T is independent of η ψ . Since E [ η ψ ] is the fixed point of T , we have E [ η ψ ] = η .

Based on the concepts of T and ψ , we can obtain the following second order distributional Bellman equation, which is similar to the classic second-order Bellman equation (Lemma 7 in [Gheshlaghi Azar et al., 2013]).

Proposition D.3 (Second order distributional Bellman equation) .

Recall the one-step Cramér variation σ ( η ) = ( E [ ∥ ∥ ∥ ( ̂ T η ) ( s ) -η ( s ) ∥ ∥ ∥ 2 ]) s ∈S ∈ R S used in the NTD setting. We denote σ := σ ( η ) for simplicity, and Σ := ( E [ ‖ η ψ ( s ) -η ( s ) ‖ 2 ]) s ∈S ∈ R S .

Proof. For any s ∈ S , where the last equality holds since the cross term is zero as below

The first term in (75) is σ ( s ) , we need to deal with the second term.

Put these together, and we can arrive at the conclusion.

Now, we can derive a tighter upper bound for ∥ ∥ ( I -γ P ) -1 σ ( η ) ∥ ∥ . Corollary D.1.

Proof. Note that all entries of ( I -γ P ) -1 = ∑ ∞ k =0 ( γ P ) k are positive, thereby ( I -γ P ) -1 σ ( η ) ≤ ( I -γ P ) -1 σ = Σ , and Σ ( s ) = E [ ‖ η ψ ( s ) -η ( s ) ‖ 2 ] ≤ ∫ 1 1 -γ 0 dx = 1 1 -γ for any s ∈ S .

E Other Technical Lemmas

Lemma E.1 (Basic Inequalities for Metrics on the Space of Probability Measures) . For any ν 1 , ν 2 ∈ P , we have ( /lscript 2 ( ν 1 , ν 2 )) 2 ≤ W 1 ( ν 1 , ν 2 ) ≤ 1 √ 1 -γ /lscript 2 ( ν 1 , ν 2 ) and W p ( ν 1 , ν 2 ) ≤ 1 (1 -γ ) 1 -1 p W 1 p 1 ( ν 1 , ν 2 ) .

(77)

And

And

Lemma E.2. ( M , ‖·‖ /lscript 2 ) and ( M , ‖·‖ W 1 ) are separable.

Proof. Recall that ( P , W 1 ) is separable [Theorem 6.18 Villani et al., 2009], and by Lemma E.1, the Cramér distance /lscript 2 can be bounded by 1 -Wasserstein distance W 1 , hence ( P , /lscript 2 ) is also separable. Let d be either W 1 or /lscript 2 , and A be the countable dense subset of ( P , d ) . For any /epsilon1 > 0 , µ ∈ M , we denote ˜ µ := 2 µ | µ | ( R ) . Then we can find a q ∈ Q , ¯ µ + , ¯ µ -∈ A, s . t . ∣ ∣ q -1 2 | µ | ( R ) ∣ ∣ ≤ /epsilon1 | µ | ( R ) 6 ‖ µ ‖ d , d (¯ µ + , ˜ µ + ) ≤ /epsilon1 3 q , d (¯ µ -, ˜ µ -) ≤ /epsilon1 3 q . Note that the set of all possible ¯ µ = ¯ µ + -¯ µ -is countable. Let ˆ µ := q ¯ µ , then we have

Hence we have found a countable dense subset for ( M , ‖·‖ d ) for d = W 1 or /lscript 2 . Therefore, ( M , ‖·‖ /lscript 2 ) and ( M , ‖·‖ W 1 ) are separable.

Lemma E.3. ( M , ‖·‖ W 1 ) is not complete.

Proof. Consider the Cauchy sequence in ( M , ‖·‖ W 1 ) µ n = ∑ n i =1 ( δ 1 i + 1 2 i -δ 1 i -1 2 i ) , which satisfies ‖ µ n -µ n + k ‖ W 1 ≤ ∑ k i =1 1 2 n + i -1 ≤ 1 2 n -1 → 0 , but its limit ∑ ∞ i =1 ( δ 1 i + 1 2 i -δ 1 i -1 2 i ) / ∈ M because its total variation is infinity. Hence, this is a Cauchy sequence without a limit, implying the space is not complete.

Lemma E.4 (Range of η π t ) . Suppose that α t ∈ [0 , 1] for all t ≥ 0 . Assume that η π 0 ∈ P S , then we have, for all t ≥ 0 , η π t ∈ P S in the case of NTD. Similarly, assume that η π 0 ∈ P S K , then we have, for all t ≥ 0 , η π t ∈ P S K in the case of CTD.

Proof. Wewill only prove the case of NTD, and the proof for CTD is similar by utilizing the property of the projection operator Π K : P S → P S K . We prove the result by induction. It is trivial that η π t ∈ P S for t = 0 . Suppose that η π t -1 ∈ P S , recall the updating scheme of NTD

It is evident that P S is a convex set, considering that P S is a subset of the product signed measure space, which is a linear space. Therefore, we only need to show that T π t η π t -1 ∈ P S , which trivially holds since T π t is a random operator mapping from P S to P S , and η π t -1 ∈ P S . By applying the induction argument, we can arrive at the conclusion.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The theoretical assumptions used in the paper are widely accepted and supported by a rich body of references in the literature, and the references are cited in our paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our paper provide the full set of assumptions and a self-contained proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Given that our paper focuses on the theoretical analysis of existing popular algorithms, it does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Given that our paper focuses on the theoretical analysis of existing popular algorithms, it does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Given that our paper focuses on the theoretical analysis of existing popular algorithms, it does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Given that our paper focuses on the theoretical analysis of existing popular algorithms, it does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Given that our paper focuses on the theoretical analysis of existing popular algorithms, it does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conform with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper only focuses on theory, hence there is no negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper only focuses on theory.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the finite sample performance/non-asymptotic analysis of distributional temporal difference by providing a tighter (minimax optimal) bound than previous works. They propose the non-parametric distributional TD without incorporating any parameterization error. By leveraging the conclusion that the zero-mass signed measure space with the cramer metric is a Hilbert space, the authors propose a novel Freedman’s inequality (Theorem A.2) in the stochastic approximation literature, which is used to derive the sample complexity results for the distributional TD.\n\nSTRENGTHS:\n* It is reasonable to investigate the non-asymptotic convergence of distributional TD.\n\n* The resulting conclusions in Theorems 4.1 and 4.2 are technically sound based on the stochastic approximation proof technique.\n\nWEAKNESSES:\n* The theoretical contribution is within a limited scope. Since the asymptotic and non-asymptotic convergence of distributional TD have already been investigated, this paper only contributes to providing tighter non-asymptotic bounds (by using a generative model), which is thus limited in the research scope. The current version is not well-motivated to me.\n\n* The proposed non-parametric distributional TD is impractical, which is mainly helpful for theoretical results. As CTD also follows the same updating rule, the theoretical results in Sections 4.1 and 4.2 apply in a parallel manner, which may not be novel.\n\n* The proposed Freedman’s inequality is one kind of invariant of its vanilla version, which may be sufficiently valued, as far as I can tell. If the authors think this inequality should be viewed as a main theoretical contribution, it should be emphasized and submitted as a stochastic approximation paper in applied probability instead of RL. As for the current version, I acknowledge that this inequality is useful for the finite sample analysis, but the authors are suggested to put more thought into how to posit this contribution more reasonably.\n\n* The writing needs substantial improvement. While it is basically clear, I find it hard to understand some parts of the paper. For example, CTD uses the approximation in (8), where $\\eta$ should be $\\hat{\\eta}$. Some related works are missing beyond QTD and CTD, such as kernel-based or sample-based TD methods [1].\n\n[1] Distributional Bellman Operators over Mean Embeddings (ICML 2024)\n\nQUESTIONS:\nHow to show the bounds in Theorems 4.1 and 4.2 are minimax optimal? Is it possible to conduct some toy examples to verify them?\n\nLIMITATIONS:\nSome assumptions may not be clearly stated in the main content. It is not very clear in what detailed conditions the derived bounds are minimax optimal.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the statistical efficiency of distributional temporal difference (TD) algorithms in reinforcement learning, focusing on non-asymptotic results. The authors introduce a non-parametric distributional TD algorithm (NTD), analyze its sample complexity with respect to the p-Wasserstein metric and Cramer metric, and show the near minimax optimality. They also revisit the categorical TD (CTD) and prove that it shares similar non-asymptotic convergence bounds with NTD.\n\nSTRENGTHS:\n- The notation is standard and clear, and the related work on sample complexity of distributional RL is well-written, making it easy to understand the authors' theoretical contributions.\n \n- The proposed analysis of NTD and CTD provides a tighter near-minimax optimal sample complexity bound compared to previous methods. \n\n- A new Freedman's inequality is presented, which seems to be useful for further research.\n\n- The bound on $\\beta_k^{(t)}$ presented in Eq. (18) is impressive and appears highly useful.\n\nWEAKNESSES:\nAs a minor comment, it would be helpful to present sample complexities of this paper and related work in a table to help understand the authors' contributions. \n\nThe matrix-wise distributional bellman operator $\\mathcal{T}(s,s')$ presented on line 257-259 uses the same notation as the standard, which can be misleading.\n\nTypos:\n- In Line 105, the font for $T^{\\pi}$ should be corrected to calligraphic.\n- In lines 211,212, $K \\geq \\frac{4}{\\epsilon^2 (1-\\gamma)^3}$ should be $K \\geq \\frac{4}{\\epsilon^2 (1-\\gamma)^2}$.\n- On line 275, the period is written twice.\n- Line 510 should be corrected to $\\log$, not $\\log_2$.\n\nQUESTIONS:\n- In Theorem 4.2, I am curious about the derivation of the statement that the number of atoms required for CTD to converge is $\\tilde{O}(1/\\epsilon^2 (1-\\gamma)^2)$.\n\n- Additionally, DCFP also achieves the same sample complexity with categorical representation, so a detailed explanation of the differences between these two papers would be beneficial.\n\nLIMITATIONS:\nThe paper clarifies the limitations by making assumptions for theoretical explanations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents last-iterate error bounds for distributional\ntemporal difference learning in the $W_p$ and Cramér metrics. The\nresults apply to a nonparametric/intractable distributional TD algorithm\n(where return distributions can be represented exactly) and a tractable\nprojected distributional TD algorithm with finite categorical\nparameterizations of return distributions. Using a novel Hilbert space\nmartingale inequality, the paper achieves tighter bounds than existing\nresults in the literature, as well as generalizations to more metric\nspaces.\n\nSTRENGTHS:\nThis paper is very technically precise, and mostly well written. The Hilbert\nspace Freedman inequality that was derived for the purpose of proving several\nresults in the paper seems useful. Moreover, the error bounds generalize and/or\nimprove upon the existing results on non-asymptotic sample complexity in\ndistributional RL.\n\nWEAKNESSES:\nMany of the mathematical derivations were very quick and at times\ndifficult to follow (particularly, in the appendix). While the work of\nRowland et al. 2024 studies an algorithm that is less similar to\npractical distributional RL methods, the conclusion of their work is\nfairly similar (distributional RL has the same statistical efficiency as\nexpected-value RL). Their work does not cover all $W_p$ distances unlike\nthis work, and the dependence on $p$ is interesting; it is also\ninteresting, again, that this work provides a certificate for good\nstatistical efficiency with a stochastic approximation TD algorithm.\nThat said, the paper does not do an excellent job of motivating why this\nnovelty is exciting.\n\nMoreover, since the Hilbert space Freedman inequality appears to be an\nimportant contribution, I believe this should have been included in the\nmain text (even just the statement of the theorem).\n\nFinally, I suspect there is a slight mathematical mistake in Appendix A,\nthough I don't expect it substantially changes the results; see the\nQuestions section.\n\n## Minor Issues\n\n\"Temporal Difference\" should really be \"Temporal Difference Learning\".\n\nLine 159 says \"the projection is unique and uniquely given by\" – this is\na little redundant, should be either \"the projection is uniquely given\nby\" or \"the projection is unique and is given by\".\n\nLine 168 mentions the BLT theorem without any citation — it might be\nnice to include a reference here, since \"BLT theorem\" may not be a\nfamiliar term to some.\n\nAt the end of line 275, there are two periods.\n\nOn line 425, \"forth\" -\\> \"fourth\".\n\nQUESTIONS:\nIn Theorem 4.2, you claim that the sample complexity bound does not\ndepend on the number of bins $K$. However, you also assume an explicit\nlower bound on $K$, so naturally the sample complexity bound does depend\non $K$ in some capacity. Can anything be said about the sample\ncomplexity (e.g., as a function of $K$) when\n$K\\leq 4\\epsilon^{-2}(1-\\gamma)^{-2}$?\n\nIs the derivative computed on the second step of equation (29) correct?\nIt looks like you went from $\\lambda$ to $\\lambda^2$, but I believe only\non of the terms on the RHS should incur an extra $\\lambda$ factor. My\ncomputation is\n\n\\begin{align*}\n\\phi''(t) &= \\lambda\\mathbb{E}\\_{j-1}\\left\\\\{\\frac{\\mathrm{d}}{\\mathrm{d}t}[\\sin(\\lambda u(t))u'(t)]\\right\\\\}\\\\\\\\\n&= \\lambda\\mathbb{E}_{j-1}\\left(u'(t)\\frac{\\mathrm{d}}{\\mathrm{d} t}\\sinh(\\lambda u(t)) + \\sinh(\\lambda u(t))\\frac{\\rm d}{\\mathrm{d}t}u'(t)\\right)\\\\\\\\\n&= \\lambda\\mathbb{E}\\_{j-1}\\left(\\lambda (u'(t))^2\\cosh(\\lambda u(t)) + \\sinh(\\lambda u(t))u''(t)\\right).\n\\end{align*}\n\nLIMITATIONS:\nLimitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}