Title: Beyond Average Return in Markov Decision Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Markov Decision Process

Dynamic Programming

statistical functionnals

Distributionnal Reinforcement Learning

Policy Evaluation

Planning
Abstract: What are the functionals of the reward that can be computed and optimized exactly in Markov Decision Processes?
In the finite-horizon, undiscounted setting, Dynamic Programming (DP) can only handle these operations efficiently for certain classes of statistics. We summarize the characterization of these classes for policy evaluation, and give a new answer for the planning problem. Interestingly, we prove that only generalized means can be optimized exactly, even in the more general framework of Distributional Reinforcement Learning (DistRL).
DistRL permits, however, to evaluate other functionals approximately. We provide error bounds on the resulting estimators, and discuss the potential of this approach as well as its limitations.
These results contribute to advancing the theory of Markov Decision Processes by examining overall characteristics of the return, and particularly risk-conscious strategies.

Beyond Average Return in Markov Decision Processes

Abstract

What are the functionals of the reward that can be computed and optimized exactly in Markov Decision Processes? In the finite-horizon, undiscounted setting, Dynamic Programming (DP) can only handle these operations efficiently for certain classes of statistics. We summarize the characterization of these classes for policy evaluation, and give a new answer for the planning problem. Interestingly, we prove that only generalized means can be optimized exactly, even in the more general framework of Distributional Reinforcement Learning (DistRL). DistRL permits, however, to evaluate other functionals approximately. We provide error bounds on the resulting estimators, and discuss the potential of this approach as well as its limitations. These results contribute to advancing the theory of Markov Decision Processes by examining overall characteristics of the return, and particularly risk-conscious strategies.

1 Introduction

Reinforcement Learning (RL) has emerged as a flourishing field of study, delivering significant practical applications ranging from robot control and game solving to drug discovery or hardware design [Lazic et al., 2018, Popova et al., 2018, V olk et al., 2023, Mirhoseini et al., 2020]. The cornerstone of RL is the "return" value, a sum of successive rewards. Conventionally, the focus is on computing and optimizing its expected value on Markov Decision Process (MDP). The remarkable efficiency of MDPs comes from their ability to be solved through dynamic programming with the Bellman equations [Sutton and Barto, 2018, Szepesvári, 2010]. RL theory has seen considerable expansion, with a renewed interest for the consideration of more rich descriptions of a policy's behavior than the sole average return. At the other end of the spectrum, the so-called Distributional Reinforcement Learning (DistRL) approach aims at studying and optimizing the entire return distribution, leading to impressive practical results [Bellemare et al., 2017, Hessel et al., 2018, Wurman et al., 2022, Fawzi et al., 2022]. Between the expectation and the entire distribution, the efficient handling of other statistical functionals of the reward appears also particularly relevant for risk-sensitive contexts [Bernhard et al., 2019, Mowbray et al., 2022].

Despite recent progress, the full understanding of the abilities and limitations of DistRL to compute other functionals remains incomplete, with the underlying theory yet to be fully understood. Historically, the theory of RL has been established for discounted MDPs, see e.g. [Sutton and Barto, 2018, Watkins and Dayan, 1992, Szepesvári, 2010, Bellemare et al., 2023] for modern reference textbooks. Recently more attention was drawn to the undiscounted, finite-horizon setting [Auer, 2002,

Osband et al., 2013, Jin et al., 2018, Ghavamzadeh et al., 2020], for which fundamental questions remain open. In this paper, we explore policy evaluation, planning and exact learning algorithms for undiscounted MDPs for the optimization problem of general functionals of the reward. We explicitly delimit the possibilities offered by dynamic programming as well as DistRL.

Our paper specifically addresses two questions:

(i) How accurately can we evaluate statistical functionals by using DistRL?

(ii) Which functionals can be exactly optimized through dynamic programming?

We first recall the fundamental results in dynamic programming and Distributional RL. Addressing question (i), we refer to Rowland et al. [2019]'s results on Bellman closedness and provide their adaptation to undiscounted MDPs. We then prove upper bounds on the approximation error of Policy Evaluation with DistRL and corroborate these bounds with practical experiments. For question (ii), we draw a connection between Bellman closedness and planning. We then utilize the DistRL framework to identify two key properties held by optimizable functionals.

Our main contribution is a characterization of the families of utilities that verify these two properties (Theorem 2). This result gives a comprehensive answer to question (ii) and closes an important open issue in the theory of MDP. It shows in particular that DistRL does not extend the class of functionals for which planing is possible beyond what is already allowed by classical dynamic programming.

2 Background

We introduce the classical RL framework in finite-horizon tabular Markov Decisions Processes (MDPs). We write P ( R ) the space of probability distributions on R . A finite-horizon tabular MDP is a tuple M = ( X , A , p, R, H ) , where X is a finite state space, A is a finite action space, H is the horizon, for each h ∈ [ H ] , p h ( x, a, · ) is a transition probability law and R h ( x, a ) is a reward random variable with distribution ϱ h . The parameters ( p h ) and ( R h ) define the model of dynamics. A deterministic policy on M is a sequence π = ( π 1 , . . . , π H ) of functions π h : X → A .

Reinforcement Learning traditionally focuses on learning policies optimizing the expected return. For a given policy π , the Q -function maps a state-action pair to its expected return under π :

When the model is known, the Q -function of a policy π can be computed by doing a backward recursion, also called dynamic programming. This is referred to as Policy Evaluation . Similarly, an optimal policy can be found by solving the optimal Bellman equation:

Solving this equation when the model is known is also called Planning . When it is unknown, reinforcement learning aims at finding the optimal policy from sample runs of the MDP. But evaluating and optimizing the expectation of the return in the definition of the Q-function above is just one choice of statistical functional. We now introduce Distributional RL and then discuss other statistical functionals that generalize the expected setting discussed so far.

2.1 Distributional RL

Distributional RL (DistRL) refers to the approach that tracks not just a statistic of the return for each state but its entire distribution . We introduce here the most important basic concepts and refer the reader to the recent comprehensive survey by Bellemare et al. [2023] for more details. The main idea is to use the full distributions to estimate and optimize various metrics over the returns ranging from the mere expectation [Bellemare et al., 2017] to more complex metrics [Rowland et al., 2019, Dabney et al., 2018a, Liang and Luo, 2022].

At state-action ( x, a ) , let Z π h ( x, a ) denote the future sum of rewards when following policy π and starting at step h , also called return . It verifies the simple recursive formula Z π h ( x, a ) = R h ( x, a ) + Z π h +1 ( X ′ , π h +1 ( X ′ )) where X ′ ∼ p h +1 ( x, a, · ) . Its distribution is η = ( η π h ( x, a )) ( x,a,h ) ∈X×A× [ H ] and is often referred to as the Q-value distribution . One can easily derive the recursive law of the return as a convolution: for any two measures ν 1 , ν 2 ∈ P ( R ) , we denote their convolution by ν 1 ∗ ν 2 ( t ) = ∫ R ν 1 ( τ ) ν 2 ( t -τ ) dτ . For any two independent random variables X and Y , the distribution of the sum Z = X + Y is the convolution of their distributions: ν Z = ν X ∗ ν Y . Thus, the law of Z π h ( x, a ) is

This equation is a distributional equivalent to Eq. (1) and thus defines a distributional Bellman operator η π h = T π h η π h +1 .

Obviously, from a practical point of view, distributions form a non-parametric family that is not computationally tractable. It is necessary to choose a parametric (thus incomplete) family to represent them. Even the restriction to discrete reward distributions is not tractable, since the number of atoms in the distributions may grow exponentially with the number of steps 1 [Achab and Neu, 2021]: approximations are unavoidable. The most natural solution is to use projections of the obtained distribution on the parametric family, at each step of the Bellman operator. This process is called parameterization . The practical equivalent to Eq. (1) in DistRL hence writes

where Π is the projector operator on the parametric family. The full policy evaluation algorithm in DistRL is summarized in Alg.1.

Algorithm 1 Policy Evaluation (Dynamic Programming) for Distributional RL

Input: model p , reward distributions ϱ h , policy π to evaluated, Π projection.

Data: η ∈ R H |X||A| N ∀ x, a ∈ X × A , η H ( x, a ) = δ 0 for h = H -1 → 0 do ′ ′ ′ h h end for Output: η h ( x, a ) ∀ x, a, h

Distribution Parametrization The most commonly used parametrization is the so-called quantile projection . It put Diracs (atoms) with fixed weights at locations that correspond to the quantiles of the source distribution. One main benefit is that it does not require a previous knowledge of the support of the distribution, and allows for unbounded distributions.

The quantile projection is defined as

which corresponds to a minimal W 1 distance: Π Q µ ∈ arg min ˆ µ = ∑ i δ i /N W 1 ( µ, ˆ µ ) , where W 1 ( ., . ) is the Wasserstein distance defined for any distributions ν 1 , ν 2 as W 1 ( ν 1 , ν 2 ) = ∫ 1 0 ∣ ∣ F -1 ν 1 ( u ) -F -1 ν 2 ( u ) ∣ ∣ d u . Note that this parametrization might admit several solutions and thus the projection may not be unique. For simplicity, we overload the notation to Π Q η = (Π Q η ( x, a )) ( x,a ) ∈X×A

For a Q-value distribution η with support of length ∆ η , and parametrization of resolution N , Rowland et al. [2019] prove that the projection error is bounded by

In Section 3, we extend this result to the full iterative Policy Evaluation process and bound the error on the returned statistical functional in the finite-horizon setting. Note that other studied parametrizations exist but are less practical. For completeness, we discuss the Categorical Projection [Bellemare et al., 2017][Rowland et al., 2018][Bellemare et al., 2023] in Appendix B.

2.2 Beyond expected returns

The expected value is an important functional of a probability distribution, but it is not the only one of interest in decision theory - especially when a control of the risk is important. We discuss two concepts that have received considerable attention: utilities , defined as expected values of functions of the return, and distorted means which place emphasis on certain quantiles.

Expected Utilities are of the form E [ f ( Z )] , or ∫ f d ν , where Z is the return of distribution ν and f is an increasing function. For instance, when f is a power function, we obtain the different moments of the return. The case of exponential functions plays a particularly important role: the resulting utility is referred to as exponential utility , exponential risk measure , or generalized mean according to the context:

This family of utilities has a variety of applications in finance, economics, and decision making under uncertainty [Föllmer and Schied, 2016]. They can be considered as a risk-aware generalization of the expectation, with benefits such as accommodating a wide range of behaviors [Shen et al., 2014] from risk-seeking when λ > 0 , to risk-averse when λ < 0 (the limit λ → 0 is exactly the expectation). To fix ideas, U exp ( N ( µ, σ 2 ) ) = µ + λσ 2 : each λ captures a certain quantile of the Gaussian distribution.

Distorted means , on the other hand, involve taking the mean of a random variable, but with a different weighting scheme [Dabney et al., 2018a]. The goal is to place more emphasis on certain quantiles, which can be achieved by considering the quantile function F -1 of the random variable and a continuous increasing function β : [0 , 1] → [0 , 1] . By applying β to a uniform variable τ on [0 , 1] and evaluating F -1 at the resulting value β ( τ ) , we obtain a new random variable that takes the same values as the original variable, but with different probabilities. The distorted mean is then calculated as the mean of this new random variable, given by the formula ∫ β ′ ( τ ) F -1 ( τ ) dτ . If β is the identity function, the result is the classical mean. When β is τ ↦→ min( τ/α, 1) , we get the α -Conditional Value at Risk (CVaR ( α ) ) of the return, a risk measure widely used in risk evaluation [Rockafellar et al., 2000].

3 Policy Evaluation

The theory of MDPs is particularly developed for estimating and optimizing the mean of the return of a policy. But other values associated to the return can be computed the same way, by dynamic programming. This includes for instance the variance of the return, or more generally, any moment of order p ≥ 2 , as was already noticed in the 1980's [Sobel, 1982]. Recently, Rowland et al. [2019] showed that for utilities in discounted MDPs, this is essentially all that can be done. More precisely, they introduce the notion of Bellman closedness (recalled below for completeness) that characterizes a finite set of statistics that can efficiently be computed by dynamic programming.

Definition 1 (Bellman closedness [Rowland et al., 2019]) . A set of statistical functionals { s 1 , . . . s K } is said to be Bellman closed if for each ( x, a ) ∈ X × A , the statistics s 1: K ( η π h ( x, a )) can be expressed in closed form in terms of the random variables R h ( x, a ) and ( s 1: K ( η π h +1 ( X ′ , A ′ )) , A ′ ∼ π ( x ) , X ′ ∼ p h ( x, A ′ , · ) , independently of the MDP.

Importantly, in the undiscounted setting, Rowland et al. [2019](Appendix B, Theorem 4.3) show that the only families of utilities that are Bellman closed are of the form { x ↦→ x ℓ exp( λx ) | 0 ≤ ℓ ≤ L }

for some L < ∞ . Thus, all utilities and statistics of the form of (or linear combinations of) moments and exponential utilities can easily be computed by classic linear dynamic programming and do not require distributional RL (see Appendix A.3).

Some important metrics such as the CVaR or the quantiles are not known to belong to any Bellmanclosed set and hence cannot be easily computed. For this kind of function of the return, the knowledge of the transitions and the values in following steps is insufficient to compute the value on a specific step. In general, it requires the knowledge of the whole distribution of each reward in each state. Hence, techniques developed in distributional RL come in handy: for a choice of parametrization, one can use the projected dynamic programming step Eq. (4) to propagate a finite set of values along the MDP and approximate the distribution of the return. In the episodic setting, following the line of Rowland et al. [2019] (see Eq.(6)), we prove that the Wasserstein distance error between the exact and approximate distribution of the Q-values of a policy is bounded.

Proposition 1. Let π be a policy and η π the associated Q-value distributions. Assume the return is bounded on a interval of length ∆ η ≤ H ∆ R , where ∆ R is the support size of the reward distribution. Let ˆ η π be the Q-value distributions obtained by dynamic programming (Algorithm 1) using the quantile projection Π Q with resolution N . Then,

This result shows that the loss of information due to the parametrization may only grow quadraticly with the horizon. The proof consists of summing the projection bound in (6) at each projection step, and using the non-expansion property of the Bellman operator [Bellemare et al., 2017]. The details can be found in Appendix C

The key question is then to understand how such error translates into our estimation problem when we apply the function of interest to the approximate distribution. We provide a first bound on this error for the family of statistics that are either utilities or distorted means.

First, we prove that the utility is Lipschitz on the set of return distributions.

Lemma 1. Let s be either an utility or a distorted mean and let L be the Lipschitz coefficient of its characteristic function. Let ν 1 , ν 2 be return distributions. Then:

Both family of functionals are treated separately, but lays a similar bound. The utility bound is the direct application of the Kantorovitch-Rubenstein duality, while the distorted mean one is a direct majoration in the integral. Again, the details are provided in the Appendix.

This property allows us to prove a maximal upper bound on the estimation error for those two families.

Theorem 1. Let π be a policy. Let η π be the Q-value return distribution associated to π with the return bounded on a interval of length ∆ η ≤ H ∆ R where ∆ R is the support size of the reward distribution. Let ˆ η π be the approximated return distribution computed with Algorithm 1, for the projection Π Q with resolution N . Let s be either an expected utility or a distorted mean, and L the Lipschitz coefficient of its characteristic function. Then:

Note that depending on the choice of utilities, the Lipschitz coefficient L may also depend on H and ∆ R . For instance, in a stationary MDP, the Lipschitz constant of the exponential utility depends exponentially on ∆ η . For the CVaR ( α ) , however, L is constant and only depends on α ∈ (0 , 1) .

Experiment: empirical validation of the bounds on a simple MDP We consider a simple Chain MDPenvironment of length H = 70 equal to the horizon (see Figure 1 (right)) [Rowland et al., 2019], with a single action leading to the same discrete reward distribution for every step. We consider a Bernouilli reward distribution B (0 . 5) for each state so that the number of atoms for the return only grows linearly 2 with the number of steps, which allows to compute the exact distribution easily.

R~Qh=@

R~@h=@

1

2

h

H

Vh ∈ [H], p(h + 1|h,a =1) = 1

2.0

CVaR(0.1)error

istance

ApproximationError

Scaled CVaR(O.1)bound

1.5

Cumulative Projection Error

4

CVaR(0.25)error

TheoreticalErrorbound

Error

Scaled CVaR(0.25)bound

1.0

2

0.5

0

0.0

0

20

40

60

0

20

40

60

Timestep h

Timestep h

We compare the distributions obtained with exact dynamic programming and the approximate distribution obtained by Alg 1, with a quantile projection with resolution N = 1000 . Note that even at early stages, when the true distribution has less atoms than the resolution, the exact and approximate distributions differ due to the weights of the atoms in the quantile projection. Figure 2 (Right) reports the Wasserstein distance between the two distributions: the cumulative projection approximation error (dashed blue), the true error between the current exact and approximate distributions (solid blue) and the theoretical bound (red). Fundamentally, the proof of Prop. 1 upper bounds the distance between distributions by the cumulative projection error so we plot this quantity to help validating it.

We also empirically validate Theorem 1 by computing the CVaR( α ) for α ∈ { 0 . 1 , 0 . 25 } , corresponding respectively to distorted means with Lipschitz constants L = { 10 , 4 } . We compute these statistics for both distributions and report the maximal error together with the theoretical bound, re-scaled 3 by a factor 5. Figure 2 (Left) shows an impressive correspondence of the theory and the empirical results despite a constant multiplicative gap.

4 Planning

Planning refers to the problem of returning a policy that optimizes our objective for a given model and reward function (or distribution in DistRL). It shares with policy evaluation the property to be grounded on a Bellman equation: see Eq. (2) for the classical expected return, which leads to efficient computations by dynamic programming.

For other statistical functionals of the cumulated reward, however, can the optimal policy be computed efficiently? The main result of this section characterizes the family of functionals that can be exactly and efficiently optimized by Dynamic Programming.

In the previous section, we recalled that Bellman-closed sets of utilities can be efficiently computed by DP as long as all the values of the utilities in the Bellman-closed family are computed together. For the planning problem, however, we only want to optimize one utility so we cannot consider families as previously. Under this constraint, only exponential and linear expected utilities are Bellman closed and thus can verify a Bellman Equation. In fact, for the exponential utilities, such Bellman Equation

exists and allows for the planning problem to be solved efficiently [Howard and Matheson, 1972]:

However, the question of the existence of Optimal Bellman Equations for non-utility functionals remains open (e.g. quantiles). More generally, an efficient planning strategy is not known. To address these questions, we consider the most general framework, DistRL, and recall the theoretical DP equations for any statistical functional s in the Pseudo 4 -Algorithm 2. DistRL offers the most comprehensive, or 'lossless', approach, so if a statistical functional cannot be optimized with Alg. 2, then there cannot exist a Bellman Operator to perform exact planning.

Algorithm 2 Pseudo-Algorithm: Exact Planning with Distributional RL

1: Input: model p , reward R , statistical functional s

2: Data: η ∈ R H |X||A| N , ν ∈ R H |X| N

3: ∀ x ∈ X , ν x H +1 = δ 0

5: η h ( x, a ) = ϱ ( x,a ) h ∗ ∑ x ′ p a h ( x, x ′ ) ν x ′ h +1 ∀ x, a ∈ X × A

4: for h = H → 1 do

6: ν x h = η h ( x, a ∗ ) , a ∗ ∈ arg max a s ( η h ( x, a )) ∀ x ∈ X

7: end for

8: Output: η h ( x, a ) ∀ x, a, h

We formalize this idea with the new concept of Bellman Optimizable statistical functionals:

Definition 2 (Bellman Optimizable statistical functional) . A statistical functional s is called Bellman optimizable if, for any MDP M , the Pseudo-Algorithm 2 outputs an optimal return distribution η = η ∗ that verifies:

Remark. This definition is equivalent to the satisfaction of an Optimal Distributionnal Bellman equation. Indeed, a statistical functional s is Bellman Optimizable if and only if, for any ( X , A , ϱ, p, η ) , s verifies

with a ⋆ x ∈ arg max a s ( η ( x, a ))

We can now state our main results that characterizes all the Bellman optimizable statistical functionals. First, we prove that such a functional must satisfy two important properties.

Lemma 2. A Bellman optimizable functional s satifies the two following properties:

Independence Property: If ν 1 , ν 2 ∈ P ( R ) are such that s ( ν 1 ) ≥ s ( ν 2 ) , then

Translation Property: Let τ c denote the translation on the set of distributions: τ c δ x = δ x + c . If ν 1 , ν 2 ∈ P ( R ) are such that s ( ν 1 ) ≥ s ( ν 2 ) , then

Indeed, the expectation and the exponential utility both satisfy these properties (see Appendix A.2). Each property is implied by an aspect of the Distributional Bellman Equation (Alg. 2, line 5) and the proof (in Appendix E) unveils these important consequences of the recursion identities.

Fundamentally, they follow from the Markovian nature of policies optimized this way: the choice of the action in each state should be independent of other states and rely only on the knowledge of the next-state value distribution.

The Independence property states that, for Bellman optimizable functionals, the value of each next state should not depend on that of any other value in the convex combination in the rightmost term of the convolution. In turn, the Translation property is associated to the leftmost term, the reward, and it imposes that, for Bellman optimizable functionals, the decision on the best action is independent of the previously accumulated reward.

The Independence property is related to expected utilities [von Neumann et al., 1944]. Any expected utility verifies this property (Appendix A.2) but, most importantly, the Expected Utility Theorem (also known as the Von Neumann Morgenstein theorem) implies that any continuous statistical functional s verifying the Independence property can be reduced to an expected utility. This means that for any such statistical functional s , there exists f continuous such that ∀ ν 1 , ν 2 ∈ P ( R ) , we have s ( ν 1 ) > s ( ν 2 ) ⇐⇒ U f ( ν 1 ) > U f ( ν 2 ) [von Neumann et al., 1944, Grandmont, 1972].

This result directly narrows down the family of Bellman optimizable functionals to utilities. Indeed, although other functionals might potentially be optimized using the Bellman equation, addressing the problem on utilities is adequate to characterize all possible behaviors. For instance, momentoptimal policies that can be found through dynamic programming, can also be found by optimizing an exponential utility function. The next task is therefore to identify all the utilities that satisfy the second property. We demonstrate that, apart from the mean and exponential utilities, no other W 1 -continuous functional satisfies this property.

Theorem 2. Let ϱ be a return distribution. The only W 1 -continuous Bellman Optimizable statistical functionals of the cumulated return are exponential utilities U exp ( ϱ ) = 1 λ log E ϱ [exp( λR )] for λ ∈ R , with the special case of the expectation E ϱ [ R ] when λ = 0 .

Of course, if U ( ρ ) is a utility and ψ is a monotonous scalar mapping, ψ ( U ( ρ )) is an equivalent utility: one should understand in the previous theorem that a W 1 -continuous Bellman Optimizable statistical functional is equivalent to U exp ( ϱ ) for some λ ∈ R . We chose here to define U exp ( ϱ ) = 1 λ log E ϱ [exp( λR )] with the log and the factor 1 /λ since it results in a normalized utility that tends to the expectation when λ goes to 0 . The full proof of Theorem 2 is provided in Appendix E.

We make a few important observations. First, this result shows that algorithms using Bellman updates to optimize any continuous functionals other than the exponential utility cannot guarantee optimality. The theorem does not apply to non-continuous functionals, but Lemma 2 still does. For instance, the quantiles are not W 1 -continuous so Theorem 2 does not apply, but it is easy to prove that they do not verify the Independence Property and thus are not Bellman Optimizable. Also, there might also exist other optimizable functionals, like moments, but they must first be reduced to exponential or linear utilities.

Most importantly, while in theory, DistRL provides the most general framework for optimizing policies via dynamic programming, our result shows that in fact, the only utilities that can be exactly and efficiently optimized do not require to resort to DistRL. This certainly does not question the very purpose of DistRL, which has been shown to play important roles in practice to regularize or stabilize policies and to perform deeper exploration [Bellemare et al., 2017, Hessel et al., 2018]. Some advantages of learning the distribution lies in the enhanced robustness offered in the richer information learned [Rowland et al., 2023], particularly when utilizing neural networks for function approximation [Dabney et al., 2018b, Barth-Maron et al., 2018, Lyle et al., 2019].

5 Q-Learning Exponential Utilities

The previous sections consider the the model, i.e. the reward and transition functions, be known. Yet in most practical situations, those are either approximated or learnt 5 . After addressing policy evaluation (Section 3) and planning (Section 4), we conclude here the argument of this paper by addressing the question of learning the statistical functionals of Theorem 2. In fact, we simply highlight a lesser known version of Q-Learning Watkins and Dayan [1992] that extend this celebrated algorithm to exponential utilities. We provide the pseudo-code for the algorithm proposed by Borkar

[2002, 2010] with the relevant utility-based updates in Alg. 3. We refer to these seminal works for convergence proofs. Linear utility updates (line 8) differ only slightly from classical ones for expected return optimization, which have been shown to lead to the optimal value asymptotically [Watkins and Dayan, 1992].

6 Discussions and Related Work

The Discounted Framework We focused in this article on undiscounted MDPs, and it is important to note that the results differ for discounted scenarios. The crucial difference is that the family of exponential utilities no longer retains Bellman Closed or Bellman Optimizable properties due to the introduction of the discount factor γ [Rowland et al., 2019]. When it comes to Bellman Optimization, the necessary translation property becomes an affine property : ∀ c, γ, s ( τ γ c ν 1 ) > s ( τ γ c ν 2 ) where τ γ c is the affine operator such that τ γ c δ x = δ γx + c . This property is not upheld by the exponential utility. Nonetheless, there exists a method to optimize the exponential utility through dynamic programming in discounted MDPs [Chung and Sobel, 1987]. This approach requires modifying the functional to optimize at each step (the step h is optimized with the utility x ↦→ exp( γ -h λ x ) ), but it also implies a loss of policy stationarity, property usually obtained in dynamic programming for discounted finite-horizon MDPs [Sutton and Barto, 2018].

Utilizing functionals to optimize expected return. DistRL has also been used in Deep Reinforcement learning to optimize non-Bellman-optimizable functionals such as distorted means[Ma et al., 2020, Dabney et al., 2018a]. While, as we proved so, such algorithms cannot lead to optimal policies in terms of these functionals, experiments show that in some contexts they can lead to better expected return and faster convergence in practice. The change of functional can be interpreted as a change in the exploration process, and the resulting risk-sensitive behaviors seem to be relevant in adequate environments.

Dynamic programming for the optimization of other functionals To optimize other statistical functionals such as CVaR and other utilities such as moments with Dynamic Programming, Bäuerle and Ott [2011] and Bäuerle and Rieder [2014] propose to extend the state space of the original MDP to X ′ = X × R by theoretically adding a continuous dimension to store the current cumulative rewards. This idea does not contradict our results, and the resulting algorithms remain empirically much more expensive.

Another recent thread of ideas to optimize functionals of the reward revolve around the dual formulation of RL through the empirical state distribution [Hazan et al., 2019]. Algorithms can be derived by noticing that utilities like the CVar are equivalent to solving a convex RL problem [Mutti et al., 2023].

7 Conclusion

Our work closes an important open problem in the theory of MDPs: we exactly characterize the families of statistical functionals that can be evaluated and optimized by dynamic programming.

We also put into perspective the DistRL framework: the only functionals of the return that can be optimized with DistRL can actually be handled exactly by dynamic programming. Its benefit lies elsewhere, and notably in the improved stability of behavioral properties it allows. We believe that, by narrowing down the avenues to explain its empirical successes, our work can contribute to clarify the further research to conduct on the theory of DistRL.

References

Achab and G. Neu. Robustness and risk management via distributional dynamic programming, Dec. 2021. arXiv:2112.15430 [cs, math].

Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617 , 2018.

G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International conference on machine learning , pages 449-458. PMLR, 2017.

G. Bellemare, W. Dabney, and M. Rowland. Distributional Reinforcement Learning . MIT Press, 2023. http://www.distributional-rl.org .

Bernhard, S. Pollok, and A. Knoll. Addressing inherent uncertainty: Risk-sensitive behavior generation for automated driving using distributional reinforcement learning. In 2019 IEEE Intelligent Vehicles Symposium (IV) , pages 2148-2155, 2019. doi: 10.1109/IVS.2019.8813791.

S. Borkar. Q-learning for risk-sensitive control. Mathematics of operations research , 27(2): 294-311, 2002.

S. Borkar. Learning algorithms for risk-sensitive control. In Proceedings of the 19th International Symposium on Mathematical Theory of Networks and Systems-MTNS , 2010.

Bäuerle and J. Ott. Markov Decision Processes with Average-Value-at-Risk criteria. Mathematical Methods of Operations Research , 74(3):361-379, Dec. 2011. ISSN 1432-2994, 1432-5217. doi: 10.1007/s00186-011-0367-0.

Bäuerle and U. Rieder. More Risk-Sensitive Markov Decision Processes. Mathematics of Operations Research , 39(1):105-120, Feb. 2014. ISSN 0364-765X. doi: 10.1287/moor.2013.0601. Publisher: INFORMS.

K.-J. Chung and M. J. Sobel. Discounted MDP's: Distribution Functions and Exponential Utility Maximization. SIAM Journal on Control and Optimization , 25(1):49-62, Jan. 1987. ISSN 0363-0129, 1095-7138. doi: 10.1137/0325004.

Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit Quantile Networks for Distributional Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning , pages 1096-1105. PMLR, July 2018a. ISSN: 2640-3498.

Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional Reinforcement Learning With Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), Apr. 2018b. ISSN 2374-3468. Number: 1.

Fawzi, M. Balog, A. Huang, T. Hubert, B. Romera-Paredes, M. Barekatain, A. Novikov, F. J. R. Ruiz, J. Schrittwieser, G. Swirszcz, D. Silver, D. Hassabis, and P. Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature , 610(7930):47-53, Oct. 2022. ISSN 1476-4687. doi: 10.1038/s41586-022-05172-4. Number: 7930 Publisher: Nature Publishing Group.

Föllmer and A. Schied. Stochastic Finance: An Introduction in Discrete Time . De Gruyter, Berlin, Boston, 2016. ISBN 9783110463453. doi: doi:10.1515/9783110463453.

Ghavamzadeh, A. Lazaric, and M. Pirotta. Exploration in reinforcement learning. Tutorial at AAAI'20, 2020. URL https://rlgammazero.github.io/ .

J.-M. Grandmont. Continuity properties of a von neumann-morgenstern utility. Journal of Economic Theory , 4(1):45-57, 1972. ISSN 0022-0531. doi: https://doi.org/10.1016/0022-0531(72)90161-5.

Hazan, S. Kakade, K. Singh, and A. Van Soest. Provably efficient maximum entropy exploration. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2681-2691. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/hazan19a.html .

Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), Apr. 2018. doi: 10.1609/aaai. v32i1.11796.

A. Howard and J. E. Matheson. Risk-Sensitive Markov Decision Processes. Management Science , 18(7):356-369, Mar. 1972.

Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018.

Lazic, C. Boutilier, T. Lu, E. Wong, B. Roy, M. Ryu, and G. Imwalle. Data center cooling using model-predictive control. Advances in Neural Information Processing Systems , 31, 2018.

Liang and Z.-Q. Luo. Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds, Oct. 2022. arXiv:2210.14051 [cs, stat] version: 1.

Lyle, M. G. Bellemare, and P. S. Castro. A Comparative Analysis of Expected and Distributional Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01): 4504-4511, July 2019. ISSN 2374-3468. doi: 10.1609/aaai.v33i01.33014504. Number: 01.

Ma, L. Xia, Z. Zhou, J. Yang, and Q. Zhao. DSAC: Distributional Soft Actor Critic for RiskSensitive Reinforcement Learning, June 2020. arXiv:2004.14547 [cs].

Mirhoseini, A. Goldie, M. Yazgan, J. Jiang, E. Songhori, S. Wang, Y.-J. Lee, E. Johnson, O. Pathak, S. Bae, et al. Chip placement with deep reinforcement learning. arXiv preprint arXiv:2004.10746 , 2020.

Mowbray, D. Zhang, and E. A. D. R. Chanona. Distributional reinforcement learning for scheduling of chemical production processes, 2022.

Mutti, R. De Santi, P. De Bartolomeis, and M. Restelli. Convex reinforcement learning in finite trials. Journal of Machine Learning Research , 24(250):1-42, 2023.

Osband, D. Russo, and B. Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26, 2013.

Popova, O. Isayev, and A. Tropsha. Deep reinforcement learning for de novo drug design. Science advances , 4(7):eaap7885, 2018.

T. Rockafellar, S. Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk , 2: 21-42, 2000.

Rowland, M. Bellemare, W. Dabney, R. Munos, and Y. W. Teh. An Analysis of Categorical Distributional Reinforcement Learning. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , pages 29-37. PMLR, Mar. 2018. ISSN: 2640-3498.

Rowland, R. Dadashi, S. Kumar, R. Munos, M. G. Bellemare, and W. Dabney. Statistics and Samples in Distributional Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning , pages 5528-5536. PMLR, May 2019. ISSN: 2640-3498.

Rowland, Y. Tang, C. Lyle, R. Munos, M. G. Bellemare, and W. Dabney. The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation. 2023.

Shen, M. J. Tobia, T. Sommer, and K. Obermayer. Risk-sensitive reinforcement learning. Neural Computation , 26(7):1298-1328, 2014. doi: 10.1162/NECO_a_00600.

J. Sobel. The variance of discounted Markov decision processes. Journal of Applied Probability , 19(4):794-802, Dec. 1982. ISSN 0021-9002, 1475-6072. doi: 10.2307/3213832.

S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

Szepesvári. Algorithms for Reinforcement Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning , 4(1):1-103, Jan. 2010. ISSN 1939-4608, 1939-4616. doi: 10.2200/ S00268ED1V01Y201005AIM009.

Villani. Topics in Optimal Transportation . Graduate studies in mathematics. American Mathematical Society, 2003. ISBN 9780821833124. URL https://books.google.fr/books?id= idyFAwAAQBAJ .

A. Volk, R. W. Epps, D. T. Yonemoto, B. S. Masters, F. N. Castellano, K. G. Reyes, and M. Abolhasani. Alphaflow: autonomous discovery and optimization of multi-step chemistry using a self-driven fluidic lab guided by reinforcement learning. Nature Communications , 14(1):1403, 2023.

von Neumann, O. Morgenstern, and A. Rubinstein. Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition) . Princeton University Press, 1944. ISBN 978-0-69113061-3.

J. Watkins and P. Dayan. Q-learning. Machine learning , 8:279-292, 1992.

R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs, L. Gilpin, P. Khandelwal, V. Kompella, H. Lin, P. MacAlpine, D. Oller, T. Seno, C. Sherstan, M. D. Thomure, H. Aghabozorgi, L. Barrett, R. Douglas, D. Whitehead, P. Dürr, P. Stone, M. Spranger, and H. Kitano. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature , 602(7896):223-228, Feb. 2022. ISSN 1476-4687. doi: 10.1038/s41586-021-04357-7. Number: 7896 Publisher: Nature Publishing Group.

A Additional remarks

The Wasserstein metric is defined as W 1 ( ν 1 , ν 2 ) = ∫ 1 0 ∣ ∣ F -1 ν 1 ( u ) -F -1 ν 2 ( u ) ∣ ∣ d u and the Cramer metric as ℓ 2 ( ν 1 , ν 2 ) = ( ∫ + ∞ -∞ | F ν 1 ( u ) -F ν 2 ( u ) | 2 d u ) 1 2 . For both metrics, we define their supremum ℓ 2 ( η 1 , η 2 ) = sup ( x,a ) ∈X×A ℓ 2 ( η 1 ( x, a ) , η 2 ( x, a )) and W 1 ( η 1 , η 2 ) = sup ( x,a ) ∈X×A W 1 ( η 1 ( x, a ) , η 2 ( x, a )) .

A.1 Remarks on the recursive definition of the Q-value distribution

The notation of the Q-value distribution η is often deceivingly complex compared to the actual object it means to represent. While the 'usual' expected Q-function Q ( x, a ) is simply understood as the expected return of a policy at state-action pair ( x, a ) , DistRL requires us to keep a notation for the complete distribution of the return. In other words, the Q-value distribution η ( x,a ) π,h should be understood as the distribution of the random variable Z = R + Z ( S ′ ) , which is the convolution of the individual distributions of these two independent random variable. It can also be written:

A.2 Linear and Exponential Utilities satisfy the properties of Lemma 2

Independence Property Any utility U f verifies the independence property. Let ν 1 , ν 2 , ν 3 ∈ P ( R ) , λ ∈ [0 , 1] . Assume U f ( ν 1 ) ≥ U f ( ν 2 ) . Then,

In particular, the mean and the exponential utility do.

Translation Property This property comes from the linearity of the mean and the multiplicative morphism of the exponential. Let ν 1 , ν 2 ∈ P ( R ) , c ∈ R . Assume that U exp ( ν 1 ) ≥ U exp ( ν 2 ) and U mean ( ν 1 ) ≥ U mean ( ν 2 ) . Then,

A.3 Policy Evaluation for linear combinations of moments

Rowland et al. [2019] prove a necessary condition on Bellman-closed utilities, namely that they should be a family of the form { x ↦→ x ℓ exp( λx ) | 0 ≤ ℓ ≤ L } for the undiscounted, finite-horizon setting. In the discounted case, the necessary condition is only valid for λ = 0 , that is, without the exponential. They also prove that moments (without the exponential) also verify the sufficient condition such that in that setting they are the only Bellman-closed families of utilities.

In the undiscounted setting, to the best of our knowledge, a similar result has not yet been proved. We provide here the sufficient condition for families of the form { x ↦→ x ℓ exp( λx ) | 0 ≤ ℓ ≤ L } . We show that they are Bellman-closed and that this implies that they can be computed by DP.

Let's consider the family s k ( ν ) = ∫ r k exp( λr )d ν ( r ) for k ∈ [ n ] and some fixed λ ∈ R .

This first proves that this family of statistical functional is Bellman closed: they can be expressed as a linear combination of the others. Moreover, on the right-hand side, the expression only depends on the distributions and functionals at the current step h and at the next step h +1 . Thus, it provides a natural way to evaluate these functionals by DP.

B Categorical Projection: an alternative parametrization

The categorical projection was proposed and studied in Bellemare et al. [2017], Rowland et al. [2018], Bellemare et al. [2023]. For a bounded return distribution, it spreads a fixed number N of Diracs evenly over the support and used weight parameters to represent the true distribution. The parameter N is often referred to as the resolution of the projection. More precisely, on a support [ V min , V max ] , we write ∆ = V max -V min N -1 the step between atoms z i = V min + i ∆ , i ∈ [ [0 , N -1] ] . We define the projection of a given Dirac distribution δ y on the parametric space P C ( R ) = { ∑ i p i δ z i | 0 ≤ p i ≤ 1 , ∑ i p i = 1 } by

This definition can naturally be extended to any bounded distribution ν , and by extension, Π C η = (Π C η ( s, a )) ( s,a ) ∈X×A . This linear operator minimizes the Cramér distance of the parametrization to the parametric space [Rowland et al., 2018].

This projection verifies this approximation bound, analog to the quantile projection,

Using the property that W 1 ( ν 1 , ν 2 ) ≤ √ ∆ η ℓ 2 ( ν 1 , ν 2 ) , the results with the quantile projection can be adapted for the categorical projection, adding √ ∆ η factors to the bounds.

This parametrization has the nice property of preserving the mean of the distribution. Yet, even for risk-neutral RL, Quantile-based DistRL algorithm seem to work better and display better properties[Dabney et al., 2018b,a].

C Proofs for Policy Evaluation with parameterized distributions

C.1 Proof of Proposition 1

We recall the statement of Proposition 1: Let π be a policy and η π the associated Q-value distributions. Assume the return is bounded on a interval of length ∆ η ≤ H ∆ R , where ∆ R is the support size of the reward distribution. Let ˆ η π be the Q-value distributions obtained by dynamic programming (Algorithm 1) using the quantile projection Π Q with resolution N . Then,

To avoid clutter of notation, we denote W 1 (ˆ η, η ) := sup ( x,a ) ∈X×A W 1 (ˆ η ( x, a ) , η ( x, a )) .

Proof. First recall that for any Q-value distribution ( η h ) h ∈ [ H ] , with the return bounded on an interval of length ∆ η ≤ H ∆ R , and Π one of the projection operator of interest with resolution n , we have the following bound on the projection estimation error due to Rowland et al. [2019] (Eq (6)):

At a fixed step h ∈ [ H ] , we have the following inequality:

Where (14) is due to the triangular inequality with T π h ˆ η π h +1 as the middle term. In (15), the first term comes from applying (13) to the first term of the previous line. The second term is a consequence of the non-expansive property of the Bellman operator [Bellemare et al., 2017]:

Using it recursively starting from h = 1 , and using the fact that ˆ η π H = η π H we get:

C.2 Proof of Lemma 1

We recall the statement of Lemma 1: Let s be either a utility or a distorted mean and let L be the Lipschitz coefficient of its characteristic function. Let ν 1 , ν 2 be return distributions. Then:

Proof. We prove the property for each family of utilities separately:

Case 1: s is a utility. There exists f such that s ( ν ) = ∫ f d ν . Let L f be its Lipsichtz constant. The Kantorovitch-Rubenstein duality [Villani, 2003] states that:

where || · || L is the Lipschitz norm. We then immediatly get:

Case 2: s is a distorted mean. There exists g such that s ( ν ) = ∫ 1 0 g ′ ( τ ) F -1 ν ( τ )d τ . Let L g be its Lipschitz coefficent. Thus:

D About the tightness of Theorem 1

The upper bound provided by Theorem 1 is mainly based on Proposition 1. The latter is obtained by summing, for every step, the Projection Bound by Rowland et al. [2019] (Eq. 6). Thus, achieving the bound would requires first to find a problem instance for which, at every step, the projection bound is tight. Then it would require to verify that the total error is the sum of those projection errors.

The experiment in Figure 2 already shows that summing the total error is very close to the sum of the projection errors. However, in that example, the projection error bound is not reached after the first step. In the following, we exhibit an MDP for which the projection bound is tight at every timestep.

First, let us consider a family of distributions for which the projection error is tight:

Proposition 2. Let N ∈ N , ∆ ∈ R + . Consider z i = i ∆ N . The distribution

has a support of length ∆ and verifies W 1 ( ν N, ∆ , Π Q ν N, ∆ ) = ∆ 2 N .

Proof. The cumulative distribution function (CDF) the distribution ν N, ∆ is

Wewrite τ i = 2 i +1 2 N , so that ∀ i ∈ [ | 0 , N -1 | ] , F ν N, ∆ ( z i ) = τ i . Wenow explicit the projection of ν N, ∆ and the Wasserstein distance relative to it. A possible Quantile projection is Π Q ν N, ∆ = 1 N ∑ N -1 i =0 δ z i ,

and

The value distribution of the following time steps is obtained by applying two operators: the Bellman operator and the projection operator. Here will consider a MDP with only one state. We need to find such operators so that Π Q T ν N 1 ∆ 1 = ν N 2 ∆ 2 , where the Bellman operator simply consists in the added reward distribution.

Proposition 3. Let N ∈ N , ∆ ∈ R + . Consider ϱ = 1 2 ( δ 0 + δ ∆ N -1 ) . Then there exists a Quantile projection operator Π Q such that

Proof. We consider Π Q ( ν N, ∆ ) = 1 N ∑ N -1 i =0 δ i ∆ N -1 . Since ∀ i, i ∆ N -1 ≤ z i < ( i +1)∆ N -1 , we have F ( z i ) = τ i , verifying that it is indeed a valid Quantile Projection. Hence,

The last egality comes down to noticing that i ∆ N -1 = i N (∆ + ∆ N -1 ) .

Here, we take advantage of the fact that the Quantile Projection is not unique (see discussion in section 2.1). By choosing the adapted projection, it is then possible to obtain one of those distribution at every timestep.

Corollary 1. Let N ∈ N , ∆ 0 ∈ R + . Consider the sequence ∆ h +1 = ∆ h (1 + 1 N -1 ) and ϱ h = 1 2 ( δ 0 + δ ∆ h N -1 ) . Consider Π Q as in Prop. 3.

2.0

Distance

CumulativeApproximationError

TrueError

1.5

1.0

0.5

0.0

0

2

4

b

8

10

12

14

timestep

.

Consider the MDP with only one state x and action a , reward distribution ϱ h , horizon H . Consider ˆ η h the value distributions obtained throught dynamic programming with quantile projection. Then, at any timestep h , the error induced by the projection operator matches the bound in Eq. 6:

While the projection error is maximal in such instance, it was verified experimentally that the bound in Prop. 1 was still not tight. This comes from the fact that the Bellman operator is not a non-contraction is this case, and that the triangular inequality used to sum the projection error is not tight either.

We hence found that every inequality used in proving Theorem 1 can be tight, but there does not seem to exist an instance for which all the inequalities are tight at the same time, meaning that this bound would be never reached exactly.

E Proof of the main results

The proof of the result is divided in parts. First we show that Bellman optimizable functionals verify the two properties of Lemma 2 (Independence and Translation). Then, using those properties, we prove that Bellman optimizable functionals can only be exponential utilities (Theorem 2). Using the known fact that exponential utilities are bellman optimizable, we obtain the full characterization.

E.1 Proof of Lemma 2

Proof. To prove that each property is necessary , we use a proof by contradiction, and exhibit MDPs where the algorithm is not optimal when the property is not verified.

Independence Property Let s be a Bellman optimizable statistical functional that does not satisfy the Independence property. That is, there exists ν 1 , ν 2 , ν 3 ∈ P ( R ) and λ ∈ [0 , 1] such that s ( ν 1 ) ≥ s ( ν 2 ) but s ( λν 1 +(1 -λ ) ν 3 ) < s ( λν 2 +(1 -λ ) ν 3 ) .

Then consider the MDP in Fig.4 (left) with horizon H = 2 corresponding to the depth of the tree: The agent starts in Start and must take 2 actions, a unique but random and non-rewarding one ( a 0 ) and a final deterministic step ( a 1 or a 2 ) to a rewarding state. Thus, by construction, the optimal strategy is ( a 0 , a 2 ) that leads to End 2 with probability λ (and End 3 with probability 1 -λ ). The true optimal distribution at Start state is η ∗ 0 = λν 2 + (1 -λ ) ν 3 . We compute the distributions output by the algorithm:

End 1

End 2

End 3

Left

Right

p(c,x)=1

a1

p(x,x)=1

a2

p(x,x)=1

a1,a2

p(x,α）=

a1

1

p（x,x)=1

a2

p=1

p=V2

p=V3

p=V1

p=v2

Left

Right

Step

0D

p（x,x)=1

p(x,x)=>

p(x,x)=1->

ao

p=dc

p=00

p=80

Start

Start

The output return distribution η 0 is not the true optimal η ∗ 0 for s so the algorithm is incorrect which is a contradiction as s is assumed to be Bellman optimizable. Hence the property is needed.

Translation Property Let s be a Bellman optimizable statistical functional that does not verify the Translation Property, i.e. there exists ν 1 , ν 2 ∈ P ( R ) , c ∈ R such that s ( ν 1 ) ≥ s ( ν 2 ) but s ( τ c ν 1 ) < s ( τ c ν 2 ) . Then consider MDP in Fig.4 (right). The optimal strategy is again ( a 0 , a 2 ) by construction. The algorithm output the following distribution:

So here again, the algorithm does not output an optimal distribution for s , hence the necessity of the property.

This proof shows that both properties are necessary, but not that they are sufficient. The other implication could be proven, but the proof would be unnecessary as those properties are enough to restrict to only 1 class of function for which we already know is bellman optimizable.

E.2 Proof of Theorem 2

Proof. For any return distribution ν , let s ( ν ) be the considered functional. Since we assume that s is W 1 -continuous Bellmman optimizable, the by the Independence Property (Lemma 2) we know from the Expected Utility Theorem that we can assume: s ( ν ) = s f ( ν ) = ∫ R f ( x ) dν ( x ) for some continuous, monotonous mapping f . Without loss of generality, we may assume by a density argument that f is twice continuously differentiable.

By the Intermediate Value Theorem, we can then define ϕ ( h ) = f -1 ( 1 2 ( f (0) + f ( h ))) , so that 1 2 ( f (0) + f ( h )) = f ( ϕ ( h )) and in particular ϕ (0) = f -1 ( f (0)) = 0 .

A very special case is when f is constant, which satisfies the theorem. We now assume that there exists point x 0 such that f ′ does not vanish on a neighborhood of x 0 . On this neighborhood, using the inverse function theorem, ϕ is also twice differentiable. Without loss of generality, we assume that x 0 = 0 .

For any fixed h > 0 , we consider the probability distributions ν 1 = 1 2 ( δ 0 + δ h ) and ν 2 = δ ϕ ( h ) . Remark that s f ( ν 1 ) = ∫ f d ν 1 = 1 2 ( f (0) + f ( h )) and s f ( ν 2 ) = f ( ϕ ( h )) = 1 2 ( f (0) + f ( h )) by definition of ϕ , so s f ( ν 1 ) = s f ( ν 2 ) .

The Translation property implies that for all x ∈ R , s f ( ν 1 ) ≤ s f ( ν 2 ) = ⇒ s f ( ν 1 ( · + x ) ) ≤ s f ( ν 2 ( · + x ) ) and s f ( ν 1 ) ≥ s f ( ν 2 ) = ⇒ s f ( ν 1 ( · + x ) ) ≥ s f ( ν 2 ( · + x ) ) . Hence, ∀ x ∈ R , 1 2 ( f ( x ) + f ( x + h ) ) = f ( x + ϕ ( h )) .

This equation can be differentiated twice with respect to h . For any value of x , we obtain:

̸

Recall that by definition, ϕ (0) = 0 . Now, for x = 0 , Eq. (18) yields 1 2 f ′ (0) = ϕ ′ (0) f ′ ( ϕ (0)) = ϕ ′ (0) f ′ (0) and, since f ′ (0) = 0 , ϕ ′ (0) = 1 2 .

Now, choosing h = 0 in (19) and plugging in the values of ϕ (0) and ϕ ′ (0) , we obtain for all x ∈ R :

We then consider two cases, depending on whether ϕ ′′ (0) is null or not.

Case 1: ϕ ′′ (0) = 0 . The equation simply becomes f ′′ ( x ) = 0 , hence f is affine: ∃ a, b ∈ R , f ( x ) = ax + b .

̸

Case 2: ϕ ′′ (0) = 0 . We write β = 4 ϕ ′′ (0) . The differential equation becomes f ′′ ( x ) = βf ′ ( x ) , whose solutions are of the form

Hence, f can only be the identity or the exponential, up to an affine transformation.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates general functionals of the distribution over returns in three situations:\n1) For policy evaluation, the authors prove some error bounds for a given distributional RL algorithm.\n2) For planning, the authors exhibit the functional form that is optimizable via dynamic programming.\n3) For reinforcement learning, they provide a Q-learning algorithm when the functional is linear or exponential\n\nSTRENGTHS:\nThe paper investigates an interesting question, which is useful for sequential decision-making with more sophisticated decision criteria.\n\nThe writing is generally quite clear.\n\nWEAKNESSES:\nThe obtained error bound for policy evaluation seems to be very loose, as also suggested by the experiments. Can we say anything about the tightness of this bound?\n\nThe novelty of some of the obtained results seems to be limited:\n1) How different is the known result about Bellman closedness and the new result about Bellman optimizability (Theorem 2)? Isn't the former property a necessary condition for the latter?\n2) Regarding the proposed Q-learning algorithm, the linear case seems to be trivial. For the exponential case, how different is the proposed update compared to Borkar's?\n\nThe paper should be checked for typos (see below for some of them).\n\nQUESTIONS:\nSee questions above.\n\nSome minor issues:\n- line 52: p_h -> p_h^a(x, \\cdot) and R_h -> R_h(x, a)\n- line 74: p_{h+1}^a(s, \\cdot)  -> p_{h+1}^a(x, \\cdot)\n- line 95: the definition of \\mathcal P_Q(\\mathbb R) should include the condition in (5)\n- line 96: W_1 should be defined here\n- line 209: m\n- line 222: \\eta vs \\eta^*\n- line 227: missing /extra word?\n- line 239: the other von Neumann-Morgenstern axioms are not needed?\n- line 269: sections addresses\n- line 271: linear of exponential\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of using distributional reinforcement learning to perform policy evaluation and planning with more general classes of reward functionals than those typically considered in standard Markov decision processes and RL. The paper focuses in particular on dynamic programming methods for obtaining distributional Q functions based on distributional Bellman operators, as well as how to use these methods for planning with the resulting distributional Q functions. The main contributions of the paper are twofold: (1) when performing distributional policy evaluation on general utilities using a certain kind of distributional Q function approximation scheme, worst-case bounds in terms of Wasserstein distance of the approximation from the true distributional Q function are provided; (2) when performing planning using distributional Q functions, a new notion of \"Bellman optimizability\" of reward functionals is given, and it is shown that the only Bellman optimizable reward functionals are the class of exponential utilities. It was previously known that standard (i.e., not distributional) dynamic programming methods (see, e.g., lines 213-214) can be applied to solve planning problems with general utilities, so contribution (2) provides a negative result suggesting that distributional RL methods, though providing a very general framework for performing evaluation and planning via dynamic programming, are not strictly necessary to solve the class of problems to which they are theoretically suited.\n\nSTRENGTHS:\nOverall, the paper is well-written and the results add clarity to the distributional RL landscape that is likely to be of interest to the community. Regarding significance, contribution (1) outlined in the summary given above is potentially useful for a wider range of utilities than previously considered, but the results appear to be straightforward extensions of existing results (see C.1, C.2 in the appendix). The main result of the paper is instead contribution (2), provided in the form of Theorem 2 (lines 248-250). This result delimits the applicability of distributional dynamic programming to classes of utilities that can be handled using non-distributional techniques, suggesting that the primary usefulness of distributional RL methods may be experimental instead of theoretical. It is refreshing to see an ML paper making an effort to soberly clarify the mathematical limits of a particular subfield of ML research. The proof of this result (see D.2) appears to boil down to first constructing a differential equation based on the integrand of the utility, then solving to show that the integrand must correspond to an exponential utility. The proof is not long, but the idea is clever.\n\nWEAKNESSES:\nI have two main concerns regarding the paper:\n* Contributions (1) and (2) are somewhat disjointed. On the one hand, (1) provides reassurances that Algorithm 1 enjoys approximation bounds for a wide variety of different utilities. At the same time, (2) undermines this result by showing that planning using the resulting distributional Q functions is only theoretically meaningful for a much-restricted subset of those utilities. Some clarification of the implications of (2) for (1) would be helpful.\n* In the proof of Theorem 2, it is assumed that the utility integrand $f$ is twice-differentiable and $f'(x) \\neq 0$, for all values of $x$. This assumption is critical to Theorem 2, thus to contribution (2), and thus to the overall contribution of the paper. It is stated on lines 516-517 that this assumption \"could be proven through long and fastidious analysis that is beyond the scope of this article\", yet no indication of how this might be accomplished is provided. Given its importance to the paper, I argue that this is within scope of the article, so that a proof sketch or at least a convincing explanation of why this assumption holds is needed.\n\nQUESTIONS:\n* Given Theorem 2, what is the main usefulness and impact of the approximation bounds provided in section 3?\n* Why is Bellman optimizability (Definition 2, lines 221-222) the right notion of what it means for a problem to be solvable using distributional RL? Are there incompatible notions of optimality that might invalidate Theorem 2? If not, why not?\n* Why does the assumption regarding the utility integrand $f$ on lines 516-517 in the appendix hold?\n* It is mentioned that policy stationarity is disrupted in the discounted setting on lines 289-290. Can you provide references ensuring sufficiency of stationary policies in the setting you consider?\n\nLIMITATIONS:\nAside from the weaknesses and questions above, the authors have adequately addressed the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nClassic RL maximizes the (discounted) cumulative reward in Markov Decision Processes (MDPs). This paper studies more general functionals of rewards, such as generalized means. These more general functionals are useful to consider for certain applications, such as those with safety concerns. Furthermore, the study of these more general functionals has been not well understood in the literature. \n\nFor finite-horizon prediction problems, this paper 1) argues that only generalized means of rewards can be obtained exactly using dynamic programming and there is no need to resort to learning the entire distribution of return (I am confused about this part, see Questions), and 2) provides an upper bound of estimation error for utility functionals, which are more general than generalized means, when approximated with Quantile Distributional RL. Both 1) and 2) mirror Rowland et al.'s (2019) results, which were developed concerning the discounted setting.\n\nThe most important result is for finite-horizon control problems. This paper shows that the only Bellman optimizable functionals (those that can be optimized by applying Dynamic Programming to the distribution of return) are generalized means. Combined with a previous result showing that generalized means can be solved using dynamic programming (no distribution involved), this result shows that the only functionals that are optimizable by Distributional RL can be optimized by Dynamic Programming, which is a much more efficient algorithm.\n\nSTRENGTHS:\nThe problem that this paper studies is an important one. \n\nThe main result, which shows that the only functionals that are optimizable by Distributional RL can be optimized by Dynamic Programming, is remarkable.\n\nWEAKNESSES:\nThe writing of the paper needs improvements. Specifically:\n1) right now, the presentation of the paper is driven by the technical results that it wants to show, rather than the messages that it wants the readers to know. The paper presents an important result about Distributional RL: the only functionals that are optimizable by Distributional RL can be optimized by Dynamic Programming. As the authors highlighted in line 260, this result is the most important one and it narrows down the avenues to explain its (distributional RL) empirical success (line 310). However, this result is not outspoken throughout the paper, until an inconspicuous discussion on page 8. Why not say your most important result in the Abstract and the Introduction sections? \n\n2) The notations given background section need to be clearly defined:\nthe notation (z_i)_i needs to be defined.\n\\delta is not defined (line 96).\n\\mu is not defined (line 97).\nF_\\mu is not defined (line 98).\nwhat is \\eta in line 100?\n\\Delta_\\eta is not defined in line 101.\nLine 124: U_f is not defined.\n\n3) typos: extra m at the end of line 209\nline 184: should be \"see Figure1 (left)\"\nline 199: should be \"Figure 2 (left)\"\nthe order of the two subfigures in Figure 2 would better be reversed in accordance to the texts.\n\nQUESTIONS:\n\"Much of the existing theory is based of discounted MDPs, but many recent efficient RL algorithms with strong guarantees are for finite-horizon, undiscounted setting\" What kind of theory do you mean for discounted MDPs? Could you give examples that illustrate this argument?\n\nCould you point to the result by Rowland et al. (2019) that gives equation 6?\n\nNote that other parameterizations exist but are less practical. Could you explain why is this true or refer to other works that show that this is true?\n\nLine 122-123: when lambda > 0, the utility is higher for Gaussian distribution with a larger sigma (higher variance), so isn't this case risk-seeking?\n\nLine 124: you don't need to mention this axiom at this point. I have been wondering why I need to know this until I found it being used in section 4.\n\nLine 152 is confusing to me: do you mean that for any L >= 1, statistics corresponding to the first L+1 moments form a Bellman closed set? \n\nLine 152: Theorem 4.3 by Rowland et al. (2019) only mentioned moment functionals but not exponential functionals. Furthermore, although you prove in A.3 that sets of multiplication of moments functionals and exponential functionals also form a Bellman closed set in the finite horizon setting, you didn't show that these are the only families of functionals that form Bellman closed sets in this setting.\n\nThe footnote on page 5: how can the support of return be -h + 1 given that all rewards are non-negative?\n\nLine 215: is the question open for utilities other than exponential ones for control problems?\n\nLine 221: why define Bellman optimizability instead of extending the definition of Bellman closeness to control problems? In addition, it is weird to define a quantity, which you want to optimize, using an algorithm that optimizes it. Do you see other ways that make this definition without resorting to a particular algorithm?\n\nLine 241: why does this result narrow down the family Bellman optimizability functionals to utilities? I mean, literally.\n\nSection 5: why would we want to see a new algorithm without any theoretical/empirical study of the algorithm?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper addresses

STRENGTHS:
1. The method is very well-presented, with notations, terms and algorithms put in a very clear and understandable fashion. They are introduced without fancy names and terms, which is great.
2. The questions are well-stated and addressed respectively from a theoretical perspective, and they are fundamental problems.
3. The significance of contribution is good. It provides a good starting point to address the statistical functional evaluation problem.

WEAKNESSES:
1. Some statements in the propositions lack references to earlier works, such as quantile functions an CVaR (metric).
2. The learning environment description is lacked.
3. The limitations, potential societal impacts and ethical concerns, and future work are lacking.

QUESTIONS:
My suggestions has been listed in weaknesses, which I will paraphrase here:
1. Please fix grammatical errors and add proper references on early works, especially for the sections with propositions.
2. Please connect your work to more practical applications. I know it might be hard to conduct experiments as additions to this paper, but some discussions would be helpful.
3. Please give more details about the experiments on functional distance calculation.
4. Most importantly, please add descriptions on potential limitations, concerns, and societal impacts about the work at the very last of the paper.

LIMITATIONS:
Although since it is a theoretical paper, I do not think an external ethical review is needed, the limitations, potential societal impacts and ethical concerns, and future work are lacking. Authors may address it later if applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25