Near-optimal Distributional Reinforcement Learning towards Risk-sensitive Control
Authors: Hao Liang
Zhi-Quan Luo
Venue: NeurIPS 2022 Submitted (2022)
Keywords: distributional reinforcement learning
risk-sensitive
sample complexity

Abstract: We consider finite episodic Markov decision processes aiming at the entropic risk measure (ERM) of return for risk-sensitive control. We identify two properties of the ERM that enable risk-sensitive distributional dynamic programming. We propose two novel distributional reinforcement learning (DRL) algorithms, including a model-free one and a model-based one, that implement optimism through two different schemes. We prove that both of them attain $\tilde{\mathcal{O}}(\frac{\exp(|\beta| H)-1}{|\beta|H}H\sqrt{HS^2AT})$ regret upper bound, where $S$ is the number of states, $A$ the number of states, $H$ the time horizon and $T$  the number of total time steps. It matches  RSVI2 proposed in \cite{fei2021exponential} with a much simpler regret analysis. To the best of our knowledge, this is the first regret analysis of DRL,  which theoretically verifies the efficacy of  DRL for risk-sensitive control. Finally, we improve the existing lower bound by proving a tighter  bound of $\Omega(\frac{\exp(\beta H/6)-1}{\beta H}H\sqrt{SAT})$ for $\beta>0$ case, which recovers the tight lower bound $\Omega(H\sqrt{SAT})$ in the risk-neutral setting. 

Near-optimal Distributional Reinforcement Learning towards Risk-sensitive Control
Anonymous Author(s)
Affiliation
Address email
Abstract
We consider finite episodic Markov decision processes aiming at the entropic risk measure (EntRM) of return for risk-sensitive control. We identify two properties the EntRM that enable risk-sensitive distributional dynamic programming. We pro- pose two novel distributional reinforcement learning (DRL) algorithms, including model-free one and model-based one, that implement optimism through two different schemes. We prove that both of them attain Ö exp([ H
2
7
regret upper bound, where S is the number of states, the number of states, H the time horizon and 7 the number of total time steps. It matches RSVI2 proposed in [2 with a much simpler regret analysis. To the best of our knowledge, this is the first regret analysis of DRL, which theoretically verifies the efficacy of DRL for risk-sensitive control. Finally, we improve the existing lower bound by proving tighter bound of Q
11
exxp(3H/G)-))) H SAT) for 3 > 0 case, which recovers the BH
13
tight lower bound QWH SAT) in the risk-neutral setting.
Introduction
14
15
Standard reinforcement learning (RL) 45 seeks to find an optimal policy that maximizes the expectation of return. It is also called risk-neutral RL since the objective is the mean functional of the return distribution. However, in some high-stakes applications including finance [3] medical treatment [21 and operations [16] etc. the decision-maker tends be risk-sensitive with the goal of maximizing some risk measure of return distribution.
17
19
20 In this paper, we consider the problem of optimizing the exponential risk measure (EntRM) in the episodic and finite MDP setting for risk-sensitive control. The entropic risk measure can trade-off between the expectation and the variance, and adjusts the risk-sensitiveness by control a risk parameter risk-sensitive RL based on the EntRM has
21
22
23
(see Equation D. Ever since the seminal work of 29
been applied across a wide range of domains [43 involve complicated algorithmic design to deal with the non-linearity the EntRM.
37 27
Most of the existing approaches, however,
25
26
Distributional reinforcement learning (DRL) has demonstrated its superior performance over traditional methods in some difficult tasks 14 under risk-neutral setting. Different from the basud approaches, it learns the whole return distribution instead of real-valued value function. Given the entire return distribution, it natural leverage the distributional information to optimize risk measure other than expectation and risk- sisk-sensi RI, is surprising that existing works on risk-sensitive control via DRL approaches (13 84 II lack regret analysis. Consequently, it is challenging to evaluate and improve these DRL algorithms in terms of sample-efficiency, which brings about a reasonable question
27
28
29
13
44 33
Despite of the intrinsic connection between DRL.
31
33
Can distributional reinforcement learning attain near-optimal regret for risk-sensitive control?
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

35 In this work, we answer this question positively by providing two DRL algorithms with provably
36 regret guarantees. We devise two novel DRL algorithms with principled exploration schemes for risk-sensitive control in the tabular MDP setting. In particular, the proposed algorithms implement the principle of optimism the face of uncertainty (OFU) at the distributional level balance the inlooaation-xploitaatii
37
38
39
trade-off. By providing the first regret analysis of DRI,, we theoretically verifies the efficacy of DRL for risk-sensitive control. Therefore, our work bridge the gap between DRL and risk- RL with regard to sample complexity.
40
41
42
Main contributions. We summarize our main contributions in the following.
1. We build risk-sensitive distributional dynamic programming (RS-DDP) framework. To be more specific, we choose the entropic risk measure (EntRM) of the return distribution as our objective. By identifying two key properties of EntRM, We establish distributional Bellman optimality equation for risk-sensitive control.
44
46
2. We propose two DRL algorithms that enforce the OFU principle in distributional fashion through exp([3]
two different schemes. We provide Ö exp( [3
- H AK) regret upper bound, which matches
48
49 the best existing result of RSVI2 in 22 is the first regret analysis of DRL algorithm in the
finite episodic MDP in the risk-sensitive setting. Compared to 22 our algorithm does not involve
51
complicated bonus design, and our analysis are conceptually cleaner and easier to interpret.
52 3. We fill the gaps in the proof of lower bound in 23). To the best of our knowledge,
23 only
implies a lower bound S2( [xp[])) H12) /K) rather the claimed bound 12( [xx((191//2)- VT). The
53
P
[2]
54
resulting lower bound is independent of S and and is loose with a factor of /H. We overcome
these issues by proving a tight lower bound of 2X( (xxx( HVSAT) for B > 0. Note that the 3H
55
56
lower bound is tight in the risk-neutral setting (3 + 0).
57
Related work. Following the paper DRL has witnessed a rapid growth of study in literature Most of these works focus on improving the performance in the risk- setting, with few exceptions
58
40
3 2 32
59
1 34 However, none of these works study the sample complexity.
60
A rich body of work studies sensiivve RL. with the EntRM 7
22
18 19
61
24
In particular, (29 is the first to introduce the ERM risk- sensitive objective in MDP. However, they either assume known transition and reward or consider
28
30
33 35
36, (38
39
42
43).
82
63
infinite-horizon setting without sample-complexity considerations.
64
Two works are closely related to ours [3 2 under precisely the same setting. 23 is the first to study the risk-sensitive episodic MDP, which provides the first algorithms and regret guarantees. Nevertheless, the regret upper bounds contain a dispensable factor of exp(13) H Additionally, their lower bound proof contains mistakes, and the corrected proof suggests a weaker bound. (22) improves the algorithm by removing the additional ((exp(13) H))) factor. However, the regret analysis is complicated, and the lower bound is not fixed. A very recent work ((I) independently proposes risk-sensitive DDP framework, but their work is fundamentally different from ours. The risk measure considered in M the conditional value risk (CVaR), and they focus on the infinite horizon setting.
66
68
70
71
72
Due to the space limit, we provide detailed comparisons with 23 22
in Appendix A
2
Preliminaries
73
74 Notations.
We write [M: :] A (M, 1, N} and [N] 2 [1: N] for any positive integers 75 M < N. We adopt the convention that Σmn a; & if n > m and II.- a & 1 if n > m. We
78 use I{-} to denote the indicator function. For any x E R, we define [x] - max(x,0), 03. We define
77 the step function with parameter as vx((x) 2 1[x > c). Note that v. represents the CDF of a 78 deterministic variable taking value e. We denote by ((a b)). 2M and 9 the set of distributions
79 supported on (a. b). [0, M) and the set of all distributions respectively. For random variable (I.v.) X, 80 we use [[X] and V X] to denote its expectation and variance. For two r.v.s, we denote by X1Y
81 X is independent of Y. We use Õ() to denote O(-) omitting logarithmic factors.
An episodic MDP is identified by M & (S.A, (P))NE(M)) (Ra)nE(A)). H). where 83 the state space, AA the action space, Pa : S Ax + (S)) the probability transition kernel at 84 step h, Rr S x A ( 1)) the collection of reward distributions at step h and H the length of
82 Episodic MDP.

---

85
one episode. The agent interacts with the environment for K episodes. At the beginning of episode k. Nature selects an initial state sf arbitrarily. In step h, the agent takes action of and observes random reward R (sh a RR(s a and reaches the next state S/ (- a²)) The episode terminates at H + with R = O, then the agent proceeds to next episode.
86
87
88
For each (k E [K] x [H]. we denote by H _ (s), at sy. ......, str a.....,.ff Sh ag) the 89
(random) history up step h episode k. We define Fk - H as the history up to episode 1. We describe the interaction between the algorithm and MDP in two levels. In the level of episode, we define an algorithm as a sequence of function s/ - (oH) each mapping Fs to policy (F&&) II. We denote by x² A (Fcc the policy at episode k. In the level of step, (deterministic) policy æ is defined as a sequence of functions л = (Th)RE(H) with TA : S - ((A).
90
91
92
94
Entropic risk measure. EntRM is well-known risk measure in risk-sensitive decision- making, including mathematical finance (25), Markovian decision processes (3. The EntRM value of r.v. 96
X F with coefficient 3 * 0 is defined as
1 B log(Ex-rloxp(xx)] = log (1e (xp(BB)dd(())
1
(x
98
With slight abuse of notations, we write (F)(F) (g(()) for X F. For B with small absolute value, using Taylor's expansion we have
99
Ux(X) E[X] + V[X] + 0(37).
(1)
100
Hence for a decision maker who aims at maximizing the EntRM value, she tends to be risk-
101
(favoring high uncertainty X) 3 > O and risk-averse (favoring low uncertainty X) 3 0. (8] controls the risk-sensitivity. It exactly recovers mean as the risk-neutral objective when B + 0.
102
3
Risk-sensitive Distributional Dynamic Programming
104
has discussed the infunite-horizon distributional dynamic programming in the risk-neutral setting, which will be referred to as the classical DDP. There is a big gap between the risk-sensitive MDP and the risk neutral one. In this section, we establish the novel DDP framework for risk-sensitive control.
105
106
108
We start with defining the return for a policy π starting from state-action pair (s, a) at step h
H
(7
2
Σ
Ri (sw, ane); 8h. 8, an The (sw),, Sh441 Phr (-1ss,,, ax).
Define (s) A Zf Th (s)). There are three sources of randomness in 2% (s, a): the reward R2 (s, a), the transition pa and the next-state return + (5h+1). Denote by ví (s) and my (s,a) the cumulative distribution function (CDF) corresponding to Y (s) and Z (s, a) respectively. To the end of sensiivee control. we define the action-value function of a policy æ at step
109
110
QT (s, a) & U3 (4T (s, a)), i.c. the EntRM value of the return distribution, for each (s,a,h) E S x A x H.. The
112
113
value function is defined as V. (s) (s, (s)) Up (YT(())))
115 We focus on the control setting, in which the goal is to find an optimal policy to maximize the value function, i.e.
118
A arg
max TREE
VTI (s).
We write (T.,...,, TH) to emphasize that it is multi-stage maximization problem. Direct search suffers exponential computational complexity. In the risk-neutral case, the principle of optimality holds, the optimal policy of tail sub-problem is the tail optimal policy [5). Therein the stage maximization problem can be reduced to a multiple single-stage maximization problem. However,
118
120
the principle does not always hold for general risk measures. For example, the optimal policy for (41)
CVaR may be waaakovian/historr-deeeeee
122
We identify two key properties of EntRM, upon which we retain the principle of optimality.
3

---

124
Lemma 1.
The EntRM satisfies the following properties:
125
Additive: X = U2(X + + = (x(X) UY(Y), VX,Y.
128
pnmnttonicity-peseeviin
v0 [0,
U2(F2) S UB(Fi) = DG) s Un((1 0)P 0G).
127
The proof given Appendix B
In particular, the additivity entails that the EntRM value of the current return 27, (s, a) equals the sum of the immediate value of R2 (s, a) and the value of the future return Y (5'), i.c.,
129
Ua(Z)(s = (p Us(()
130
The
property together with the additivity suggests that the optimal future return (s') consists in the optimal current return (s,s,a) a)
131
(s, a) Ra a) + Yn (8))
132
These observations implies the principle optimality.
135
Proposition (Principle of optimality). Let = { m2. ..., TH be an optimal policy and assume when we visit some state using policy a at time-step h with positive probability. Consider the sub-problem defined by the the following maximization problem
135
max VT (s) (R (s. ()) +Ug(PPRR+ WE
136
Then the truncated optimal policy (NT Th+1 .... 3 is optimal for this sub-problem.
The proof is given in Appendix It further induces the distributional Bellman optimality equation.
Proposition 2 (Distributional Bellman optimality equation). For arbitrary initial state the optimal
139
policy (TT,, NE(B) is given by the following backward recursions:
( = V/u mh (s. a) = [Prr fa,))s,a). Th = arg max Q7 (n (s(s) ())s)
(2)
aεA
140
where fa,ss, a) is the probability density function of RA Furthermore, the sequence (ee) 141 and (V])E[[H) are the sequence of distributions corresponding to the optimal returns at each step.
142 The proof is given in Appendix For simplicity, we define the distributional Bellman operator
143 B(P,R): ge GSXAA with associated model (P, R) = (P(s, a), R(s, (s. as
[B(P, (s, a) e [Pv](s, a) (1 a), ( A.
Hence we can rewrite Equation in compact form:
v = vho mi (x, a) = [B(P,, Rr Rx)
(x) = arg max (s, (x) = v (8)), V(s, E Ax [H]. acA
145 Finally, we define the regret of an algorithm of interacting with an MDP M for K episodes as
Regret (egret(od,Ao) M,
Σ (s - V (sf).
Am
148 Note that the regret is a random variable since n is random quantity. We denote by (f K)] the expected regret. We will omit π and M if it is clear from the context.
147
4
Algorithm
149
For a better understanding of the readers, we present our algorithms under the assumption that the reward is deterministic and known]] The algorithms for the case of random reward are given
150
The algorithms for random reward enjoy the regret bounds of the same order.

---

151 in Appendix C
We denote by (r. (s.a (s....)ESSx. [i]] the reward functions. For the case of deterministic reward, the Bellman update Equation 2
152
takes the form
Th (s, a) [Prr a)(- (-- (s,
since adding deterministic reward (s, a) corresponds shifting the distribution ( i1(s, by an amount of Tha (s, a). We thus define the distributional Bellman operator B(P, R): 9 - 9SxA with associated model (P,r) (P(s, a), (rs,sa)))a.)))) ()) as
154
155
[B (s)((s))) a) _ [v][(s a)(- rn(s,a)), V(s,0) ESX..
We propose DRL. algorithms in this section, including model-free algorithm and model-based algorithm. We first introduce the Model- Free Risk-sensitive Optimistic Distribution Iteration (RODI-MF) in Algorithm [ For completeness, we introduce some additional notations here. For
157
158
two CDFs F and G over reals, we define the supremum distance between them || G] - sup₂ [xr) G(z)]. We define the & distance between two probability mass functions (PMFs)
159
180
P and Q as || - Ql1 ف E; PP Qil- We denote by (x, == {G G|| Fl S c} with radius c. With slight abuse of notations, we denote by B1 (P. e) the 4 norm ball centered at P with radius c.
181
162
the supremum norm ball centered
163
184
4.1
Algorithm overview
185
4.1.1
RODI-MF
186
In each episode, the algorithm includes the planning phase (Line 4-12) and the interaction phase
187
(Line 13-17).
168
Planning phase. In a high level, the algorithm implements an optimistic version of approximate DDP from step H + 1 to step 1 in each episode. In Line (5-7), it performs sample- cample-based Bellman update. To make clear, we introduce the superscript to the variables of Algorithm [ in episode For example, v/h denotes 7/n in episode k. Specifically, for those visited state-action pairs, we claim a) & [[ at) (s. a)).
169
170
that Line 6 is equivalent to model-based Bellman update. Denote by (s,
Fix tuple (s, a, k. h) such that a)) 2 1. We denote by P/ (+1s, a) the empirical transition model
175
(s'1s, a) (s) rE(M-1]
Σ IT(saa) 115 = s'}.
Observe that for any Φ5, we have
pk v
(s,a)= = Σ
() Σ Σ ITss 1(87+1 = s')u(s')
PES
s'ES rE(k-1]
Σ a) E1sh4 = (u(()+1)
(H a)
TE(k-1]
S'ES
Σ T(
(s, a)
TE(K-11
Hence the update formula in Line of Algorithm can be rewritten as
Th [P
(s, a)) [B(P)
ra)ett) [B(Phr
(s.
implying the equivalence to model-based Bellman update with empirical model P Alternatively, the unvisited (s. a) remains be the return distribution corresponding to the highest possible reward H + h. The algorithm then computes the optimism constants (Line 8) and enforces OFU
177
178
180
through the distributional optimism operator cf (Line 9) to obtain the optimistically plausible return
distribution //f The choice of of will be discussed later. The optimistic return distributions yields the optimistic value function, from which the algorithm generates the greedy policy o The policy #i
181
182
will be used in the interaction phase.
5

---

184 Interaction phase. In Line (15-16), the agent interacts with the environment using policy и and
185 updates the counts N based on new observations.
Algorithm RODI-MP
1: Input: 7 and 8
2: Initialize (:,-..) O: um(-)).u)() < WH+1-A for h E [M]]
3: for k = 1: K do
4: for H do
5: if (x, > then
6:
(.) Na - ru(:.))
7:
end
8:
c.(.,)) - Mu(---vi'
29
9:
m....)) t O (c
10:
Wh (-) < arg max, U (-)))())a)
Vh (-) - mi(:, (-))
12:
end for
13:
Receive of
14:
for h 1: H do
at +- nn (5 and transit to sh+1
16:
(s a + Nn (s att) +
end for
18:
end for
186
4.1.2 RODI-MB
187
We introduce the second algorithm Model- Based fliskssensitive Optimistic Distribution Iteration (RODI-MB). Algorithm 2 is model-based algorithm because it requires explicitly maintaining the empirical transition model each episode. However, it can reduced to a non-distributiona rein- forcement learning algorithm that deals with the one-dimensional values instead of the distributions, which saves the computational complexity and space complexity. Likewise, the algorithm includes the planming phase (Line 4-10) and the interaction phase (Line 11-15).
189
191
193
Planning phase. Analogous to Algorithm I the algorithm also performs approximate DDP together with the OFU principle. First, it applies the distributional optimistic operator to the empirical transition model Ph to get the optimistic transition model PV Then the algorithm uses P to execute Bellman update to generate the optimistic return distributions v/. The remaining steps are the same as Algorithm
197
Interaction phase. In Line (13-14), the agent interacts with the environment using policy nt and updates the counts N/++ and empirical transition model pk+1 based on the observations.
Algorithm RODI
Algorithm 3 ROVI
1: Input: T' and 8
1: Input: T and 6
2: ()(-.)) + 0; ()().)) $1 for all h e [H]
2: (c. -) + 0: (.())) + 1or h [H]
for k=
do
3: for = 1: K do
4:
(-) + d0
4:
(++1(+)) +- +1
5:
for
H do
5:
for h = H do
6:
(A()))) +- O O2 Ph
6:
(h()))) +- O O2
Pk (A.)))
7:
-/(;,-) - [B (Pp
(Pk k 1h (1))
7:
JK(.)) < @nalrr))
pAWk k+1 (4)
8:
()(-) - arg max, Up (n (,)
8:
Wh(+)) max, J (.aa)
9:
V/ (.) + (-))))
9:
end for
10:
end for
10:
Receive st
11:
Receive st
11:
for 1: do
12:
for = 1: do
12:
af arg max, J (s stt a) and tran-
13:
a nz. (sf) and transit s/+1 Compute A (:-)) and p p(+1(.)))
sit to 11+1
Compute N N)++((.)) and p
14:
13:
end for
14:
end for
16: end for
15:
end for

---

200
Equivalence to ROVI. Resk-sensitive Optimistic Value Iteration (ROVI) is a non non-distributional algorithm that deals with the real-valued value function rather than the distribution. is motivated by the exponential Bellman equation proposed by [22] We define the functional exponential EntRM (FERM) F.g as the EntRM after the exponential transformation
201
203
(P( - exp(B(Ux(()))
exp(3x)dF(x))
Define the exponential value functions (s) e Es(Mn(s) and JN a) & Eg(na a)) for all (s, a, h)s. Applying EERM to Equation Baatiionn@yee yields the exponential Bellman equation
205
J (s. a) exp(Bra(s. ()), Il,,,,
(4)
W (8) sign(B) max sign(3)J W (++1($)=1) =1
To verify the equivalence, it is sufficient to show that J in Algorithm corresponds the exponential function of v/h in Algorithm 2
Observe that Eg is linear in F, hence follows that
Es(ti (s, a)) = EB (1
( (s, a)) rn rn (8, exp(3ra(s,a)) Ph E (s,a)
(
exp exp(Brn a))
PA (s, (s.
208
The two algorithms generate the policy sequence in the same way, implying that their trajectories H H follow the same distribution. The formal statement is given in Appendix
210
4.2
Distributional Optimism
211
is common to add bonus to the reward to ensure optimism in the risk-neutral setting. Specifically, the bonus is closely related to the level of uncertainty, which is quantified by the concentration inequality. Yet, this type of optimism cannot adapted to the distributional setup. As one of our technical novelty, the distributional optimism is introduced for algorithmic design and regret analysis. In particular, we specify two types of distributional optimism operators, which map a statistically plausible distribution (either the empirical model the return distribution) to optimistically plausible distribution. Either of them is applied by Algorithm| 2 or Algorithm
212
213
215
217
Distributional optimism on the return distribution (in Algorithm 1. For two CDFs F and G,
we say that F is more optimistic than G (w.rt. EmtRM) if (F(F) 2 Ug( This reflects the intuition that the more optimistic distribution should own larger EntRM value. Following [31], we define the distributional optimism operator 0 ([a, b)) + ([a, 6)) with level c e (0,1) 1),1)ar
219
221
(O & [F(x) clI(a,) (x)]+
222
The optimistic operator shifts the input F down by at most e over (a, b), and retain the value b. ensures that O F remains in ([a, b)) and dominates all the other CDFs in ((a, b)) in the sense that (0 (x) < G(x) for any G E B c). Since EntRM is monotonic, it holds that
223
224
Ua F) 2 U2( AG E (F,
225
Hence F F
the most optimistic distribution in the infinity ball (F, In other words, for < c, we have O F. When specialized to the return
226
any CDF F and G satisfying distributions, we can apply the distributional optimism operator to the estimated return distribution
227
t (Line 9 of Algorithm| 1 with the constant cf: to ensure (n (s, a)) 2 Uz(n (s, a). The constant (s, - as
229
cx quantifies uncertainty in the model estimation, i.c..
Given the model, we consider the optimism among the space of PMFs rather than CDFs. Using the & concentration inequality [46], we get a concentration bound of the empirical PMF of model: with probability at least 1 -
Distributional optimism on the model (in Algorithm 2
(s) Pa(s,a) a w)]{ S
ct (s)
2S 1 (s,
log
25 (x,
230 We wish to obtain a optimistic transition model 75 (s, a) from the empirical one ( Q). To be more specific, the return distribution n/, computed from p a) and w should be more optimistic than
7

---

232
the optimal one nh (s, a) with high probability. We thus define the distributional optimism operator
0- (S) 9 (S) with level c and future return V E g as
233
o (P(s,.).)) _ arg ((aaa))
max
Up([Pv]).
The ERM satisfy an interesting property that enables an efficient approach to perform O (see Appendix []. The following holds by using the induction
235
Us ( (nt(s)))) a)) - ru (a a) + Up ( [ vit
[s, (s. 2 r)( + +U ((Pruh [s, [s)a])
2 rs, + + ((Pr [s)a])
= (s(1m()aa),
which verify the optimism of ()) over mj (s)
5
Regret Analysis
237
5.1
Regret upper bounds
Theorem 1 (Regret upper bound of RODI-MF). For any ā E (0, 1), with probability 1 - 8. the regret of Algorithm under deterministic reward or Algorithm 4 under random reward is bounded as
240
Lu (13 (
L AK AK = /exp() (1 H) H S AK
K) S Zo(
|
241
The proof is given in Appendix [
242
Theorem 2 (Regret upper bound of RODI-MB/ROVI), For any 5 E (0. 1). with probability
1 - 8, the
243
regret of Algorithm [yAlorithhm under deterministic reward or Algorithm AAlgorithm random reward is bounded as
6 under
244
Regret( RODI-MF, K)= = Regret (ROV, K) S 0
LH H /
52 AK log(4SAT/8)
|
= [ex(113
H)
HVSSAK
|
245
The proof is given in Appendix D The above results match the best-known results in 22 In particular,
algorithms attain exponentially improved regret bounds than those of RSVI and RSQ in [23) with a factor of exp(1 H²). By choosing [3] = O(1/H), we can eliminate the exponential term and achieve polynomial regret bound akin to the risk-neutral setting.
247
248
249
Compared to the analysis is distribution-centered called the distributional analysis. The distributional analysis deals with the distributions of the return rather than the risk measure values of the return. For example, involves the operations of the distributions, the optimism between different distributions, the error caused by estimation of distribution, etc. These distributional aspects fundamentally differ from the traditional analysis that deals with the one-dimensional scalars (value functions). Now we recap the technical novelty of our analysis in the following.
analysis dealing with one-dimensional values, our
251
253
255
296
Lipschitz continuity and linearity. We identify two important properties of EERM that establishes the regret upper bounds, including the Lipschitz continuity and linearity. Denote by LM the Lipschitz constant of the EERM Eg- 2(00, M) + IR with respect to the infinity norm ||-Ilce Lemma 2 provides right Lipschitz constant of EERM. The Lipschitz constant relates the difference between distributions to the difference measured by their EERM values.
258
280
Lemma 2 (Lipschitz property of EERM). Eg is Lipschitz continuous with respect to the supremum norm over Da with LM = exp(1 - 1. Moreover, LM is tight in terms of both |3] and M.
262
263
Notice that limg- oo LM = O, which coincides with the fact that limg-0 Eg = The linearity of EERM is key property that sharpens the regret bounds. In contrast, EntRM is non-linear in the distribution, which could induce a factor of exp(| when controlling the error propagation across time-steps. It would further lead to compounding factor of exp(13) H2) in the regret bound. In summary, the Lipschitz continuity property enables the regret upper bounds of DRL algorithms, and the linearity tightens the bound.
264
265
267
8

---

289
Distributional optimism. Another technical novelty in our analysis is the optimism in the face of uncertainty at the distributional level. The traditional analysis uses the OFU to construct a sequence of optimistic value functions. However, our analysis implements the distributional optimism that yields a sequence of optimistic return distributions. In particular, we first define a high probability event, under which the true return distribution concentrates around the estimated one with a certain confidence radius. Then we apply the distributional optimism operator to obtain the optimistically
270
272
279
274
plausible return distribution and the optimistic EntRM value. Hence the regret can be bounded by the surrogate regret, with the optimal EntRM value replaced by
276
Regret(K) - log (W7 (s+)
1 log (WT" (of)
1 E
W WT (sf)-
277
Distributional analysis vs. non-distributional analysis. When analyzing Algorithm Algorithm2AAlgornin Allgorithm
278
278
3
proving the regret bound of either algorithm suffices due their equivalence relation. Since K is a non-distributional algorithm, one may consider using the standard analysis that does not involve distributions. However, we show that this induces factor of [3] exp(18) H). which
280
Algorithm
282
explodes as | - O. We overcome this issue by invoking novel distributional analysis of Algorithm leading to the desired factor of [ (exp(13) HH) 1).
283
Although focus the algorithms for the deterministic reward in the main text, the regret upper bounds also hold for case of random reward. Algorithm||] Algorithm[5Jann and Algorithm (6) corresponds to Algorithm| T Algorithm 2 and Algorithm 3 respectively (cf. Appendix
285
286
287
5.2
Regret lower bound
288
We provide more details of the mistakes in the lower bound of [23] in Appendix D The proof of 23 reduces the regret lower bound to the two-armed bandit regret lower bound. Since the two-armed bandit is a special case of MDP with S = 1, A = 2 and H = 1, the reduction-based proof only leads to lower bound independent of S, A. and H. Instead, our tight lower bound follows totally different roadmap motivated by 20. 20 proves the tight minimax lower bound H / AT for risk- neutrall
289
290
291
292
293
MDP. However, the generalization to sensiscns MDP is non-trivial. The main technical challenge heavily relies on the linearity of expectation, allowing the exchange between taking the risk measure (expectation) and the summation. In the risk- sensitive setting, the non-linearity of EntRM requires new proof techniques.
294
due to the non- non-riine of EntRM. The proof
2
295
296
297
Assumption 1. Assume S 2 6, A 2 2, and there exists an integer d such that S 3 +
A" A-I
We
H further assume that H 2 3d and H A 3 2 1.
298
Theorem 3 (Tighter lower bound). Assume Assumption I holds and B >> 0. Let T, 2 (1 1)
300
3) + 3. Then for any algorithm at there exists an MDP Met such that for K 2 2exp(3 H
301
d)) HTA we have
1 cxp(3H/6) - HVSAT.
E|Regret of MM 72V6
31
302
The proof is given in Appendix Theorem 3 recovers the tight lower bound for standard episodic
303
MDP, implying that the exponential dependence on | and H in the upper bounds is indispensable.
304
Yet, it is not clear whether a similar lower bound holds for 3 < O, which is left as future direction.
6 Conclusion
305
306
We propose risk-sensitive distributional dynamic programming framework. We devise two novel DRL algorithms. including a model-free one and a model- one, which implement the OFU principle at the distributional level to balance the exploration and exploitation trade-off under the risk-sensitive setting. We prove that both attain near-optimal regret upper bounds compared with our improved lower bound.
307
309
310
311
There are several promising future directions. The current regret upper bound has an additional factor V HS compared with the lower bound. It might be possible to remove the factor by designing new algorithms or improving the analysis. Besides, it is interesting to extend the DRL algorithm from tabular MDP to linear function approximation setting. Finally, it will be meaningful to investigate whether the DDP framework holds for other risk measures.
312
314

---

References
316
317
[1] Mastane Achab and Gergely Neu. Robustness and risk management via distributional dynamic programming. arXiv preprint arXiv:2112.15430, 2021.
319
[2] Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional determin- istic policy gradients. arXiv preprint
320
321
2018.
322
[3] Nicole Bäuerle and Ulrich Rieder. More risk-sensitive markov decision processes. Mathematics of Operations Research, 39(1)100--120 2014.
323
[4] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforce- ment learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
325
326
327
[5] Dimitri P Bertsekas et al. Dynamic programming and optimal control: Vol. 1. Athena scientific Belmont, 2000.
328
329
[6]
Tomasz R Bielecki, Stanley R Pliska, and Michael Sherris. Risk sensitive asset allocation. Journal of Economic Dynamics and Control, 145-
330
331
[7] Vivek S Borkar. A sensitivity formula for risk-sensitive cost and the actor-critic algorithm. Systems & Control Letters, 44(5):339-34,, 2001.
332
88 Vivek S Borkar. Q-learning for risk-sensitive control. Mathematics of operations research, 27(2):294-311, 2002.
334
335
99
Vivek S Borkar. Learning algorithms for risk-sensitive control. In Proceedings of the 19th International Symposium on Mathematical Theory of Networks and Systems- volume 5, 2010.
337
338
110] Vivek Borkar and Sean P Meyn. Risk-sensitive optimal control for markov decision processes with monotone cost. Mathematics of Operations Research, 17(1):192-209 2002.
340
[1]] Rolando Cavazos- and Daniel
Discounted approximations for risk-sensitive average criteria in markov decision chains with finite state space. Mathematics of Operations Research, 36(1): 1331))133-1444 146, 2011.
341
342
343
122 Stefano P Coraluppi and Steven Mareus. Risk-sensitive, minimax, and mixed risk- neutral/minimax control of markov decision processes. In Stochastic analysis, control, opti- mization and applications, pages 21-40. Springer, 1999.
344
345
346
[13] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1 1096-1105. PMLR, 2018
348
349
[14] Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
351
352
[15] Mark Davis and Sébastien Lleo. Risk-sensitive benchmarked asset management. Quantitative Finance, 8(4):415 426, 2008.
354
[16] Erick Delage and Shie Mannor. Percentile optimization for markov decision processes with parameter uncertainty. Operations research, 58(1):203-213, 2010.
355
356
[17] Giovanni B Di Masi al. Infinite horizon risk sensitive control of discrete time markov processes with small risk. Systems & control letters, 40(1):15-20, 2000.
357
358
[18] Giovanni B Di Masi and Lukasz Stettner. Risk-sensitive control of discrete-time markov processes with infinite horizon. SIAM Journal on Control and Optimization, 38(1):61-78, 1999.
[19] Giovanni B Di Masi and Eukasz Stettner. Infinite horizon risk sensitive control of discrete time markov processes under minorization property. SIAM Journal on Control and Optimization, 46(1):231-253 2007.
361
10

---

363
[20]
Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory, pages 578-598. PMLR, 2021.
384
365
366
[21] Damien Ernst, Guy-Bar Stan, Jorge Goncalves, and Louis Wehenkel. Clinical data based optimal sti strategies for hiv: a reinforcement learning approach. In Proceedings of the 45th IEEE Conference on Decision and Control, pages 667-672. IEEE, 2006.
367
369
[22] Yingjie Fei, Zhuoran Yang, Yudong Chen, and Zhaoran Wang. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
370
372
[23] Yingjie Fei, Zhuoran Yang. Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk- sensitive reinforcement learning: Near-optimal risk fisk-sample tradeoff in regret. arXiv preprint arXiv:2006.1388 138277 2020.
373
[24] Wendell H Fleming and William M McEneaney. Risk-sensitive control on an infinite time horizon. SIAM Journal on Control and Optimization, 3386)11881 1995.
376
377
[25]
Hans Föllmer and Alexander Schied. Stochastic finance. In Stochastic Finance, de Gruyter, 2016.
379
[26] Aurélien Garivier, Pierre Ménard, and Gilles Stoltz. Explore first, exploit next: The true shape of regret in bandit problems. Mathematics of Operations Research, 44(2)377-399, 2019.
380
381
[77]
Lars Peter Hansen and Thomas Sargent. Robustness. In Robustness. Princeton university press, 2011.
383
[28]
Daniel Hrmnändez-Heránnder and Steven I Marcus. Risk sensitive control of markov processes countable state space. Systems & control letters, 197-515 1996.
384
385
[29]
Ronald Howard and James Matheson. Risk-sensitive markov decision processes. Manage- ment science, 18(7):356-366 1972.
386
[30] Anna Jaškiewicz. Average optimality for risk-sensitive control with general state space. The annals of applied probability, 17(2):654-675, 2007.
386
309
[1]
Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a evar policy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4436-4443) 2020.
381
382
[32] Clare Lyle, Mare G Bellemare, and Pablo Samuel Castro. comparative analysis of expected and distributional reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 4504-4511, 2019.
394
385
[33] Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004. 14547, 2020.
396
397
398
[34] Yecheng Ma, Dinesh Jayaraman, and Oshert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
389
[35] Steven I Marcus, Emmanual Grnnandez-Gauchraand Daniel Hemáández-Hernandee, Stefano Coraluppi, and Pedram Fard. Risk sensitive markov decision processes. In Systems and control in the twenty-frst century, pages 263-279. Springer, 1997.
401
[36] Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 4922)2267- 2002.
404
405
[37] David Nass, Boris Belousov, and Jan Peters. Entropic risk measure in policy search. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1101-1106.
407
IEEE, 2019.
408
[38] Takayuki Osogami. Robustness and risk-sensitivity in markov decision processes. Advances in Neural Information Processing Systems, 25:233-241, 2012.

---

410
[39] Stephen D Patek. On terminating markov decision processes with a risk-averse objective function. Automatica, 3779):177--1386, 2001.
411
412
[40]
Mark Rowland, Mare Bellemare, Will Dabney, Rémi Munos, and Yee Whye Teh. An analysis of categorical distributional reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 29-37. PMLR, 2018.
413
414
415
[41] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszezynski. Lectures on stochastic programming: modeling and theory. SIAM, 2021.
416
[42]
Yun Shen, Wilhelm Stannat, and Klaus Obermayer. Risk-sensitive markov control processes. SIAM Journal on Control and Optimization, 51(5):3652-3613 2013.
418
[43] Yun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcemnnt learning. Neural computation, (6(7):1298-1133 2014.
420
[44] Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distributional reinforcement learning. In Learning for Dynamics and Control, pages 958-966 PMLR, 2020.
422
424
[45] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
425
426
[46] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the Il deviation of the empirical distribution. Hewlet--Pache Labs, Tech. Rep, 2003.
427
428
Checklist
429
430
1. For all authors...
431
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
432
contributions and scope? [Yes]
435
(b) Did you describe the limitations of your work? [Yes]
434
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
436
437
2. If you are including theoretical results...
438
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
439
(b) Did you include complete proofs of all theoretical results? [Yes]
410
3. If you ran experiments...
441
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
443
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [N/A]
444
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
446
447
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
448
419
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
450
(a) If your work uses existing assets, did you cite the creators? [N/A]
451
(b) Did you mention the license of the assets? [N/A]
452
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
453
454
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
12


Reviews:
[{"review_id": "hSFF4U8z8MA", "reviewer": "Reviewer_wafk", "summary": "This paper considers finite episodic Markov decision processes aiming with risk-sensitivity. Two properties are identified to develop both the model-based and model-free algorithms for risk-sensitive distributional dynamic programming.  Both upper and lower bounds for regret haven been proved.", "questions": "How does the performance of the proposed algorithms compare to that of DRL algorithms using CVaR? Is it possible to extend the proposed algorithms for continuous control tasks, such as Cartpole or more complicated robotic learning application? Can the authors clarify the unique novelty of the regret analysis technique?\n\n", "limitations": "Yes, the authors have discussed the limitations of their theory. However, I mean some study on the actual performance is necessary for this paper. The proposed regret guarantees are meaningful only when the algorithms actually do perform well. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. A specific entropic measure is used, enable the derivations of useful distributional Bellman optimality equation as well as risk-sensitive distributional dynamic programming. This contribution is very interesting. \n\n2. OFU principle has been used to develop deep RL algorithms with provable regret bounds. \n\n3. An improved lower bound has been obtained when beta is positive. \n\n4. The paper is well written. The presentation is clear.\n\n\nWeakness:\n1. This paper focuses on the episodic MDP setting. The most common setting in the risk-sensitive control literature is the setting with continuous state/action spaces (see Jacobson 1973 or Whittle 1990). Then stability also becomes an important issue. It is unclear how useful the results in this paper will be for continuous control applications. Maybe rephrase and focus on \"risk-sensitive MDP\"?\n\n2. This paper proposes two algorithms with provable regrets. However, it is unclear whether the proposed algorithms will really perform well on any MDP tasks. Numerical examples are needed in this case. This is different from analyzing existing algorithms which already perform well.\n\n3. CVaR vs Entropic Risk Measure (ERM):  CVaR has been used a lot in deep RL tasks and seems to be a better risk measure than ERM for deep RL. One reason why ERM was popular in some early risk-sensitive control literature is that for linear dynamics, ERM leads to a dynamical programming solution with relatively simple analytical form. In the deep RL setting, it is unclear whether ERM has any practical advantages over CVaR.\n\n4. The regret analysis technique is not that new (similar analysis has been used to study episodic MDPs (without ERM) before).\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gnByhT0mhP", "reviewer": "Reviewer_SdUB", "summary": "This work proposes to build a distributional dynamic programming framework to optimize the entropic risk measure of the return. Under the new framework, both model-based and model-free algorithms are proposed, which achieve regret bound that matches one of the existing works in risk sensitive control. The lower bound of rish sensitive control problem is also improved using the new framework.", "questions": "1. In Appendix A of [22], they also provide connections between risk-sensitive RL and distributional RL through the exponential Bellman equation. How is the proposed framework related to theirs? It would be helpful if there's a section dedicated to explain the relation and potential difference w.r.t. their analysis.\n\n2. If Alg. 2 is actually equivalent to Alg. 3, what is the benefit of distributional framework over existing risk sensitive control approaches? E.g., what are the cases when the proposed framework is more favourable, either theoretically or empirically? \n\n3. Does the improvement in the lower bound in Section 5.3 come from the newly proposed framework, or just different proof techniques? ", "limitations": "The limitation of the proposed method are discussed, but not adequately.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The proposed distributional dynamics programming framework is novel to my best knowledge. Differences in the analysis between distributional RL and previous work [22] are discussed, which shows that the distributional analysis can avoid the explode factor and shows the significance of the proposed framework in theory.\n\n2. The presentation of the paper is clear. Key steps in the analysis are highlighted, with detailed discussion on how they would affect the final regret bound. Algorithm designs are also explained in detail.\n\nWeaknesses: \n\n1. Since this work is highly relevant to [22], the significance of the work can benefit from a more comprehensive comparison  w.r.t. to previous work [22]; See Questions 1 and 2. Also, further discussion on another relevant work [1] can also improve the significance. \n\n2. The novelty of the proposed algorithms can benefit from further discussion. The equivalence of the proposed model-based algorithm to ROVI is discussed, which raises a natural question: if distributional RL is somewhat equivalent to the existing algorithms, existing analysis could also apply to DRL, then is the claim in line 10 \"this is the first regret analysis of DRL\" still valid?\n\n[22] Yingjie Fei, Zhuoran Yang, Yudong Chen, and Zhaoran Wang. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QknJAJxacpC", "reviewer": "Reviewer_CsUD", "summary": "The paper deals with a risk-sensitive reinforcement learning problem where the risk functional is the entropic risk. The authors provide both the regret upper bounds for a proposed model-based algorithm and a model-free algorithm and a tighter regret lower bound compared to prior works. \n", "questions": "- The authors suggest that the lower bound proof in [22] has a mistake. Can the authors elaborate more on this?\n- Can the authors compare their proposed algorithms with that proposed in [22] in a more detailed way to explain how optimism may have been used differently in this paper and what that implies?\n\n\n----\n\nStylistic suggestions:\n\n- Using ERM to refer to entropic risk measure may be confusing since ERM is known to be Empirical Risk Minimization in standard ML literature.\n- \\citep should be used for the first two citations in L101.\n- In L168, (s,a)s may be confusing to the reader.\n", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The ideas in the paper are clearly explained. A tighter lower bound is presented in the paper. \n\nWeakness: Given that prior works, e.g., [22, 23], have also used optimism to solve the exact same problem and achieve similar regret guarantees, it is unclear how much novelty/significance the newly proposed algorithms are. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6qNKVYCYgfo", "reviewer": "Reviewer_vdwj", "summary": "This paper is about distributional reinforcement learning (DRL), which aims at learning optimal policies (in terms of total payoff) in a Markov decision process. While the classical reinforcement learning (RL) paradigm is based on the expected total return, in DRL one is interested about the whole probability distribution of this total return. \n\nIn this work, the idea is to leverage this distributional information for risk-sensitivity purpose.\nThe focus is on the entropic risk-measure (ERM) of the total reward in an episodic MDP. Properties of ERM are given (additivity and monotonicity preserving) leading to an optimality principle akin to the classical one based on expected value.\nTwo algorithms (one model-free and the other model-based) called RODI are proposed: they both rely on the OFU principle applied to ERM.\nWhile most risk-aware DRL papers have no regret analysis, here both upper and lower regret bounds are provided.", "questions": "(Minor) questions/comments:\n\n- line 80: more accurate to say that P_h maps state-action pair to probability distribution over S, i.e. P_h : SxA -> Delta(S)\n- line 108: replace Z(s) by Y(s)\n- footnote page 4: strange formulation? maybe rather say: \"The algorithms for random reward enjoy regret bounds of the same order\"\n- notational issue: at line 176, reference to \"c_h^k\" but in Algorithm 1 it is just written \"c_h\". Same holds for eta_h. Is it normal or a notation issue?\n- what is f_h in Proposition 2?", "limitations": "The scope of the paper is limited to the ERM criterion, but it is a reasonable \"limitation\" in order to design a risk-sensitive DRL approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and motivated by risk-sensitive applications. The DRL perspective makes a lot of sense to tackle risk-aware problems. The main strength of the paper is that its algorithmic RODI procedures are naturally derived from DRL applied to ERM and combined with the OFU principle.\n\nThe main weakness is the lack of numerical experiments, which is balanced by the theoretical nature of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject