Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
Authors: Yash Jhaveri
Harley Wiltzer
Patrick Shafto
Marc G Bellemare
David Meger
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
distributional reinforcement learning
entropy-regularized reinforcement learning
Primary area: reinforcement_learning
TL;DR: We present a theoretical framework for policy convergence in RL, which permits convergence of return distribution estimates.

Abstract: In the pursuit of finding an optimal policy, reinforcement learning (RL) methods generally ignore the properties of learned policies apart from their expected return. Thus, even when successful, it is difficult to characterize which policies will be learned and what they will do. In this work, we present a theoretical framework for policy optimization that guarantees convergence to a particular optimal policy, via vanishing entropy regularization and a *temperature decoupling gambit*. Our approach realizes an interpretable, diversity-preserving optimal policy as the regularization temperature vanishes and ensures the convergence of policy derived objects--value functions and return distributions. In a particular instance of our method, for example, the realized policy samples all optimal actions uniformly. Leveraging our temperature decoupling gambit, we present an algorithm that estimates, to arbitrary accuracy, the return distribution associated to its interpretable, diversity-preserving optimal policy.

Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
Yash Jhaveri" Rutgers UniversityNNwwark Newwrk
Harley Wiltzer'
Patrick Shafto
Mila Québec AI Institute McGill University
Rutgers University- Newark
Marc G. Bellemare! Mila-Quêbec AI Institute
David Meger
Mila-Québec Al Institute
McGill University
McGill University
Abstract
In the pursuit of finding optimal policy, reinforcement learning (RL) methods generally ignore the properties of learned policies apart from their expected return. Thus, even when successful, it is difficult to characterize which policies will be learned and what they will do. In this work, we present a theoretical framework for policy optimization that guarantees convergence particular optimal policy, via vanishing entropy regularization and temperature decoupling gambit. Our approach realizes an interpretable, ularization temperature vanishes and ensures the convergence of policy derived objects-valus value functions and return distributions. In a particular instance of our method, for example, the realized policy samples all optimal actions uniformly. Leveraging our temperature decoupling gambit, we present an algorithm that esti- mates, arbitrary accuracy, the return distribution associated to its interpretable, Dnversity-preseryy optimal policy.
optimal policy as the reg-
Introduction
In generic Markov Decision Processes (MDPs), many optimal policies exist. Thus, while certain policy optimization approaches can ensure convergent approximation to an optimal policy, they do not have control over which states these policies will visit, which actions they will play, which long-term returns they can achieve. Indeed, the non-uniqueness of optimal policies renders any discussion of the properties of optimal policy ambiguous, beyond its expected value.
partial remedy to this problem is to regularize the RL objective in order to induce uniqueness. One popular approach regularization is to penalize the value of policy according its KL divergence to a reference policy -- This branch of RL is known as entropy-regularized RL (ERL). ERL, for any positive regularization weight (also known temperature), optimal. Moreover, tabular MDP, --ppimal policies and their derived objects (value functions, occupancy measures, and return distributions) converge to classically optimal policies and their derived objects. However, beyond tabular MDPs, the evolution of 7-optimal quantities, as a function decay the temperature zero, are, in
and only policy is
of the temperature, not well understood. Thus, some sense, back to where we started: living in ambiguity.
In this work. we introduce temperature decoupling gambir, through which guarantec the convergence of resulting policies and their derived objects the vanishing temperature limit,
"Equal contribution. Correspondence yash. Snhaveri@rutgerr edu, wiltzerhemila. lquueee................ CIIARR Al Chair.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Much like how gambit in chess sacrifices an immediate and shallow proxy of the objective (e.g.. material count) for long term positional advantage, the temperature decoupling gambit plays notably suboptimal policies for the 7-ERI. objective to ensure convergence to RI. optimality as T- This scheme entails estimating action-values under a target regularization temperature while playing policies with an amplified temperature. Furthermore, we characterize this limiting policy as a modification of the reference policy which "filters out" suboptimal actions. Even when T-optimal policies converge in the vanishing temperature limit (such in tabular MDPs), the limiting policy produced by the temperature decoupling gambit is distinct from the limiting policy found otherwise. The limiting policy found via our gambit preserves, quantifiably, more state-wist siite-wise& action diversity. Moreover, we show that this limiting policy achieves a notion RL, characterized by new Bellman-like equation, whose unique fixed point upper bounds the (RL) performance of 7-optimal policies general.
-orrencc-opiialityy for
Our analysis additionally sheds light on the convergence of return distributions the central objects of study in distributional RI. (DRL. [6]. While optimal policies achieve the same return in expec- tation, they may vary drastically in other statistics, such as variance. In safety-critich applications, for example, understanding the distribution over returns crucial. DRL provides techniques for estimating return distributions, primarily based on distributional dynamic programming methods which generalize dynamic programming approaches for estimating expected returns. However, is well-known that existing distributional methods do not produce convergent iterates in the control setting [5]. Leveraging our convergence results for policies in ERL, we define the first algorithm for accurately estimating reference-opting return distribution, the return distribution associated to the interpretable, diverse policy realized by the temperature decoupling gambit.
2
Preliminaries
Given a Borel set S C. R", for some N. we let (S) and (S) denote the space of Borel measurable and bounded Borel measurable functions on S respectively. We let (S)) denoted the space of Borel probability measures on S. From Borel sets. Moreover, for any E (Y) with Y C Thm and any measurable function f + S, the push-forward of p by f is f#p == po f-1 E (S)). Here f is the preimage of f-
on, measurability will always be with respect to
We single out two particular functions. The function proj : YI X ..... x Yn + YB defined by = yk is the projection function of Y onto Yy- We note that the push- forward of the projection map marginalization: nM := proj 3 the Y-marginal of μ E P (Y X - IR is defined by ba.b(2) := + bz from [6].
proj Y= (*y)) (y.1..... V....... yn
The bootstrap function Da,b functioo
Our analysis works with conditional distributions, which we formalize as probability kernels, as well tensor-product notation constructing product measures and for disintegrating product measures. For any Y C The and Z C TR" the space of (Borel) probability kernels from Y to Z, denoted K(Y, S (Z)). is the set of all indexed measures À for which y + (S) is measurable for each S E (Z), the Borel subsets of Z. Given À E K(Y, (T))) and E P (Y), the generalized product
measure @ p E (Y Z) is defined as follows:
/odA p) = [ My =) d))(())))))))))))))))))))))))d dply) Vo E LMMYxX))
Additionally, we can disintegrate any μ E (Y x Z) generalized product between either of its marginals and the induced conditional probabilities:
r TH & nt An important subset of K(Y, P (Z)) consists of those kernels with bounded pth moments,
where Dd proj and T# E K(Y, (Z))
R' (Z)) = { E(Y
sup [=P'd)y())) < oo for P [1, yeY
which can be metrized as complete metric spaces. In this work, we consider their metrization via the following metrics based on the Wasserstein metries [40] dp.
(xx 1/9
(A, = (p X,) and (2, (x, x=
(2.1)
2

---

where 9 E [1, c0) and w E (YY). These metrize topologies on K" (Y, akin to the weak topology on probability measures with finite pth moments.
2.1 Markov Decision Processes and Reinforcement Learning
discounted MDP is a five-tuple (X, A.A,, Here X C IR is the state space, C IR" is the action space, r M&(X A) is the reward function, and y (0, 1) is the discount factor.'
Central to RL are policies. policy is a probability kernel K(X, (AA). Policies induce state transition kernels pa as well as a state-action transition kernels p given by
P proj (Pa... (Ps.. & Ta) (X) and P
@ Pa.o IX A),
respectively. Therefore, policies yield sequences of states as well as state-action pairs, labeled (ST) and (XT,
, AT)ezo respectively, whose sequences of laws (vī )22 and (nz 1220 are given by
@ yo for some PO E (X). Given vo E (X), the long-term behavior of any policy т can be encoded via its (discounted, state-action) occupancy measure μ², the set of which we denote by (v6).
Mit PT with v0 Vo and p7+1 := P" with Mg
{{11 10 E ((XxA) p² 7) Σ Y Pppī for some E K(X, P(A))
(vo))= =
Policies also induce return distribution functions 5" E EK(X x A, (R)) and "7 E K(X, (RR)).
S law Σ>'((x, A7)
X5 x. Ag
and T : proj & (C., @ T2)
whose means, the action-value functiin of E M x A) and the value function of E Mx().
9" (x,a)= = Eg~cg [2] and 2d (x) == EGiing [G],
lead to the RL objective: find a xt K(X, (A)) such that q²¹ 2 for all π. Such policy is called optimal. Generally, many policies optimal. However, their associated action-value functions are identical (see [32]). We denote this optimal action-value function by q".
2.2 Entropy-Regularizee Reinforcement Learning
In ERL, the value of policy is penalized by how far it diverges from fixed reference policy ref K(X, P (A)). In particular, the EELL problem with temperature 0s
(1²) where (+) / rd RR(()) and R(n) = KT. me (xr). (=(((
sup
When T = 0, we recover the linear programming formulation of the (expected-value) RL objective. In ERL, the regularizer R is strictly convex. Thus, J, ia strictly concave and its maximizer unique.?
Lemma 2.1. The functional R PX x A) + IR is strictly convex.
[Proof]
Given Lemma 2.1, one might hope that the well well-posednnss. of 7-ERL could be realized through simple, yet power methods like the direct method the calculus of variations. However, outside the tabular case, this is unclear, for many reasons, the first of which is that M, (X A) is not separable.
The well-posedness of ~ERL, however, can be established through other means. In particular, in T-ERL, only one optimal policy exists, and it is characterized as Boitzmann- (BG) policy. Detinition 2.2. Let q (XX A) and O. We denote the Boltzmann Gibbss policy associated to and T by 9,g, and it is characterized by
/ ca(zras/T (
(a))))z(() = (VV,).a)-(V,)(e)))) d (a) with (V+4)(x) = 7 log
We note that (9+g). well-defined if and only if (v) E RR
We expect many of our results can be extended Polish spaces.
2 The only work we are aware of that establishes a comparable result is [28]. However, their result is on tabular MDPs and establishes convexity on (1o)) not all of (X x A).
3

---

More specifically, it is well-known that the optimal policy of r-ERL is the BG policy associated to the unique fixed point of of the soft Bellman optimality operator B M x A) - M(X A),
(B,a)(r, = == + ( (V.a)e")
(See Lemma A.7.) The following theorem summarizes the well-posedness of T-ERL.
Theorem 2.3. Let v > O. The policy x x²** := 9 is optimal, and uniquely so. More precisely, for all vo. V/S E 9 (XX), we have that arg max d, x arg max maxo(v)) (v1)) dr.
[Proof]
In Appendix A, we prove Theorem 2.3 as well as collection of supporting and related results that generalize well-known results in tabular MDPs. We include them for completeness.
In the remainder of this work, study the evolution of T-optimal objects vanishes. In the tabular regime, where M&(X x A) is separable, one can establish the existence and uniqueness of a 7-optimal occupancy measure: p.-. Furthermore, under a compatibility assumption, one can prove that the limit of the sequence (1,, ) as vanishes exists and is unique as well.
Assumption 2.4. The intersection of { (arg sup do} and {R < оo} is nonempty.
Assumption 2.4 asks that regularizer isn't identically too the set of optimal policies. Without such an assumption, t T-ERL and RL have no meaningful relationship, as we shall see in Section 3. Theorem 2.5. Suppose that r M₁(X A) and that X x A is finite. For every T > O, let μł be the maximizer of a, over (vv)). If Assumption 2.4 holds, the sequence (μ+)>>o has unique setwise limit as T tends to zero. This limit p is the minimizer of over arg supo(va) Io.
[Proof]
Consequently, in the tabular setting, the sequence (x==">>> has unique limit.
Remark 2.6. Even if Theorem 2.5 could be extended to hold true in continuous MDPs, occupancy measure convergence does not guarantee policy convergence, outside 2.5 is statement about sequence of join! distributions. A policy convergence statement would be one about a sequence of conditional distributions (i.e., probability kernels). In general, the convergence of a sequence of joint distributions does not imply the convergence of the associated sequence of conditional distributions with respect to fixed marginal (see, c.g., [7, Example 10.4.24]). While it is possible that the structure (1o) permits type of policy convergence, we unaware of any such result for continuous MDPs.
tabular setting. Theorem
Convergence to Optimality: The Temperature Decoupling Gambit
While ERL has unique solution, this identifiability comes a cost with respect to RL: the resulting policy is suboptimal for RL. In this section, we analyze 3.9 and 3.10- show that policies and their return
limits in ERRL.
Our main results for this distributions converge under the scheme of Definition 3.7 to interpretable, optimal limits as v +
To understand the ways in which -ERL converges to RL, we define (new) re -ensitive variant of the Bellman optimality operator, the Bellman reference-optimality operator. We call its unique fixed point the reference-optime
action-valus fanction.
Lemma 3.1. Let r E M2(X A), 7 < 1, and B
M(X x A) + +M(X x A) be defined by
(B a) rzz, a) +7 / ess sup g(2', (.x) d
Then B is contraction on M2 A). Thus, has a unique fixed point Gee-
[Proof]
Generally, Let is distinct from q² Yet, ERL recovers Lor in the vanishing temperature limit. Theorem 3.2. We have that q: - g'et monotonically as T
[Proof]
Theorem 3.2 implies that optimal policies, in general. cannot be recovered by taking vanishing temperature limits ERL. We formalize a notion of reference-optimale to highlight this distinction.
Definition 3.3. A policy x E K(X, (A)) is said to be reference-optimal (against T rf ar 2 Giet Moreover, is said be e-reference optimal if qx giet E.

---

Generally, Viet < q" For instance, consider an MDP with one state 1 (a bandit), A = [0. 11, and Ti = =(AA). fr(,,) = 61,2, then SupA q* (L.)) = 1, while ess sup, (= -) = 0. However, in many interesting cases, reference-optimal policies are optimal in the classic sense. When A is discrete and n supported on A-a ubiquitous assumption in ERL-then indeed q¹ Feet Likewise, when A is continuous and (P, r) satisfy certain regularity conditions, then q" is continuous [20]. In these case, a reference-optimal policy is optimal.
When Feet # q², even state-of-the-art continuous-come control methods, entropy-rege reguularege or otherwise. can at best hope achieve gret. and not q" This is because, when get * q", optimal actions form measure O set. And so, even rich policy classes, such as neural-network paraeo--pprmmeterizzeedd Gaussian policies [19] or diffusion policies [9] will not sample these actions, with probability Thus, moving forward, we establish 9ief "skyline" for optimal performance. In other words, we strive to achieve convergence to reference-optimal policies.
Under the next assumption, we can derive convergent policy optimization schemes as t tends to zero
Assumption 3.4. A constant Pref > O exists for which
inf inf oe (La E A: (2, a) oss m Tf (x. -13) Pref- T>0xEX
Remark 3.5. If is discrete and T/a is uniformly lower bounded, Assumption 3.4 holds. This is standard assumption. When A is continuous, this assumption more difficult to guarantee. Intuitively, it asks that there is enough mass surrounding the optima of the rerroy--egglarized optimal value functions 9,7 for KL((S-45) o to remain bounded in the limit.
A result key to the remainder of our work is the following bound on the total variation distance between pairs of BG policies in terms of their temperature and the distance between their potentials. Theorem 3.6. Let of M(X x A). For any T > 0 and any X,
[[(9,9)= (5,g'))
< <min
V T- (g(xx,) -- (I, -I, (= 2
sinh (1g(2,) - ( (x, (1))))))))))))
In particular,
1(9-9), (0:0))=((99)))m... (9,99)x - 2 -3 T -11((1,))))) - - (zx (Z,,)))
if (1))(x,) -) qx(x (x(x,)))1. (1g))))))) T/2.
[Proof]
While o. and V.g= converge in the zero-temperature limit, whether or not r-regularized optimal policies T.** converge is still unclear. Indeed, under Assumption 3.4, l1 q (c T (see Lemma B.10). However, the log- probabiliities of an action under ग** are amplified by -- Hence, the total variation difference between the BG policy at temperature т and potential dief and x may not vanish as vanishes. Based this insight, we introduce the temperature decoupling gambir.
Definition 3.7. Given т > 0, the temperature decoupling gambit specifies an alternate temperature (T) and constructs x T. = S. In particular, it requires that a
as 0.
At any 0, terpuerrtdttmprrauuuree policies -. are necessarily not optimal for the r-regularized problem. Nevertheless, unlike π T the policies -. produced by the temperature decoupling gambit realize long-term advantages: they have convergence guarantees the vanishing temperature limit, and they recover an interpretable reference-optimat policy.
Definition 3.8. Let q" denote the optimal action-value function in a given MDP, and let ref K(X, P (A)). The aptimality-ilteree reference policy ref, is defined by
T ref.*
x Tzz C (x) where (ir(()) = {a E A q' a) = ess sup (x(x,)))
Here XY is the characteristic or indicator function for the measurable set Y.
Heuristically, the optimality-filtere reference n.eef. is the restriction of - onto the set of expected- value-optimal actions in the state x.3 When , is the uniform random policy, that is, n = ((A)
3 This is exact when diet-
5

---

for all I E X, we see that प.. = u(N fer (z)))()))) (z)) the uniform policy on optimal actions. In sense, ref is the most diverse (reference-)optiie
policy; it does not discriminate between optimal actions.
In general, even when n² does converge т converges to zero, its limit is different from re We demonstrate this explicitly in Section 3.1. On the other hand, our next result proves that the temperature decoupling gambit enables convergence to x xeef,*
Theorem 3.9. Under Assumption 3.4, if = ((T) is such that lim,-o o/T Ttr///0 = 0, then T T.a + T ref as T + for all z E X, in TV if A is discrete and weakly if A is continuous.
[Proof]
At the heart of the proof of Theorem 3.9 is the following inequality (a direct consequence of Theorem 3.6 and Lemma B0),), which relates the BG policies at temperature т and potentials 9 and Pef
lim sup |1(9,))= (St9fer) X lim T-0
log Pref
This inequality reduces questions of convergence 9,92 to those of S- (the vanishing temperature limit of a BG policy with fixed potential is well-studied). Note that the smaller the fraction o //T is, the closer these two policies are. For instance, taking ((r) = 7 ensures that 9 1 is more like 9, leet than taking (T) = 72. In particular, it is from this inequality that the temperature decoupling gambit's requirement that O/TT + as T + arises.
Beyond enabling policy convergence in the vanishing temperature limit, the temperature decoupling gambit also ensures return distribution function convergence.
Theorem 3.10. Suppose A is discrete and Assumption 3.4 holds. If = (T) is such that O/TT - 0 as - O, then, for any p.p' E [1, oo) and w E IP x A), as T 0, the return distribution functions ÇT the the temperature
policies π - satisfy dppp' (CT.0 ( ->
[Proof]
While Theorem 3.10 does not yet provide an algorithm for approximating (*, this result serves as inspiration for such developments in Section 4.
3.1 Numerical Demonstration
In this section, we demonstrate that the policies learned via the temperature decoupling gambit differ from those learned in ERL, even in the presence of stochastic updates.
Figure 3.1 shows given tristate MDP with two actions (blue: a1: green: a2), as well learned policies # and a TT estimated with soft Q-learning [18]. Here ref = u(A) for all æ E and y = 0.9. As this MDP is tabular, Theorem 2.5 implies that the policies ग* converge 7 - 0. Thus, the temperature decoupling gambit is not necessary to guarantee convergence. Yet we see different limiting behavior. As predicted by Theorem 3.9, the estimates tTe converge to ref, as t O. With uniform re this is the policy that samples all optimal actions, given a state, with equal probability. As + O, the estimates * do converge in Vo. where * collapse to 60g- We take policies found emphasize different notions of diversity. The limit of i filters out optimal actions in order play actions more uniformly on average with respect to state occupancy in the long term, while the limit of π looks to maximize state-wise action diversity.
different optimal policy. This difference 22, in line with Definition 3.7. The two optimal
T..
A
A
Action probabilttie
1E-02
Tempperature
IE-01
06
1F-07
1E:03
1E:04
15:05
Acice
Aciin
Action Action
Acian Aciion
Arrie Amtza
Arice Ar
Arize Arto
Figure 3.1: Differences between T and #T. approximated with soft Q-learning. Left: Graphical model of the MDP: arrow colors encode actions. Center: Depiction of the estimated policies **** each state, as r - 0. Right: Depiction of the estimated policies n. each state, as r - Summary: Learned policies differ in zo, but are otherwise the same.
We discuss the benefits of this optimal policy in Appendix D.

---

4 Convergent Approximation of Optimal Return Distributions
In this section, we formalize new branch of DRL and introduce distributional ERL (DERL). Our main results in this section, Theorems 4.5, 4.6, and 4.7, establish convergent iterative schemes for approximate peerrence-joptinan return distribution estimation. In Section 4.1, we introduce novel soft distributional Bellman operators, for evaluation and for control, and establish the convergence of their iterates. The behavior of the resulting return distribution approximations in the vanishing temperature limit is treated in Section 4.2. To conclude, simulation is presented in Section 4.3 to illustrate the resulting optimal return distribution approximations.
4.1 Entropy-Regularized Distributional Reinforcement Learning
We begin by defining soft distributional Bellman operator, as an analogue to the distributional Bellman operator [5, 35]. It, under certain conditions, computes
SF,a = law ((X8 A.)
(r(xF) TKL(TXp || m X5 z, A5 = =))
Notationally, for any π E K(X, 9 (A)). we define k1[T] X + R via (x) KL || b
Definition 4.1. For any T 0, < and K(X, P (A)). the soft distributional Bellman operator JT is given by
(Ia))aa=
(br(a,a), oproj - [r]] proj*) # (5... 2 Pr. P2.))
Theorem 4.2. Ifr E Mx(x x A), << < 1, and π E K(X, (A)) is such that
sup || < 00,
(4.1)
the soft distributional Bellman operator Th is 7-contraction in for every 2 0. Thus, it hus unique solution to the fixed point equation Ç = T.C. which we denote by (5.T_
[Proof]
Next, we move to policy improvement. ERI. improving the action- function g involves policy evaluation with the policy 9,g. We leverage this insight to enable control. Definition 4.3. For any + > 0, the soft distributional optimality operator T is given by
(I:Daa = (J.. 26 a),a (br(z,a),) orroj [Tki o proj x )+ (5__ X p9. (6)
where Q: K(X x A, 3 (R) + M(X x A) is such that (2,) a) == Ez~cr. 12].
We proceed by establishing simple, but useful algebraic property.
Lemma 4.4. For any > 0, QT; = B,O.
[Proof]
Now we prove that iterates of JG converge, unlike iterates of J* [5].
Theorem 4.5. For any Č E (X x A, (R)) and lemperature > 0 define the iterates (5") )nEN given by gu+1 = J²5" for n0 -T Then, for ÇT = =
dr((", (T*) no n/p id (T) and di( (TTT (1 (1->)VT Cry" di (2 ST*),
where C, Cp.re < are constants depending on ||rllsup- (p, T, 7, ||rl)spp) respectively.
[Proof]
Theorem 4.5 leads to stability in entropy- reguoayrrguulrrized optimal return distribution estimation. In Figure 4.1, demonstrate the stability of T; and the instability of T* The iterates defined in Theorem 4.5 converge to soft return distributions, which are influenced by stepwise regularization penalties and correspond to policies that optimal in ERL. To estimate optimal return distributions, must consider vanishing temperature limits.
Independently and concurrently, similar results were established by [26] in the fied-tempeerature regime, but only with discrete action spaces and प being the uniform policy.
7

---

55.55
K 86
88
2
+800 5D
-06 KD
-2
-20 50
D
Return
Reture
Retum
Ratum
Raturn
Return
Reture
Retum
Return
Return
Figure 4.1: Evolution of the soft optimality iterates (I))* a) (bottom row) and the iterates the distributional optimality operator (I*)* a) (top row). Video entire iterate sequence available htttp://haaril //harwiltz. wwwplltz..ttttttt github. wttiaia////staallllleeeee reee-retuunddiiee witm--ddssiiiiiise......
4.2 Convergent Optimal Return Distribution Estimation in the Vanishing Temperature Limit
In this section, we instantiate the first methods for computing iterates that approximate reference- optimal return distribution functions in a stable manner.
Theorem 4.6. Suppose Assumption 3.4 holds. Let p.p/ E [1, 00) and w E P(X x A). For any 6 > O, there exists a > 0 for which day' u(TT="" "x²² ) S 8/2 and qx=" is e/2-reference-
optimal. In turn, an n4..5 (T, N exists for which
15" ,5 " s and 9,25" is e--reference-optimal Vn 2 Re,5
where gu+1 - and -. for any (x
[Proof]
Theorem 4.6 the first example of a convergent iterative scheme for approximating the return distribution of (reference-Joptimall policy. While it ensures convergence e-reference-optimal return distribution, it is still not possible a priori to characterize which return distribution will be learned. As - O, there may be stable trend in the return distribution that will be estimated because æ may not converge. To achieve (characterizable) convergence to a reference-optimal return distribution, turn back to the temperature decoupling gambit.
Theorem 4.7. Suppose Assumption 3.4 holds and is discrete. Let p. E (1) and w E (X. For any c, 5 > O and no E & A, 30 (R)), there exists > 0, decoupled or > and Nopt: Neval E such that
(S"toal) Apip' (omp:w(s" (TTeeal , (= )="" 5 and 9.2("eal is e-reference-optiaal
where gu+1_ = TT 4T.0 Enae, and gut = TT t1a 2", =0= =hoott
[Proof]
Theorem 4.7 outlines algorithm for estimating -. First, approximate ço.- via napt applications of T (control). Second, extract the mean: Ø = &h Finally, apply J neval times, with _ 9, (evaluation). I & 1, 72 for example, and Nopt Reval >> then the resulting return distribution is as desired. This ensures convergence, (efferencc)-opimmalli and interpretability of the final iterate.
4.3 Numerical Demonstration
Here we validate that ट:. approximates "a We consider the MDP given in Figure 4.2. Arrow colors correspond
0
different actions. Dashed lines represent transitions that occur with probability 1/2. In this MDP, different optimal policies have distinct return distribu- tions. From I1. the blue action yields return of 2y(1 - 7) while the green action achieves return 4-
==0
r=
3/2
5/2
-'errnoulll() /2). In Figures 4.3 and 4.4, we compute estimates Ç = द. and दي: 5n by (soft) distributional dy- namic programming using 64-bit precision and
Figure 4.2: An illustrative MDP.

---

32-bit precision respectively. 32-bit precision is the default in many scientific computing libraries, such as Jax [8]. Here ? 1/2. "r U(A) for all X, and -². We consider
7 E (10-(2m+1) m 0, 2,3, 4). Our simulation a practical implementation of Theorem 4.7.
First, we approximate napt 1000 iterative applications of our soft Bellman optimality operator at
Temperature
JE-01
1E-03
1E-05
1E-07
1E-09
O: 12
Probability
0.23
(1:0)
Return
Return
Return
Return
Coupled Mi
Decoupled 1i2s"
Ground Truth Mi
Figure 4.3: Estimates of return distributions via soft distributional dynamic programming - using the temperature-deccopling gambit and fj without- < 0. As the temperature vanishes, 17"h recovers the return distribution of T shown on the right.
(control). Then, extract G7, approximation of of. and construct two policies: the BG policy at and the BG policy at 71/2, both with potential Ø7. Next approximate Teeval 1000 iterative applications of our soft Bellman operator (policy evaluation) at temperature т with the first policy and temperature 11/2 with the second policy. These yield approximations of Çr.* and Ç", respectively, Figures 4.3 and 4.4 depict the policy-averaged return distributions V29 and f compared to the
Temperature
1E-01
1E-03
1E-05
1E-07
1E-09
0: 1%
Probability
0.0
Return
Return
Return
Return
Return
Return
Coupled T
Decoupled
Ground Truth 7/
Figure 4.4: Return distribution estimation with vanishing temperature using soft distributional dynamic programming, with 32-bit floating point precision.
R (Ct... =f The iterates are approximated via categorical representations baseline THe := proj #
[5, 34] supported on 121 uniformly-spacee atoms on [-2, 8), and MMD projections [43] with the energy distance kernel E3/2. In both figures, we see that the sequence of temperature-decom decoupled return distribution estimates approximate the return distribution associated ref, (right). Return distributions estimates of ÇTi* also converge to those of optimal policies, as predicted by Theorem 4.6, but find reach different return distributions in each While the temperature-decouplingg gambit is not impervious to precision issues, stabilizes BG policy estimation.
Related Work
Entropy regularization in RL was introduced by [48] for inverse RL, where it necessary to disambiguate optimal policies and identify the most likely reward function to explain demonstrated behavior. ERL with rf as the uniform policy-termed maximum entropy or MuxEnt RL, has been highly influential in deep reinforcement learning. Heuristically, MaxEnt RI encourages policies to be more uniform, thereby enhancing exploration, sample-effiiiency, behavioral diversity [29, 18, 17), well robustness [16. 2, 11. 12]. Heuristic approaches to adaptive temperature schemes in deep

---

MaxEnt RL have been effective in practice [19, 47]. Policy optimization in MaxEnt RL has been shown to be equivalent to a form of inference, conditional on notion of behavioral optimality, in certain graphical model [24, 14], and further characterizations of MaxEnt RI have lead to principled algorithms for efficient exploration [30, 39]. Alternative forms of regularized RI. objectives and optimizers have been proposed and analyzed [25, 31, 36, 4, 37, 15].
Policy optimization algorithms for entropy-regularizatto [28]-these methods apply to tabular MDPs and fixed nonzero temperature. [27] provide improved convergence rates for entropy-regularrize policy optimization. They also derive convergence results in the vanishing temperature limit, but only in the bandit setting. Exceptionally, [23], based on the work of [1], studies global convergence of policy gradient methods in continuous entropy- regularized MDPs, for fixed and vanishing temperature, with neural network policies via mean-field analysis. However, their analysis requires an extra regularization term to distribution over neurons, precluding convergence to an optimum of RL. To the best of our knowledge, our work is the first to introduce a convergent policy optimization scheme for general MDPs in the vanishing temperature limit.
in general are presented and analyzed by
Entropy regularization in DRL is largely unexplored. [22] experimented with an adaptation of Rainbow [21] MaxEntRL, but without analysis formalism. The concurrent work of [26] also introduced soft distributional Bellman operators, but did not study vanishing temperature limits, and did not establish convergence rates for iterates of JH even for fixed т. Moreover, the work of [26] established convergence only in the case of discrete A, and only for a uniform reference policy. Works have investigated the challenges of estimating optimal return distributions [5, 42], and more generally, the influence of particular tractable distribution representations on learning dynamics and fixed point accuracy [45, 46, 43, 3]. In [6], the authors show that distributional analogues of B produce iterates that converge when there is a unique (deterministic) optimal policy. The interplay between policy optimization stability and return distributions was studied in [33]. Their empirical study found that distributions of returns following stochastic policy gradient updates tend to have long left tails, and called for methods to guide policies into smoother regions ("quiet" neighborhoods) of the return landscape, the manifold of policy returns across parameters. This study focused primarily on deterministic policy gradient methods.
6 Discussion
In this work, we have investigated policy and return distribution convergence as the temperature vanishes in ERL. Our findings motivate iterative schemes for achieving convergence results beyond expected returns. However, they come with several limitations. In particular, while we have estab- lished policy convergence via the cemperature-decopplin consequence, our ability to derive approximation algorithms for <" with x ref, is limited; it a priori unclear which temperatures are required for ÇT.² to be an e-approximation of ( with x "ef in dæpp, and, therefore, to deploy for iterative applications of J or JØ with π 9 At the moment, however, our results ensure that by progressively annealing т, the scheme discussed in Theorem 4.7 will approach (x with exciting direction for future work. Another exciting direction for future work is to try to incorporate the pepperrauuredecouplin gambit into the many algorithms in ERL/RL.
gambit, this convergence qualitative. As
rf, Nevertheless, quantifying Theorem 10 is an
Acknowledgments and Disclosure of Funding
The authors wish to thank Wesley Chung, Mark Rowland, Jesse Farebrother, Arnav Kumar Jain, Siddarth Venkatraman, Athanasios Vasileiadis, Aditya Mahajan, and Doina Precup for helpful comments and discussions. HW was supported by the National Sciences and Engineering Research Council of Canada (NSERC) and the Fonds de Recherche du Québec. MGB supported by the Canada CIFAR AI Chair program and NSERC. This work was supported in part by DARPA HR0011-23-9-0050.
References
[1] A. Agazzi and Lu. Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime. arXiv preprint arXiv:2010.11858, 2020.
10

---

[2] Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans. Understanding the impact of entropy on policy optimization. In Interational Conference on Machine Learning (ICML), 2019.
[3] Alhosh, Wiltzer, and Meger. Tractable representations for convergent approximation of distributional HJB equations. Multidisciplinary Conference on Reinforcement Learning and Decision Making (RIDDM) 2025.
[4] K. Asadi and M. Littman. An alternative softmax operator for reinforcement learning. In Interational Conference on Machine Learning (ICML), 2017.
[5] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In Interational Conference on Machine Learning (ICML), 2017.
[6] M. G. Bellemare, W. Dabney, and M. Rowland. Distributional reinforcement learning. MIT Press, 2023.
[7] V. I. Bogachev and M. A. S. Ruas. Measure theory, volume 1. Springer, 2007.
[8] J. Bradbury, R. Frostig, P. Hawkins, J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[9] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 2023.
[10] R. Dadashi, A. A. Taiga, N. Le Roux, D. Schuurmans, and M. G. Bellemare. The value function polytope in reinforcement learning. In Interational Conference on Machine Learning (ICML), 2019.
[11] B. Eysenbach and S. Levine. If MaxEnt RL is the answer, what is the question? arXiv preprint 1rXiv=1910.01913, 2019.
[12] B. Eysenbach and S. Levine. Maximum entropy RL (provably) solves some robust RL problems.
In International Conference on Learning Representations (ICLR), 2021.
[13] E. A. Feinberg and A. Shwartz. Handbook of Markov decision processes: methods and applications, volume 40. Springer Science & Business Media, 2012.
[14] M. Fellows, A. Mahajan, T. G. J. Rudner, and S. Whiteson. VIREL: A variational inference framework for reinforcement learning. In Advances in Neural Information Processing Systems
(NeurlPS), 2019.
[15] R. Fox. Toward provably unbiased cemporal-differemee value estimation. In Optimization Foundations for Reinforcement Learning workshop (OPTRI. @ NeurlPS), 2019.
[16] R. Fox, A. Pakman, and N. Tishby. Taming the noise reinforcement learning via soft updates. In Conference on Uncertainty in Artificial Intelligence (UAI), 2015.
[17] D. Garg, J. Hejna, M. Geist, and S. Ermon. Extreme Q-learning: MaxEnt RL without entropy. In International Conference on Learning Representations (ICLR), 2023.
[18] Haarnoja, H. Tang, Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies. In Interational Conference on Machine Learning (ICML), 2017.
[19] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Interational Conference on Machine Learning (ICML), 2018.
[20] O. Hernández-Lerma and J. B. Lasserre. Discrete-time Markov control processes: basic optimality criteria, volume 30. Springer Science & Business Media, 2012.
[21] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
[22] D. Hu, P. Abbeel, and R. Fox. Count-based temperature scheduling for maximum entropy reinforcement learning. In Deep RL Workshop @ (NeurIPS), 2021.
[23] J. Leahy, B. Kerimkulov, D. Siska, and L.. Szpruch. Convergence of policy gradient for entropy regularized mdps with neural network approximation the mean-field regime. In Interational Conference Machine Learning (ICML), 2022.

---

[24] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: petp:000000000000 2018.
[25] M. L. Littman and C. Szepesvári. A generalized reinforeemennt-eamiing model: Convergence and upplications. In Interational Conference on Machine Learning (ICML). 1996.
[26] X. Ma, J. Chen, L.. Xia, J. Yang, Q. Zhao, and Z. Zhou. DSAC: Distributional soft actor-critic for risk-sensitive reinforcement learning. Journal of Artificial Intelligence Research, 83, 2025.
[27] J. Mei, C. Xiao, C. Szepesvari. and D. Schuurmans. On the global convergence rates of softmax policy gradient methods. In Interational Conference on Machine Learning (ICML), 2020.
[28] G. Neu, A. Jonsson, and V. Gómez. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv: 2017.
[29] B. D'Donoghue, R. Munos, K. Kavukcuoglu, and V. Mnih. Combining policy gradient and Q-learning. In International Conference on Learning Representations (ICIR), 2016.
[30] B. O' O'Donoghue, Osband, and C. lonescu. Making sense of reinforcement learning and probabilistic inference. In International Conference on Learning Representations (ICLR), 2020.
[31] J. Peters, K. Mulling, and Y. Altun. Relative entropy policy search. In AAAI Conference on Artificial Intelligence, 2010.
[32] M. I.. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[33] N. Rahn, P. D'Oro, H. Wiltzer, P.-L. Bacon, and M. Bellemare. Policy optimization in a noisy neighborhood: On return landscapes in continuous control. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[34] M. Rowland, M. Bellemare, W. Dabney, R. Munos, and W. Teh. An analysis of categorical distributional reinforcement learning. In Interational Conference on Machine Learning (ICML), 2018.
[35] M. Rowland, R. Dadashi, S. Kumar, R. Munos, M. G. Bellemare, and W. Dabney. Statistics and samples in distributional reinforcement learning. In Interational Conference on Machine Learning (ICML), 2019.
[36] Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In Interational Conference on Machine Learning (ICML), 2015.
[37] Z. Song, R. E. Parr, and L. Carin. Revisiting the softmax bellman operator: New benefits and new perspective. In Interational Conference on Machine Learning (ICML), 2019.
[38] U. Syed, M. Bowling, and R. E. Schapire. Apprenticeship learning using linear programming. In Interational Conference on Machine Learning (ICML), 2008.
[39] J. Tarbouriech, T. Lattimore, and B. O' O'Donoghue. Probabilistic inference in reinforcement learning done right. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[40] C. Villani. Optimal transport: old and new, volume 338. Springer, 2008
[41] H. S. Wilf. generatiigffinciionllgy CRC press, 2005.
[42] H. Wiltzer, M. G. Bellemare, D. Meger, P. Shafto, and Y. Jhaveri. Action gaps and advantages
in continuous-time distributional reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[43] H. Wiltzer, J. Farebrother, A. Gretton, and M. Rowland. Foundations of multivariate dis- tributional reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[44] H. Wiltzer, J. Farebrother, A. Gretton, Y. Tang, A. Barreto, W. Dabney, M. G. Bellemare, and M. Rowland. A distributional analogue the successor representation. In Interational Conference on Machine Learning (ICML), 2024.
[45] H. E. Wiltzer, D. Meger, and M. G. Bellemare. Distributional Hamilton-Jccbi Jacobi-Bellmaaa equations for continuous-time reinforcement learning. In Interational Conference on Machine Learning (ICML), 2022.
[46] R. Wu, M. Uehara, and W. Sun. Distributional offline policy evaluation with predictive error guarantecs. In Interational Conference on Machine Learning (ICML), 2023.
12

---

[47] Y. Xu, D. Hu, L. Liang, S. M. McAleer, P. Abbeel, and R. Fox. Target entropy annealing for discrete soft actor-critic. In Deep RL Workshop (NeurIPS), 2021.
[48] B. D. Ziebart, A. L.. Maas, A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, 2008.
13

---

A Entropy-Regularized RL in Continuous MDPs
Here we prove Theorem 2.3 as well as collection of supporting and related results that generalize well-known results in tabular MDPs.
We start with characterization the geometry of the space of occupancy measures. The following result extends the well-known counterpart in tabular MDPs [13, 38, 10] to continuous MDPs. While certain parts of this result are proved by [20], not all connections made, which we state here for the first time.
Theorem A.I. Let ((vo) {er 7 E K(X, {(A))} the space of all occupancy measures under the initial state distribulion vo E (X). Then 0 (vn) is equivalen to the space of all E P( A) that satisfy
(E) (11-))ooEE) + (aa(() VE C X Borel.
=
(A.1)
The space (vo) is convex, it is closed under setwise convergence.
Before proceeding with the proof of Theorem A.I, we recall the state occupancy measures v" given by
2 = (1 - Σ Av..
120
where (vz))21 122 is the sequence of laws generated by pa starting vo-
Proposition A.2. Let of and for L denote the laws generated by pa and pa starting at vo, and 1 = @ vg- Then io =T @ for all I 2 1. Hence, given and vo, the state marginal of the associated occupancy measure μe is the associated state occupancy measure v".
The proof of this proposition will use the following lemma.
Lemma A.3. Under the hypotheses of Proposition A.2, for every 2 1, the conditional probabilities of μe with respect to its state marginal are π.
Proof. It suffices to prove that the conditional probabilities of pf are n__ Let 1/1 denote the state marginal of pT. By definition,
I Mid' dei ((ledde
de' f duddddi /VV(())dPaalll duō (x)
Thus, for any E Mb(X x A), with (xxx)) = I so(a,, ,a) dow (a'), observe that / Vxx') (x))
/ pla', a²) (') du du'
/ [/ V(a') ((a'))Pa duō (2,
/ [/ ple' (la',''ddaolll ap dPaala') dx)(
/ [/ gla', a') a aP dpū(z,
/ plz,a) (z.
So the conditional probabilities of ut with respect to v are na. desired.
14

---

Proof of Proposition A.2. By Lemma A.3, it suffices to show that the state marginal of ui is VT. This holds:
[/ (a d deo(a)
/ [ M(2') deaaa(x)) drrs(a) dvo(z)
/ [/ Max' d dPaa(a") a(T @ vo)(2,))
/ [/ Max dP (x,
By this computation and Lemma A.3 applied successively to each pair (1T+1 M4 for every t 2 1, we deduce that nr7 тT_ p²", for all t 2 Finally, by the linearity of the integral, we conclude. Indeed,
1 =
7)
Σ Ch' = (1-1) -0--)) 7) Σ 120
(am 0 = 2 (1-1) 7) Σ tv = R0.
t20
120
Proof of Theorem A.I. We prove this theorem in three steps.
Step 1: (vn) $ (vp). First, recall that proj nr = ux for any policy n, by Proposition A.2. Thus, we have that for any æ and any Borel E C
proj (E) nd (E) (1 7)v (E) + + -11-)) 7) Σ Nvv (E)
-7) Σn 7 / (a.((Eddu((s,)))
(1 ()) + +7(1-7)
120
= (1 - (E) + + / P (E) (x, a).
This shows that (po) C (vo). It remains to show that & (vo) C (vo). Let μ E (vo). whtth is,
and let n denote its conditional action probabilities with respect to its state marginal D -hat = Th @ ve Moreover, let Φ be any bounded measurable function. By the definition of P, we note that (A.1) can be written
J (o((z) duy du (zo) (1 (1 f (o0( dvu( x f ( to (m1) dv (xo).
Defining (x)) = Ix (0((0)))))))) d (z'), the rightmost term Ix di (20) du"((z)) can be again expanded via (A. 1),
[ (u(()) du = (1 () / (60((0)) + ( dzo
+ +2² /// 01 (1(1)) dp (01) d=*()o)
Continuing, we define (x)( = Ix on (n(xx) dx')d""" dp="" (x'), which is bounded and measurable for each E N, yielding
f (o((0) du = =(1 71) / Σ
7 dx (20) den()
k=
In /
+
On (2ro) ap (x1) dx)(
11.
15

---

By the definition of on. we have that
In -1->) / )o(()) duo(zo) (1 (1- - 7)7 // Φz( dP (oi) doo(zv) +...
= =11 / ( duo(vo) +(1 (1 >)> / 60(20) dui (ro) +...
- - o0( - 7)
n
Σ de (zo).
Moreover, by the boundedness of On. we deduce that II, - O. Substituting, we have
/ (bo(z)) (xo))
lim Ia + lim II, n
n
= f (o((0) lim (1 7) Σ 7h do (so)
k=0
/ @o(zo)d (zo).
Since do was an arbitrary bounded and measurable function, it follows that Dn = va* Thus, J = т @ yd p -the occupancy measure for the policy n². Consequently, any μ E & (vb) is member of (vo).
Step (vo) is convex. The convexity of ((v)) follows immediately from the structure of & (vo). Consider any Po, μn E (vv) any E [0, 1J. and define au4 + (1 - a)p1. For any Borel E X, have that
proj (E) = cprroj x (E( + (1 - a)proj
Since NA, 111 F (uo), they solve (A.1). 1), expand the RHS,
x Ma (E) a(1 - ()) + on / (e.(()ddu((z proj
+ (1 a1 - 7))o(() + (1) (11-a)) / (e
(1 -))))(E) + 7 /
(1 - ())(() + 7 (---oolt+++//eeeeineeeee (p)
So Na E (vo) = (vo), as desired.
Step 3: (v1) is closed under setwise convergence. Let (μu)KeN C ( be sequence that converges setwise μ. Since a) + Pz,a(E) is bounded and measurable for any Borel X,
I
+ / (Psola))))ppp((
(A.2)
Likewise,
proj x (E) = ( A) m(E A) proj
(A.3)
as Pk + a setwise. Consequently, we have that
projx1(E) lim proj*M(E) k
lim x (1 - 7))o() + (
(aa((Eddp((e.o)
(1 (1-7))((E)) + (
where the first equality follows from (A.3), the second follows as Me (Vo), and the final equality follows from (A.2). Thus, see that 9 (vo) (vo)).
D
16

---

Now we prove Lemma 2.1.
Lemma 2.1. The functional R (XX A) E is strictly convex.
[Source]
Proof. Observe that
R(p) KL R_ @ v²).
We prove this in steps. First, for every Borel / X + [0, o0), have that
f
d dr (a) d( @ (z) a) = [// ((x a)
da
dr (a) dum(x)
[xx a)
d
/ [/ [(x, a) drt((a)) do)(())
/ (a, d( @ (T)
Hence, μr xtth ② DM < nt & vn if x < o for vn -lmoost every and
du
dr (a).
d( @ wu) (I,
dl
Second, = rz 8 DM < _e @ t implies that xh < T for vl-almost every x. Indeed, suppose that set X exists such that y (S) > 0 and for each x S, have that
(B=) > 0 but π (B,) = 0.
Let
E U x B #
Then,
(n 3 4d (E) / ref (B..) dx)))) =0 and (* 60 (E) = / T/ (Bx) (xr) >
This is contradiction. And so,
R(p)) = /
[ log
do drreet (a) (a) dx)())
/ / log
cl(ree @ vn) (c, a) drt((a) dv)
du
/ log
du
d( @ wu) (1, du(or,))
KT 7 3 vM),
as desired.
Now recall that
KL(tuz + (1 - () 11 + (1 - ()16) S (KL(m) (1 + (1 ()KL(HU |l (1).
Moreover, note that
-11+(1-1)pa tuml (1 t)umo,
In turn,
R(ti + (1 t)4m)
KL(t + (1 - t)Ho || ree 2 (lux+(1-1100)
KT((M + (1 - t))1o || ,e @ (tDM) + (1 ())(Ho)
KL(ttn + +))o || 1 wHI) (1 ( @ vHO) S [KL(M) || @ vn) + (1 t)KL ref vHo
ER(p1) + (1 - -))))()))) Thus, R is convex. In particular, 3. is strictly convex as KL is strictly convex in its first argument.
17

---

With Theorem A. and Lemma 2.1 in hand, we use the direct method from the Calculus of Variations to prove the vell-posedness of 7-ERL, in the tabular setting.
Remark A.4. The space Mx x A) endowed with the supnorm is a Banach space. Note that M,(X x A)* = ba(X x A), where bo(X x A) denotes the set of finitely additive set functions on (X x A) equipped with the total variation norm. Note that the set of probability measures on X x A is a subset of the closed unit ball in ba(X x A), which is weak compact, by Banach-Alaoglu, The duality pairing for any μ E (X x A) and for any &p E M2 x A) is given by integration: {n, p) := f (p dp. In other words, weak convergence is setwise convergence when P (X x A) is considered as subset of the dual of (a x A).
Theorem A.5. Suppose that r E (x x A), X x A is finite, and let vo E (X). A μ E (vn) that achieves the supremum in (2.2) exists. Moreover, other occupancy measure does so.
Proof. Let the supremum (2.2) be denoted by 07 and (uR)REN C (VV)) be such that
1
Jr h < <+((p)) S 8.
In other words, let (μę)aEN C (po) be a maximizing sequence. By Remark A.4, owing to the fact that Mx(X A) separable (since X is finite), let (1ss )eEN be a weakly* convergent subsequence, with weak* limit Poo In particular, MMk Mox setwise. As ((vo) is closed under setwise convergence, by Theorem A.I, we have that Poo E (vo). Furthermore, n- @ v 7 rto & pd* setwise as well. As setwise convergence implies weak convergence and as the KL 1() is lower-semiccontinuus in the pair (H,p²) p²) in the weak topology, find that
0
lim sup
limsup 80000
(ddue
T lim inf KL || T! & w
1+000
lim 80000
rddue TKL(Ux |l nt @ uHoo
rdduc - -KTo(Moo || m s vnte
(moo)
The penultimate equality uses that r is bounded. Thus, Ir (m60) - 0z The previous argument applies to any sub-sequential weak* limit of our maximizing sequence. But as R is strictly convex, by Lemma 2.1, and (24) is convex, by Theorem A.I, only one such limit exists.
We now move to prove Theorem 2.3. To do so, we state and prove some helpful results. We begin with policy evaluation.
For any x E <(X, (A)), define of X x A + R U {-00} by
(x. 0) = E
r (Xx A7)
(1(77, A7) - TKI(TTp |1 mx
(Xx. (8) = (2, a))
By the tower property of condition expectation, we have that
gf [z, a) = r(x, + 7 g. (1', a') - TKL(TT x di (x', a').
It is convenient to be able to evaluate policy (find o.)) in an iterative fashion. This can be done via the soft Bellman operator B MX x A) + M(X x A) defined by
(B,g)x. a) == r(x, a) + / aldd,a') - 7KL( ||
1 dp= n.a (x', aa),
but only on a restricted collection of policies.
Lemma A.6. Ifr E M,(X x A), 7 11, and w is such that (4.1) holds with p = 1, then the By is contractive on Mo (X A) endowed with the supnorm. Its unique fixed point is q;-
18

---

Proof. Observe that
[B S ||rl|sup + tllglbbup + sus | [0__[____<"__ (l < co.
5.0
by (4.1), and
[B.q B" Isuu S <>11V, - V S 57l14 o d'llupp
Next, we proceed with policy improvement.
Lemma A.7. Ifr MM(X x A) and y < 1, then the soft Bellman optimality operator is a contraction on M&(X x A) endowed with the supremum norm. Thus, it has a unique fixed point oz.
Proof. Observe that
[Bggllaup I|Ilaup + lllp < 00
and
|B7q Blaullupp <>IV V-g'llaup S 51l1g q llaup.
Lemma A.8. The following equality holds true: of g. Zr.
Proof.
Observe that
B o7 = +7 / &. ---KKI((Sgg). -|= d.S.. [
=r( + / V. dPz.a
=BiT
In words, of is fixed point of the soft Bellman (policy evaluation) operator with - As 9, is Boltzmann-Gibbs policy with a bounded potential, by Lemma A.6 and the preceding note, this operator contraction with unique fixed point. Hence,
9
the unique fixed point of B with π 9- as desired.
Lemma A., For every E K(X, (A)), have that
9 2 o
Proof. First, we prove that
B 2 प..
(A.4)
By definition and the Donsker-Varadhan variational principle,
(x, = r((, + [I of a'd -KT,(T' || ( dP dPz
<((x,a) +7 / (V,07) = (Btg7))
Now conclude. Let go max (4F, 0}. By (A.4) and since B; is monotone operator,
of B < B 3. B; S .... lim 9". of. n=000
where the final equality holds by Lemma A.7, noting that loo < 00. Finally, prove Theorem 2.3.
19

---

Theorem 2.3. Let > O. The policy ,² := 9 is optimal, and uniquely so. More precisely. for all Vo 1/ E (X), we have that arg max O7 = ,. arg maxo(v() d7. [Source]
Proof.
For any E K(X, (())), let
(x) :0| / (z) -KKL(T 1
Note that
a -1 /
duo
u E (vr). Hence, suffices to show that
9. 0y 2 sup v
(A.S)
11
Observe, by Lemma A.8, o (x)
(v)
Thus, by the Donsker- Varaahan variational principle,
- (x) (x) (VV+))()) d 7KL(TT (T(e
2 (V-+)((z) - (2)
Finally, by Lemma A.9, we have that
V - V, V, 2 0,
for all ,, as desired.
To conclude this section, prove Theorem 2.5.
Theorem 2.5. Suppose that Mr(X A) and that X x A is finite. For every T 0, let μt be the maximizer dr over (vv))- If Assumption 2.4 holds, the sequence (Ht) has unique setwise limit tends zero. This limit pg is the minimizer R over arg supo(va) Do. [Source]
Proof. Let n² E (argsup dol n (R < oc). Then,
d (14 30(4)) I +((((())) - R(u+)) < oc.
In turn, for all 7 > O, we deduce that that. R(x;) S R(1()).
Now let Na be any limit of any setwise convergent subsequence of (u;),> (cf. the proof of Theorem A.5 and Remark A.4). As R is weakly lower semi-continuous we find that
R(po) I lim R(H)) I R(x*).
Moreover, since R(u*) < oc, by Lemma B.2, and as r E M&(X x A), we deduce that
lim (RR("") R(x))) ( - and Jo(mo) (o(")) 7-0
Therefore, Ml E argsupo Jo and minimizes R over arg sugsup Jo-
Since is strictly convex, by Lemma 2.1, and the set arg sug To convex, R has at most one minimizer among this set. In turn, only one such limit μa exists, call /i-- Hence, no) - 1 setwise, as desired.
D
20

---

B Proofs for Section 3
Before proving the results from Section we introduce some helpful notation. For any g: XxA - R we define
(M) = f q(x, d(9,9)..
Additionally, we will define M, - (X A) R U {00} according to
(4) sup ess sup ((z,,)) V)g(z)}
We start by proving that B is contractive on MX x A). Lemma 3.1. Let r E (X A), 1, and Bi M(X x A) - M(X A) be defined by
(B a) = r(z, a) + ess q(x', d'aa,,
Then Bid
contraction on Mx(X A). Thus, has a unique fixed point Geet-
[Source]
Proof. First, observe that
B Birt sup & |||lssup + Illalsup--
Second,
[ Bi lleup
S y ssup ess sup and q(x', ((x',)) ess sup d'(x')
x²'
S 7 sup ess aup |g(2", -) -d' (z',
S >111 o' ssp-
The lemma follows by the Banach fixed point theorem.
Next we prove value function convergence.
Theorem 3.2. We have that q; Viet monotonically as O.
[Source]
Proof. Since o; is bounded (as the fixed point of a contractive operator on Ms(X (X x A), there exists 4o X x A - IR such that x go monotonically and pointwise as 0, direct consequence of Lemma B. Therefore, by the monotone convergence theorem,
lim V, (I:) lim log exp(g (x, ())l)))))) o
log || expe(p (xc, (1
a-0
Consequently,
lim lim V- lim log exp((g((,, ()ll)) 7-00-0
log exp((0 :) (=())
sup.. (z.
The second step holds since for any f E _ ||I converges up to || as - co. So, since the sequence (z) is monotone and bounded, its limit exists, and coincides with that computed above:
lim V (x) ess sup (x.
Since v is the unique fixed point of B;, by the monotone convergence theorem, have (x a)
lim (x, a)
lim (B; (z, a) 7-0
= +~ /
lim V V.g- dP (x')
= = +7
esssup عr(2 .) d (Pr,a(x'))
so that go is a fixed point of B Since the B has fixed point giet it follows that go g'd-
21

---

Now we prove our core estimate.
Theorem 3.6. Let 4 of E M(X x A). For any T > 0 and any E X,
(1(5,9). - (9,T')=lITV
< <min { (VT=)l((z,)) ( sinh (4T "10((,,)))))) - (a, (
In particular,
2e - (1q(zzz, (2.)))
1(9,9). (9,9') )2llv S
(ll((,,)) - d( (((x,))l2==(71) < </
[Source]
Proof. Let 9-g. == 9-9/. . By Lemma B.6, we have
|| - - -V - (2, (14t)
Moreover, by Lemma B.9,
|| - l/llrV I sinh (1q(x,) g (z,, (x77))
This concludes the proof first claim. Next, we recall that
2 (2h + 1)!
sinh(y)
which is convergent for any E C. Therefore, for y E (0, 1), we have
y² y
sinh(y) S y +
+
31 5!
5
y
+ e
3
y
2
So, when (1g(2,)) (1. <
T/2, follows that
3
| -
<
(o--11o(z,,) of (1))))))))
2
2
2e
T
||q(z, :) - g' (z, (1100))))))
4
Finally, we prove policy and return distribution convergence.
Theorem 3.9. Under Assumption 3.4, o = (T) is such that lim, O/T = 0, then T. as O. for all X, in TV if A is discrete and weakly if A is continuous.
+ T
[Source]
Proof. Recall π T.0 = g. By Theorem 3.6 and Lemma B.10,
lim sup [1(9-g)) - (9-4ir)zlliv S lim log Pref +++
(B.1)
+-U
Consequently, T.. - -.. if and only if (9,9te)) + ref. and in whatever sense the later convergence occurs. In particular, if A continuous, this the weak sense. While, if discrete, this in total variation.
22

---

Theorem 3.10. Suppose A is discrete and Assumption 3.4 holds. Ifa = (T) is such that O/T 0 as
+ O, then, for any p.p' E [1, 20) and w E (X x A), as + 0, the return distribution functions çTh of the temperature-decouppeer policies T- satisfy dya a ( 5 + O.
[Source]
Proof.
By the distributional Bellman equation [5], we have that
dp(St,av S2.1)
s ffe dp ((Dr(e.)) proj")es (S @ "... (Dr(z,)), o proj" (S22 )m dPaa(x))
f dp (lbur
proj oroof (S @ )) (bo,, o proj R (C7, @ m dP dPza(r')
=7 / T PPa.a
where
(x') dp (IDus o proff) (S & @ ") (bo o ) (C2 2 o
We now derive a bound on a Starting with triangle inequality,
(x,) s do ((oa o oproj* # (Sz, @ o "), (bo.1 proj )) (C @ )))
hp (Dba,) o orooj ) (CP @ - (Do,1 o ) ( @ 3T7))
/ dollsora's ) (a')
+ ((bo. ( oproj") ) (S 8 *) (bo.1 o projf)*((5 8 3T4))
(6)
$ / aa dm
+2 ( lal af
dl
refi
nT
Toa Th (ad))
1/8
( / dolGs, or a do (a')
2 p-1
||r| lup
refi
TT
1.0
1/p 1|
where (a) follows by the convexity of the Wasserstein metrics [40, 6]. (b) applies [40, Theorem 6.15), and (c) leverages that the support 3 lives in ball of radius ||r]|sup/( 7), for any and (x', X A.
So, thus far, we have shown that
dp ST, Si ST
f af
(x' + Oy / | -
7. U dP (x').
By Theorem 3.9, the total variation term tends to zero as tends to zero. Thus, defining (xx' a') = this implies that
lim sup, (Sg a c..
dx',a') s f uy, b) din (v, b),
In turn, sup < sup,, implying that . = O. Therefore, dp(Ct.as (7.a) - 0 pointwise over x A, so by the dominated convergence theorem, (S", - for any p' E (c) and (X x A).
B.1 Supplemental Lemmas for Section 3
The following lemma translates immediately from the corresponding result in tabular MDPs; we prove it here for completeness.
23

---

Lemma B.1. IfT S o, then oz S gz.
Proof. By the monotonicity of B+. 3 --r+
/ V V. dP_,. sr Sr / V-9 dP = =B I ... $ 97
(cf. the proof of Lemma A.9).
Lemma B.2. Let (F) and suppose as 0. Then
rKL((S,9)) || (| 140 0.
Proof. Expanding the KL, we have
rKL((9, 11 = / (1)(z,-)
- (1)) d(),
I css sussup and 9 (x. (V,9%)
I (x) - (V,9%) (z))
where the final inequality holds by Lemma B.1. Since = (T)) S we have
V.9. (z) log exp(1 (z, +)) (xgg)))) 2 log exp(a (x,-)) () V-- where the inequality again is due to Lemma B.1. Consequently, we have
lim sup ( < o (x)
lim inf V, g (x) vee =0,
T-0
7-0
where the penultimate step is due the fact that V.4. + ofet monotonically, as shown in the proof of Theorem 3.2.
Lemma B.3. For every q Mr(X A), with the notation above, Mr(g) + 0 0. Assumption 3.4 is satisfied, then for any a > O,
M+ S -7 log Pref-
Proof. First, we observe that lim V, V, lim log || exp(a(x, ()((),))))l ()ll = log exp(((xx-))]l = css sup ( +).
7-+0
-
This is a monotone limit in r, as it known that for any f E T [| converges up to || 1== as
p 00. Thus, we that
Mf (g) = sup
css supar ((2) :) - -V- (V-9(a)) - sup(esssup,,.e q(z, ess sup g(x, :)) =
as claimed. Now, under Assumption 3.4, we have
Mr(++) sup(ess sup sup and C0 (Ir, V2 (x))
ess ess and (z, log / exp((= (x, (r, -) dm
sup
sup
- log log
exp(( (1 (z,
ess (x, dor ref
Let B, {a E A
2o (z. a)
css sup, oup o (z. )) Then,
Mr Mx(1)) = sup
- loo
exp(T (z, ess sup oo (x, -)) dr
+ /
exp(r- (1) oss sus Loe (x,, -) dm ]
AlB.
Tref (Ba) + / exp(r- (2)(2)) esssup (z, -) do
sup
-og
AAB.
sup -7 log (B...
x
< r log Pref.
where the final inequality invokes Assumption 3.4.
24

---

Lemma B.4. For all r > 0 and any E (X A).
Bir 3,
where B denotes the Bellman optimality operator (cf. Lemma 3.1).
Proof. A direct calculation gives
(ʃ exp((= ((2, a)) dx (a)
V. (z((x) = 7 log
/ exp(( css sup q(2, .)) do
S log
ess sup ((x,)).
Therefore, it immediate follows that
( = (r(( +7 / ess (()',)) d
2 r(z, a) +r /
(z,))
The follow proof is essentially the performance difference bound in [37].
Lemma B.S. For all n 2 and any т > O,
(B )0
(9)).
Rm
If, additionally, Assumption 3.4 satisfied, then
(B,))" 4 - E S - log Pref
Proof. We begin with the first statement. Recall that of is the fixed point of BF, so that & B We will proceed by induction on n. For n = 1, we observe that
(B. (Bar))))), a) & (z. a) (Brr (B, a)
=7 / csssupp, (x',, V.oi(aa)) aPraalx) S M{,),
recalling the notation established above. This proves the base case. Now, assume the statement holds for all < We have
(Bie) +++ 4; - 4; = (Bier) ynt
B+n+1 4f
n
S Bid
B²"h a +
7 *M)((i))
-B² B,7+1 0-
A=1
Bierg"
A+1 M, (g.) -B,
n+d
<M,(4
Σ M,,()")
2=2
n+1
Mc))")
Am
25

---

where the first inequality invokes the induction hypothesis, and the second inequality is due to the base case. Thus, we have shown that the claimed statement holds for any n E N.
When Assumption 3.4 satisfied, by Lemma B.3, we have M, Mr(a;) S - log Pref, and the second statement follows.
Lemma B.6. Let q E (X A). Then for any > 0 and any E X. (((,,)) - (1s.))mlmm S V r [1((,,) - (((x, (1 200))
Proof. Let π 9 and let = 9, . By Pinsker's inequality, we have
| - Fll & V 2 KL |l my)
Since of E - (X x A), Tay m are mutually absolutely continuous. Expanding the KI. divergence, have
(T || T²2)= =
log Tx dm
A
log The (a)))p(+ V,((z))) d
(x()))xx(= (=)(
T (g(0, a) - V, o (z, 0) + V. (x)) (a)
$ 5 11g(2,,) :) - o/ (z, ()1+))l1 (--)) - 11 | (()) - V, V,
2r 11 n (z, (
where the last inequality holds since V- I-Lipschitz, shown in the proof of Lemma B.4. Substituting back into Pinsker's inequality, we have
TT TTV S - lll((a) - a' (1g
as claimed.
Lemma B.7. Let प. x' (YY) for some measurable space Y be mutually absolutely continuous. Then
du ess inf
dı dm'
|| - l/v S ess dx'
Proof. Define h = dr d= and write M == sup, h, ess inf h. Note that
/ (dT dz') / (h- (h 1)d71)d'-- (h-1)d7' + -yLE (h 1)dx',
for any measurable E Consequently, we have
[[ (h 1), L (1-h) hi- hddd'.
Now, we derive the following upper bounds,
(E) zE) (h 1)dm's S (M- 1))'(E)
JE
π -E)- = JY (1- h)) < (1-m))) (1- E).
Multiplying these inequalities by + E) and n' EE) respectively, and adding the results, we have
(m(E) (E()) \ E) + > (E)) < ((M (1) E) :. (()(E) - n' (E) < (M- - (M-
26

---

In fact, the same bound can be achieved for r'EE T(E); to see this, note that
m1 n' (E) -(E)= = [ - h)dm's = (1 - m)m (()) (E)
. <'(E) ((E) 1 JyyE (h 1)dT's S (M (M-1))((E)) (E),
so by the same procedure as above, π x'(E) π (E) 2 (M - m)T (E) E). Therefore, we have shown that
(EE n (E)) - (M m) (Y E)
for any measurable E C Y. Since r'(E)T (Y E) is maximized at N r'(E) = T' TY E) - 1/2, we have
|| x' l/ = sup (<() x' x(E)) S (M-
claimed.
Lemma B.8. Let u, w. E (Y) for some measurable space Y, and let À be measure on Y. Define
n², (YY absolutely continuous with respect to X such that dx dx for {u,w). Then
1
||x" - π 1rv sinh (21lu w wl))=())).
Proof. Firstly, since u, L (Y), it follows that n² nh are mutually absolutely continuous. Now, define h dn" daw with M = sup, h and ess inf h. Note that
dx e (( d)(z),
(x) e e-w(w)
d)(z),
Z
Z
where Zus Zw normalizing constants, Defining := w, have
Zu -f()).
h(x)
Zv
Additionally, we have
Zu
fye e m(x) dl(x)
1
Z.
(ee d)(x) E [c-
Consequently, it holds that ess inf h 2 essinff f-ess I and ess sus I cessuup, f-ess inf So. by the definition of f. we have m 2 e 21| and M s (ll Then, invoking Lemma B.7, we have
1 || n 1IIVV S 1 (62llu- - 2//u w0 = sinh wl)z~(a))-
Lemma B.9. Let q E 1, A). Then for any and any E X,
(2, (1,-))lx~((r)) [(994)z (9,4)) (9,4))ullrv s 2 sinh (27-111q(x,))) [1((x1
Proof. Note that, for any E M& x A), we have
d(S,g), exp(7- 1g0 )ddm
So, invoking Lemma B.8 with u = - - -), - (x, and x we have
1
[(()-g). (9.g')xllTV s sinh (27-1116 و1g d'll=~(z)))
27

---

Lemma B.10. For every т >> 0, recalling the notation above,
S gint
9 - 1 1-9 M- M-(-.).
If Assumptiio 3.4 is satisfied, then and o converges uniformly up to qiet-
Mr - log Pref
Proof. By Lemma B.4, we have that for any g E (X x A),
Geef O lim (B (Beer)" )nq q- lim B, q 2
2
n
Then, by Lemma B.5, we have
Tref of
lim (( ret 10))""/ - B+
lim Mr Mr(9r) 22000
M, (9:),
proving the first claim. When Assumption 3.4 is satisfied, we have Mf I -Tlooggee Pref by Lemma B.3, so that (g²) converges down to 0, and consequently q, converges up to q"
C Proofs from Section
Theorem 4.2. Ifr E M&(X x A), 7 < I, and π E K(X, (A)) is such that sup [Tr][[]]]LL(P..
00,
(4.1)
the soft distributional Bellman operator J. is a 7-contraction in d, for every T 2 Thus, it has unique solution the fixed point equation J7C. which we denote by ÇT,T.
[Source]
(X For Proof. To begin, let us show that J. maps elements of K2 x²(X A ((R)) to (O) any (x A, S (R)). observe that
f |e| d(JF() (JFF)).(())
1/P
sup x,a
(/
|| m + y2l" dhx (z) dār (e 1/v
sup
sup / = (Gr
1/8
< ||r||sup + sup || +
2,0
< 000
by assumption, as desired.
Next, by the convexity of the Wasserstein metric [6, 40], we have dp (.TT7T) 2z... (IFC)
$ / dp ((D(E,.)) ll (x3)),)) ) Sat, (Dr(z.a) (129))))) (or, aiz de (o',
I 7 / dp (s att) dea d (z',a')
S y sup do ae)
= =
28

---

where the second inequality holds since the common transformation b (r,,a)) ( mg) is affine. As consequence, we have that
2 (T:C, S rd,(5.2),
which validates that J7 is 7-contraction in dp. Consequently, since (K"(X x A, (RR),, dp) is complete and separable [6], it follows that TH has a unique fixed point. That (. coincides with this fixed point follows precisely by [6, Proposition 4.9].
Lemma 4.4. For any T > O, QT; = 272.
[Source]
Proof. For any EX A, 9 (R)), we have
(z, // (r(z, (TKL(19 x19 + 72) dh (z) (2) d (x',a')
/ (((er.a) TT,L a5) || +> 2 cl ar (2) dis. (x',a)) (x',a')
= / (r(z,a)- (() (| + (22) (
Defining q = QČ, this is equivalent
(o, = / (r(s;)) (TTL(( (1 + rg(al, (')) (
Moreover, note that
(L((S,9) || ) -f [log
d(S,g)m dor
d(S-9))
=T- / - ((.
= c (1000 -V,00(
Substituting, we have shown that
(x, / (r(0,0) -7
ap.."" +-V7(+0) + / (r(z,a) + (x)))))) (x',aa)
t1(1,a) + aFf
a)
= B))((
Theorem 4.5. For any Č E K" (X A, (R)) and temperature given by gutl_ = J² for n" Tzc. Then, for Ş == --
define the iterates ((") n")
d,(C", (T)) < un p/r dd and A,(5",TT)) (1 (7 Cnn" (T ((
where C, C,Cpsy < 00 are constants depending on ||llaup (p, T,,,7, lIrl|suu) respectively. [Source]
29

---

Proof. We begin by defining some helper notation. For any č E K"(X x A, (R)), we define
ş E (X, ( x A)) where
< : (b_ II aghha o "roj"*) (Gc. (9,2()x)
(C.1)
In turn,
(3 (br(a,)), o (proj (5 Ø P..)).
(C.2)
Next, we define the following helpers,
オ = S,0 == :t=
By [40, Theorem 4.8], we have that for any (I, EXx A,
dp . ( ddP.a())
CC.3
Invoking the triangle inequality together with the expansion of the terms by definition, have that for any x E X,
dp
= dp ((proj- -KL(TT [| S- & m²), (proj* 7KL( || (
< <dp ((proj rKL(TT [| (22__ @ T (proj² R - [KL(= || (( (C @ TT))
+ dpp ((proj* - ( क 1) CC, & कh (proj - (KL(T (| 4 C @ m)*)))
Since the measures being compared in I, are both translated by the same pushforward map, another application of [40, Theorem 4.8] yields the following inequality:
In S dp (a) S 2,(5"
Next, we bound HI, Let ((p,, be the set of couplings between measures P1, P2. Then
(f (florrrroo [.rr), (2) - b b ( ( dri
1/2
IIn S
inf 1
(f1= 21P dr)
1/8
inf (E: (00::::000.. (7:1011")))
+ KL|KL(TT ll (
KL(TT [[)))]]
(b) 2 C - of 1/pp
for some constant C depending on T, y. ||r]]sup where (a) applies Minkowski's inequality, noting that the KL terms are independent of k. and (b) invokes Lemma C.5 and Lemma C.6. Indeed, for n large enough, Lemmas C.5 and C.6 assert that CST-1 for fixed p. and more generally that -1/2 for any (and fixed p). Substituting back into (C.3), we see that
1/p. d,(5*+1, (TT) S " 1t//// 9f sup
Let O,, : d,((", We have shown that 0a+1 I ?a, + C'rltn/p, where C' = - q," sup is constant depending on p and r. We will apply techniques of generatingfunctioollgg) [41] to bound this sequence. We define A : R + I2 as formal power series given by
-
A(v) = Σ a,
n=0
30

---

and we will pick off the coefficients on from the power series representation of A. Our recurrence above, upon multiplying through by y" and summing over yields
an+ 5
Onl" + C'yy
prr y
n=0
n=
n=
A(y) au S {)(()) + C²yy
- 1
C'
: A.y - 71 7V
(1-1\py)(1-9)
where = dp(( (.*)). Now, the formal power series expansion gives
xc n=
a07" C"Ty y ""/p"""") y" p
A(v) 1
Em=1
x [o0 + C'n""ll
== I.
Combining, have
an S (1+ (1+)p(5) "(""))""m"""""
where C" = C" when p 1, and = (7 -7) otherwise-in any case, C" is a constant depending only on p. T, " and the proof is complete.
Theorem 4.6. Suppose Assumption 3.4 holds. Let p.p E [1, and w P x A). For any 8 > O, there exists > 0 for which dew u((T." t S5""") s 8/2 and qx is c/2-reference- optimal. In turn, an ne,6 = (T,
exists for which
:5 ,5t ) 2 and 9,2 is c-reference-optimal Vn 2 ne,5
where gu+ll = T2L" and ę = J for any ( Č ( A, (R)).
[Source]
Proof. By Lemma B.10 and under Assumption 3.4,
7 ylog Pref E 2
l1 dier Isup
1-7
choosing T S Te = <(1-1) P.r
Hence, by Lemmas 4.4 and A.7
-105 Teer Iuuup I "0 4 Isup + 11 - - loup S 195" lsap +
S 2
which holds when n. 2 (log 7) - 211020-0rdllss
Next, we will show that the soft return distribution estimates will approximate "x" For notational simplicity, define X₂ == X²= and A AF" fort N. Recall that
",. law (((,a)+ + Σ 7' (r(X,, A.) Ab) KKL (| (())
Xo x. Ao
Moreover, we define
Frix"r Snja
= (-TKL[TT"* [ + id)
so that
Sn.a
दाa law
(r(X,))))))))))))))))()))))))))) -KL(KXK - ({))))
Xo x,, Au =
120
Now, by the triangle inequality, we have
/
"T " < 20'-1
dy' (CT=""
Fram
dy (T=:"
xT du(z,,). (C.4)
Sa.a
22.0
(a,0)
(1,a)))
31

---

We proceed by analying I, By coupling states and actions, we immediately have
[2, a) S (TKL( | )
and so, since x 9 by virtue of Lemma B.2 we have
lim [2 a) = O.
Next, we bound II, Denote by Fn, X x A + R the reward function defined by
T (2. a) r(z, rKL || mr
The work of [44] shows that, for any policy x, there is unique 7ad K(XXA, 9 x A))) for
which (μ + (1 (1-1) 2) (ur)(z, a))# Th SH where ŞTt"" denotes the return distribution function
associated to the policy π for the reward function r. Noting that ÇT, = =5""" Fnrr, we have
[T, a) = di.
(ut 1 1-g (1r)) Tz
T
1-
< 1-7 VV (imrr(l'a)) - r((a,( ,d)p data', dd) atz (w)
1/1p
1- W (KL(T ll (
du(x', d') d (m)
1110
where the penultimate step is simply coupling argument (coupling the samples of "F" )) Once again, since lim sup,-.o [ m O, and [ o is bounded by Lemma B.2, the dominated convergence theorem asserts that lim,- [1, a) = 0 pointwise.
Altogether, we have shown that lim,-o a) + [I, a)) O pointwise, and is bounded as a consequence Lemma B.2. Thus, by another application of the dominated convergence theorem together with (C.4), we have that
lim d (( --0
) =0
It follows that there exists some 78 0 for which drp i (5T.
8/2 whenever S Ta.
For any such r, by Theorem 4.5, there exists nı E N for which
,
S 2
For this choice of r and nı. by the triangle inequality, dppp (5"" (5T) < 8.
Altogether, taking 7 = min To} and n = max ne, n. ), we have that
dpp (5 (T, (55) S and l1 Fee llsup S 2
as well as To complete the proof, note that
dysp' (5" (*) < 8 and
1|65" - leurlsuup S
9,02" 2 0,00" 2 Gief
so that 9, 55 is e-reference-optimal.
Theorem 4.7. Suppose Assumption 3.4 holds and A is discrete. Let p, p/ (1) and w E P (XxA). For any e, 8 > 0 and T" (²(XX A, (R)), there exists O, decoupled o+ and Nopt: neval E N such that
dppp (Stneal (xthli) 50 and 9.0 is e-reference-optimaa where Çn+1_ T" 4T.0 nwr, and gu+1 J for 5o S"oat
[Source]
32

---

Proof. Appealing to Theorem 3.10, for any 5 > 0, any temperature decoupling gambit yields TB and an associated decoupled temperature 0g o (Ta) such that
dpw s0 (CT.O, .(* - 8/3
whenever S Tg. Moreover, as shown in the proof of Theorem 4.6, for small enough T/J-
( 5
whenever T S here, we recall that Ço the entreps-regalarized return distribution function for the decoupled policy x=10.
Now, define दي = " gont, and Ç0 = (m7 Ço By the triangle inequality, we have dpw wo ((Tii ,5 S domp ((T 10 ymu çav, (J. y
+ dmpp (mmprul(1 ((T7" T1" yrum (o0,,,200
(0)
yoout got (JZ y gas) + daw un (1J7" "or"""""onn your gat (J7 10 ymac (Ti0)
dpw' un (1F y
(6)
I dear no youugrt (J y gast)
+ y m dp(( ((Ti (
(0)
S maal 20 yr
Here, (a) leverages the fact that दTe is the fixed point of Ih by definition, (b) invokes the contractivity of JT shown in Theorem 4.2 appealing to the fact that x= is a BG policy for reference ne and (c) follows by Lemma C.I. As a consequence, again since |y| < 1, for sufficiently large Ropt, Neval E N, we have
( S 8/3.
Altogether, by the triangle incquality once again, for the choices of Ropt neval T, o above,
ap (-)) S dpw' dp::
(i(:i10 ((000
+ dpp' nn (5 (7000 + dpp (5 (())
8
5
s
+
3
3
8.
This completes the proof of the first claim. It remains to show now that S is e-reference- optimal. Towards this end, we note that by Theorem 4.6 and 4.7 that there exists Te O, n. N such that
di di(Snow 50 S e/3
(C.5)
whenever max (Tevall Nopt) 2 n1e and 7 S Te, To proceed, we note that for any (I, E.
10, Tm" (x,, a) - (ir (2, a)l S l0," (x, a) - (x, a) + (x) - Fee (2,
< 1 (z, a) (x. a) te/3,
where the last inequality holds for small enough by Theorem 3.2. Continuing, we have
l (z, a) - (2 a),, = ,,," (s, - 9." (x,
=> / (V-gz - -V, dPa.a
5 < - -V, -Vlufllsup
< 7110 9,- lllaap:
33

---

where the final inequality holds since V,, as log-sum-exy exp, is I-Lipschitz. Now, again by Theorem 3.2, for small enough (inducing small enough σ). we have
2/1 q lleup I 7116 Verllaup + 7119 gee lestlaup c e// =//3.
Altogether, have that
supla, -
(2, 1 (z, a) S s t/3 = 22/3.
8,0
Next, since di (pi, p2) 2 E (2 2.) 0p2 11 Z21l. we have that
|0 of Ilaup - (Ç (T-0) S 5//3,
by (C.5). Now, by yet another triangle inequality,
|| - flup S ||2 17 2 || + 11
gref laup
S 2 - 20/3 =.
Consequently, we have
g 8.26
2 209 2 Tref
Thus, we have shown that 9, Qua is e-reference--ptimaal completing the proof.
Lemma C.1. Ler (XxX,, (R)), T,O >> O, neval Rapt N be given. Define g "unc. and let n = a0, Then, we have
dp((J yumat , (J+ 7 ymond goit) S meal 7 12
Proof. By Lemma C.2, we have
dp((T²") ) " yoout (ooul) (iit)
5 17- - 4h sup)) + + 1 Ta llaup + 110 9 llup..
remains bound || 2 sup However, by Lemma 4.4 and the contractivity of B we have that
110 - leup S mmpe
Since |y| < 1, follows that
7z((T²7" t0 yrow gou (I7
yreud youn S mope/p
Lemma C.2. Let Ç K" (X A, (R)), J. 0, and neval N be given. Then
(T(()))ool) ( () S (T ||20 - 4:6 (tllsup) 1/20 T- - Gllaup + ||2g 90 ssu.
Proof. For simplicity, we define मे = 2 and = - We want to bound
Zp (TT)) ( (J7 (Iz)nam) novel ().
By Lemma C.3, we have
(,( Rnd 5, Nin
$ ,d, (T Real -1,, (T7 27 sup n.b
||id|| [|Ie||10((TE)))==0000))) ((0.))) + 02)
< 7 2d ((T
Reo -2, (JP) 2()) + 2 2
sup
| 01 + C22
y.
+ 27 sup
sup n.
||id|| ((TT)) ()a + C2
< (td +
Σ
sup (e + C2
km
n.b
34

---

where
sup 1
T all and 02 = 1lll + 2 - To
Now ((T5-%)
% )r go EeN is the sequence of return distributions generated by iterative applications of
a contractive operator on K (X A, (R)). Thus,
sup id ( 22k))))))) S c3 < oc, n.b
where 03 constant depending only on p,7,O,T, ||rllsup- remains bound ei and cg. By Theorem 3.6, we have
C1 = sup ||, 2,. -
$ (T gd (lau)
Thus, since | llaup is uniformly bounded for any > 0, we have shown that
) S, (I+) () $ (5-S - 9 ((llp))) 11/20- +- - Gallaup + 1125 9 leup.
Lemma C.3. Let 0, e. [L, q. E Mx(X A), S. E (X ((R)) Then.
9..-996,---90 5)
I ydp(si5))
||id|| (F( TH 1/ c - 22 1rrv + 211a 9'll d'llsup]
+
sup (xxxx)
where Cq,q" == llupp
Proof. Observe
dx( 9,
J.. ()) S (T,. () (T ".
S d, () Cd,
So by Lemma C.4, we conclude.
Lemma C.4. Let E K(X x A, (R)) and q, q' E M1(X A). For any T > defining x S for E Eg..9'. denoting Cq.s² minfll4/lla I'llspp 11g }. we have
27
sup (xxxxa)
( n 11 Cg.g T 11ll + 211g q d'llsup)
Proof. For notational simplicity, we define
E = (proj rk1[9 ) @ (9)
Then, by the definition of J, have
(T ( ((),a (br(a,)))) orojj ( s Pa.a)
Following, by [40, Theorem 4.8), we have
d Dn,. (T (2,a)) S> ( )dP.
35

---

We will now estimate the integrand above. By the definition of g for any E X, denoting T 9,9. have
dp (b- m5)).1 o )rooj*) (Sr, *
@ mz), - | (ggh.1 proj (G., @ m
(flz- - 2 drd
1/9
inf
|| ( KL( |
( onn')
[I(a)
I(x)
The inequality is due technique employed in the proof of Theorem 4.5. Next, by [40, Theorem 6.15], we bound I via
[(x) < 2 sup
|id |idd| LF m 11 < 2 sup
|id || T 11TV
z'
r'aa'
Now, for I1, we have
l1(x) S min (ll''laapp: Ilqllsup - 2 || + 2/14 o' slsup
as shown in the proof of Lemma C.s. Therefore, we have shown that
dp (JT+))) (J ().a)
sr /
sup || (. 11l - Tz 11tV + caarr 11 - z 1|Tv + +211g d'lssup d.a..
C.1
Supplemental Lemmas for Section 4
Lemma C.5. Let Č K (X x A, (R)), and for any N, define gn+ J²5" with =P=T5
(Xx
Also, define v" := 5n Then for any x E X, denoting C2 == [Q((I,)) - of (z, )
KKL TH 1 K
s (2 C TV max (~ Cr
where C1 < 30 is a constant.
IfT 2 27 Ca, then for constant C2 < oc, we have - T|KL(=T || ( - KL(TI** - (2 +2 -2
Proof. First, observe that
T [KL(TT) T T - KL (
/ 2 (z. da) - V+2("(x) - / a) + v
/ (x,
/ (1(2, a) d da) + 1.255 (z)
dr a)) $ (z )1=== - 'Irv + 2|105" (z. (x )1(
By Lemma 4.4 and the contracttvvty of B;, we note that
(z, -) (2, . ( S 7m 11 (2,-) 2. )1
Then, by Theorem 3.6, we have
1127(,,) :) - (1, [0²( 9ill (w)))) S $5 "7 101 & (xzg)
|| TI 1| 1
otherwise.
Note that | llup I ||r|lsup/ (1 - -7). Indeed, the upper bound is free; the lower bound comes from comparing of with o.t for क re Altogether, we have that
r KL[ xeef) - KL( ( - (2
|| I|rll 1/2
max {rrCas V
1 1-7
36

---

for C2 = 1105(2,) (z, )l (1z))
If r 2 2y, then have the stronger bound
[ || -
KL || s C'+
where C" = (2e /1lp 7)
Lemma C.6. Let E 12 x A, (RR). For any E N, define Çn+1 -n,", with e J. Denoting by (p1:P2) the space of all couplings between the measures P1. all x E X we have
inf RE 2 / dx S
7012 (1-1- V
2 (x, (1o0
where nh = 9, and Cp < 00 is a constant depending only on p and ||rllsup- Moreover, when 7 21og 2() 9 (2, ) log T). we have
/ |a - 2/P S C 7"8
inf
7)p -10((,)) (7 (llm
where C = (2e 33CC/
Proof. For notational convenience, define w = -I 8 m.. for E (T, Then,
/ 1 12--2'P 21P = de( wi'"
W =
inf
(a)
S 2 / |1|pd|=>> x
2- /
|=| ((P
d|= -
(b) 320 10r- 11r11Bup
|| 1/V,,
(1 7)p
where (a) applics [40, Theorem 6.15], and (b) uses that the support of ÇTi is contained ball of radius 3/|r||sapp/ - -)) By Lemma B.6, it follows that
Wh S (2,5"(z.)) - (1, (1l
2/2
$ CCp p (1 7)p V - 1-11)0(( - qf ==
where the last inequality holds by Lemma 4.4 and the y-cottractivity B, with Cp == 320-11|r || Moreover, if > logg -log 2|| (z((zz -) - of (z. ))( - -logg)) then
1105 - 2 (r. (1)) <T/2
for each X, so by Theorem 3.6,
W <
2e 3 3
-- - 105(x,.) (x, (11x=)))))
Cp (1
7m
(1-7)²
D Comparison between vanishing temperature limits of ERL with and without temperature decoupling
In this section, we compare and contrast the properties of vanishing temperature limits of standard ERL (assuming they exist) with those achieved by the temperature decoupling gambit. As we showed in Thcorem 2.3 and Theorem 3.9, both schemes achieve refereence-ppimmaliy in the limit; yet, their
37

---

limits may be notably distinct according to criteria beyond the RL objective, we saw in Sections 3.1 and 4.3.
In the remainder of this section, we will define çref, == <" <a as the return distribution function corresponding to the limiting lemperature-decouplet policy, and C := = as the return distribution function corresponding to the limiting ERL policy xRRL, assuming such a limit exists.
very nice property of re is that it is easy to characterize as the optimality-fltered reference, as per Definition 3.8. In particular, ne.. is characterized entirely in terms of the optimal action-value function qº and the reference policy π' On the other hand, as we see explicitly in Section 3.1, ERLL does not have such simple characterization: influenced also by the transition dynamics of the MDP (as well as the 9º' and 7
A notable consequence of this fact that one reason about π- generically across MDPs, which not the case for ERL.. For instance, in any MDP, if x0 the uniform policy, x - is the uniform policy optimal actions. Thus, one say definitively that all actions leading to optimal behavior are played equally under rf, But this is not true of π ERL,; in general, it is difficult to characterize exactly how x ERRL behaves: among set of MDPs with equal q", the corresponding ERL. can vary significantly.
Similarly, this property of ,e enables one to easily influence the optimal policy that is achieved via temperature decoupling by intervening on ,- Again, this is possible due to the simple character- ization of re as the aptimality-filterrt reference. Suppose, for example, there exists particular action a acarry that you want to avoid whenever possible (e.g., certain controversial phrases in language generation). It may be undesirable filter this action out completely (say, by choosing n- to never play a duary), because perhaps from some states this action is necessary to achieve optimal return. Instead, with temperature--eccoulingg you can choose re to play this action with very low probability (e.g., o (a*Cary) = Pref for each 2)). By Theorem 3.9, scary will only ever be played when it achieves optimal returns, and moreover, as long other actions exist that achieve optimal returns, scary will be played with much lower probability.
The same logic does not hold, in general, for π ERL. As to play autrry with high probability even if ाe' plays it with low probability. Suppose, for instance, that after playing acary in state I, it is optimal to play re subsequently for the rest of the episode. Then x EERL,* may strongly prefer to play accary from state æ, even if other actions can achieve the same expected return. In fact, depending discount factor, π ERRL, may play a scary from state x with arbitrarily high probability.
in Section 3.1, n ERL, may continue
the transition kernel, the scale of the rewards, and the
38

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe our abstract and introduction outline and faithfully summarize the content of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
We have included discussions of the limitations and scope of our results throughout our work, rather than within a specific "Limitations" section. See, for example, Section 6.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
39

---

Answer: [Yes]
Justification: For each theoretical result, we provide rigorous proofs in the appendix, and the assumptions stated very explicitly.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We present all details about the experiments we ran in order for them to be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
40

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification:
The experimental results merely visualizations, all of which can be verified mathematically.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
Seglli/gguides/ooeeSuue ierriiisssooooliii onPolicy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wttpp.ccpppuiiiieeeeeeeepppppppp cctpublic//aeccccccccccccccc gwwpi/gg///////eeee// /Ce///eeeubbiiic settsubbmsssiiono ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: Our experiments are on toy problems where such details are vacuous; though, all experimental details are fully specified in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Statistical analysis does not apply for our empirical results, which have deterministic outcomes.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
41

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: Experiments can be trivially run on any modern computer.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https :tttpp:://nuuu http///eeeri...eeeeeeeee cc Taabblc//Ehicsuttie
Answer: [Yes]
Justification: Yes, we have read the NeurlPS code of ethics, and believe our work conforms it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We believe our work is foundational, without a direct path to negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
42

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe our work poses no risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We did not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
43

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work did not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our research did not involve crowdsourxcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
44

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy btttp::/neeuips... httpr//ps..c/ooofeeeeee////2255/ for what should or should not be described.
45


Reviews:
[{"review_id": "e54U3cFeuQ", "reviewer": "Reviewer_4ZKB", "summary": "In this paper, the authors analyze the convergence properties of the optimal policy and the associated return distributions in entropy-regularized reinforcement learning as the regularization temperature approaches zero. They show that, under a temperature decoupling mechanism, the optimal policies converge to a \"reference-optimal\" policy, and the corresponding return distributions converge to \"reference-optimal\" return distributions.  They use toy examples to verify their theoretical findings.", "questions": "See the Strengths And Weaknesses part above.", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n* The paper demonstrates the limiting behavior of the optimal policy and the associated return distributions for entropy-regularized RL when the temprature vanishes. The theoretical findings are novel.\n* I checked most parts of the proofs. The theoretical analysis is rigorous and sound.\n\nCons:\n\n* **- Motivation and Contribution:**\n   The motivation of the paper is unclear, and the contributions appear limited in scope. On the theoretical side, the proposed temperature decoupling mechanism seems artificial and lacks intuitive justification. The paper does not address the fundamental question of whether convergence can occur in standard entropy-regularized reinforcement learning (without the temperature decoupling mechanism) as the regularization temperature vanishes. On the practical side, the theoretical results seem disconnected from practical applications. Although the authors claim that their theoretical insights lead to a practical algorithm with convergence guarantees, the algorithm described below Theorem 4.7 relies on a fixed ($\\tau$, $\\sigma$, $n_{\\text{opt}}$, $n_{\\text{eval}}$), and the resulting policy sequence will not converge to the stated \"reference-optimal\" policy under the fixed choice of  ($\\tau$, $\\sigma$, $n_{\\text{opt}}$, $n_{\\text{eval}}$). Furthermore, Theorem 4.7 only asserts the existence of certain parameters ($\\tau$, $\\sigma$, $n_{\\text{opt}}$, $n_{\\text{eval}}$) for which the algorithm performs well, but provides no guidance on how to select them in practice.\n\n  **- Clarity and Presentation:**\n   The paper is written in an extremely technical style that makes it difficult to follow. Understanding the theoretical results and verifying the proofs is a struggle due to the dense technical details and uncommon notations. I recommend that the authors adopt more standard notation used in the RL literature and reduce the introduction of new symbols such as $ \\mathcal{G}_\\tau, \\mathcal{J}_\\tau, \\mathcal{B}_\\tau, \\mathcal{V}_\\tau, \\mathcal{Q}, \\mathcal{O}, \\textup{N}_{\\text{ref}}, \\odot, \\otimes $, and various subscripts/superscripts. Additionally, the authors should highlight their main theoretical results more clearly and consider moving unimportant lemmas and technical details to the appendix.\n\n  **- Empirical Evaluation:**\n   The empirical evaluation is limited to simple toy environments, and no experiments are presented to validate the proposed control algorithm. This significantly weakens the practical impact of the paper.", "was_revised": true, "final_justification": "After reviewing the authors’ rebuttal, I find that they have not made sufficient efforts to address my concerns or improve the quality of their paper. While the work contains some interesting theoretical findings, I maintain my negative evaluation of the paper.", "rebuttal": "We thank the reviewer for their feedback and appreciate their verification of the proofs.\n\nBelow, we speak to the remaining points that you have raised. \n\n> The motivation of the paper is unclear, and the contributions appear limited in scope.\n\nCan you please share something concrete about your concerns regarding the motivation and scope that we might address specifically?\n\n> On the theoretical side, the proposed temperature decoupling mechanism seems artificial and lacks intuitive justification. \n\nCan you please explain what you mean by “artificial” in this context? Our temperature decoupling mechanism is justified by its ability to realize a unique interpretable, diversity-preserving (reference-)optimal policy in the vanishing temperature limit. The mechanism itself arises directly from our mathematical analysis of ERL—is this not sufficient?\n\n> The paper does not address the fundamental question of whether convergence can occur in standard entropy-regularized reinforcement learning [...] as the [temperature] vanishes. \n\nWe do address this question. Remark 3.6 speaks to its underlying mathematical difficulty. Thm 3.12 (precisely, Eqn (B.1)) establishes an explicit relationship between ERL-optimal policy convergence and ERL-optimal action-value function convergence. And Figure 4.4 shows how standard ERL is numerically unstable at very small temperatures. While we do not close this open problem, we nonetheless provide a clean formula (via temperature decoupling) that realizes a (reference-)optimal policy---the best one can hope for, as we show---that is more interpretable than a potential vanishing temperature limit of ERL-optimal policies.\n\n> On the practical side, the theoretical results seem disconnected from practical applications.\n\nPlease see our response to Reviewer 4Nx8 wherein we discuss some ways in which our results can inform practical algorithms.\n\n> Although the authors claim that their theoretical insights lead to a practical algorithm with convergence guarantees, the algorithm described below Theorem 4.7 relies on a fixed $(\\tau, \\sigma, n_{{\\rm opt}}, n_{{\\rm eval}})$, and the resulting policy sequence will not converge [...] under the fixed choice of $(\\tau, \\sigma, n_{{\\rm opt}}, n_{{\\rm eval}})$. Furthermore, Theorem 4.7 only asserts the existence of certain parameters […] for which the algorithm performs well, but provides no guidance on how to select them in practice.\n\nThm 4.7 and the discussion following asserts the existence of an approximation scheme. Rather than directly lead to  “a practical algorithm with convergence guarantees”, as you have stated, it “outlines an algorithm for estimating” the return distribution associated with $\\pi^{{\\mathrm{ref}}, \\star}$, as we have stated. Specifically, we prove that by decreasing $\\tau$ (and consequently, $\\sigma$) and increasing $n_{\\rm opt}$ and $n_{\\rm eval}$, estimation accuracy increases. \n\nIn practice, one can simply run the algorithm suggested by Thm 4.7 with increasing $n_{\\rm opt}$ and/or decreasing $\\tau$, as we did in Sec 4.3, until the resulting distributions stabilize.\n\nIn contrast, before our work, the best one could do is iterate the distributional optimality operator. This is highly unstable (Bellemare et al. 2017). In particular, there is no number $n$ of iterates beyond which we get accurate approximations of a particular return distribution function.\n\n> [...] I recommend that the authors adopt more standard notation used in the RL literature and reduce the introduction of new symbols such as $\\mathcal{G}\\_\\tau, \\mathcal{J}\\_\\tau, \\mathcal{B}\\_\\tau, \\mathcal{V}\\_\\tau, \\mathcal{Q}, \\mathcal{O}, \\mathsf{N}\\_{\\text{ref}}^\\star, \\odot, \\otimes $, and various subscripts/superscripts.\n\nWhenever possible, we adopt standard notation from distributional RL, particularly with regard to the operators such as $\\mathcal{G}_\\tau$ (see Distributional Reinforcement Learning by Bellemare, Dabney, Rowland, 2023; Chapter 7). The notation $\\mathcal{J}$ is standard for describing the performance of a policy in RL (Sutton and Barto, 2018). \n\nOther symbols and operators, such as $\\mathcal{V}_\\tau$ and $\\mathcal{Q}$, are crucial for lifting ERL operations to distributions; Def 4.3 and Lem 4.4, for example, would be unruly without them. Again, dealing with such issues via the introduction of operators is standard practice in distributional RL (Bellemare, Dabney, Rowland, 2023; Chapter 4).\n\nWe concede that we have taken some artistic license with other notation, but only when no universally agreed-upon notation exists (e.g., $\\mathcal{O}$ for the set of occupancy measures). \n\nFinally, as a practical matter, our proofs and statements would be much longer without our notational choices like $\\otimes$ and $\\odot$.\n\n> [T]he authors should highlight their main theoretical results more clearly and consider moving unimportant lemmas and technical details to the appendix.\n\nWould you please point to which of our main theoretical results could be highlighted more clearly?\n\nMoreover, are there any specific lemmas or technical details you think would be better suited for the appendix? If so, we would be happy to move them, if moving them did not disrupt the flow of our paper by, for example, requiring the reader to go to the appendix to be able to read another statement in the main body of the text.\n\n> The empirical evaluation is limited to simple toy environments, and no experiments are presented to validate the proposed control algorithm. [...]\n\nWe stress that the purpose of our experiments was to illustrate the different behavior induced by our schemes compared to standard ERL schemes, not for evaluation. Our proof of Thm 4.5 that iterative applications of the soft Bellman optimality operator converge conclusively validates this theoretical algorithm. Likewise, our proofs of Thms 4.6 and 4.7 validate the theoretical algorithms they define. Realizing any of these as a practical, deep RL algorithm, for instance, is important and non-trivial. Hence, while we are excited to do this work, we defer it to the future, where it can receive its due care."}, {"review_id": "1Ko6jc10Ng", "reviewer": "Reviewer_x7BH", "summary": "This paper develops a rigorous theoretical framework for analyzing the limiting behavior of entropy-regularized reinforcement learning as the temperature parameter vanishes. It introduces a novel **temperature decoupling mechanism** that ensures convergence to an interpretable, diversity-preserving optimal policy. The authors further extend their results to **distributional reinforcement learning (DRL)**, presenting the first convergent dynamic programming schemes for estimating return distributions in the control setting.", "questions": "1. Can the authors provide any empirical intuition or heuristics for selecting the temperature-decoupling schedule $\\sigma(\\tau)$? For instance, what are reasonable scaling laws?\n2. *In practical RL with neural policies, does the temperature-decoupled policy retain its convergence properties? Have the authors considered the effect of function approximation?\n3. How sensitive is the convergence to the choice of $\\pi_{ref}$, especially when $\\pi_{ref}$ poorly overlaps with the support of the optimal policy? Can regularization adaptive to support mismatch be incorporated?\n4. For continuous actions, can the authors elaborate more on the implications of weak convergence? What does this imply for action sampling and training?", "limitations": "see weakness and question part", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The paper rigorously establishes convergence of both value functions and return distributions in the vanishing temperature limit—solving long-standing open questions in DRL.\n2. The temperature decoupling idea is theoretically elegant and provides a clear path to interpretable and diverse policy learning.\n3. The paper includes complete, detailed, and technically solid derivations that should be reproducible by other researchers.\n4. It offers the first convergent algorithm for distributional control, which could significantly influence future work on safe and robust RL.\n5. Despite the mathematical density, the flow is clear and well-motivated.\n\n### Weaknesses\n1. The toy examples, while illustrative, do not demonstrate practical effectiveness in realistic RL benchmarks.\n2. There is no concrete guidance on selecting $\\sigma(\\tau)$ in practice, nor any adaptive scheme proposed.\n3. Convergence of policies in continuous action spaces remains weakly addressed. While weak convergence is claimed, its implications for learning with function approximation are not explored.\n4. The framework relies heavily on the referenc policy. Although reference-optimality is well-defined, performance heavily depends on the support and smoothness of the reference policy. This may limit practical deployment in high-dimensional settings.", "was_revised": true, "final_justification": "The authors have provided detailed explanations addressing my initial concerns, which have resolved them fully. Overall, this is a solid piece of work with strong theoretical results, exploring distributional RL and its convergence to a particular optimal policy. However, I believe the paper would benefit from including experiments to demonstrate the effectiveness of the proposed algorithms, as well as an empirical sensitivity analysis. Therefore, I am maintaining my rating of borderline accept.", "rebuttal": "We thank the reviewer for their feedback and appreciation of the quality, elegance, and significance of our work.\n\nBelow, we speak to the remaining points that you have raised.\n\n> The toy examples, while illustrative, do not demonstrate practical effectiveness in realistic RL benchmarks.\n\nIn this submission, we establish a theoretical foundation upon which such work can be done. We view this kind of important, non-trivial empirical study as a follow-up to our theory-oriented work. \n\nWe are certainly excited about the prospects of leveraging our findings to inform the design of new algorithms. Please see, e.g., our response to Reviewer 4Nx8 for some of our ideas regarding this. If you’d like to continue this discussion, please let us know.\n\n> There is no concrete guidance on selecting $\\sigma(\\tau)$ in practice [...] \n\nAs stated in Def 3.10, we simply require $\\sigma(\\tau)/\\tau \\to 0$ as $\\tau \\to 0$. In practice, any choice that obeys this relationship within either a fixed or adaptive annealing scheme is viable. We will be sure to state this explicitly in our revision. Note that in our illustrations, we set $\\sigma(\\tau) = \\tau^2$. \n\n> Can the authors provide any empirical intuition or heuristics for selecting the temperature-decoupling schedule $\\sigma(\\tau)$? For instance, what are reasonable scaling laws?\n\nHeuristically, the faster $\\sigma(\\tau)/\\tau \\to 0$, e.g., $\\sigma(\\tau) = \\tau^3$ rather than $\\tau^2$, the closer the resulting policy is to the Boltzmann–Gibbs policy associate to $q^\\star_{\\rm ref}$ at temperature $\\tau$. This \"theoretical scaling-law” is made precise in Eqn (B.1). We will be sure to move this equation and surrounding text to the main body of the paper, in our revision.\n\n> While weak convergence is claimed, its implications for learning with function approximation are not explored. \n\nOur focus is to understand the nature of the soft optimal policies themselves, independently of how they are represented. While exploring the implications of weak convergence for learning with function approximation in ERL is certainly an interesting topic, we leave this to future work.\n\n> In practical RL with neural policies, does the temperature-decoupled policy retain its convergence properties? Have the authors considered the effect of function approximation?\n\nPlease see above.\n\n> For continuous actions, can the authors elaborate more on the implications of weak convergence? What does this imply for action sampling and training?\n\nOne implication of only having weak convergence of the policies is that the associated return distributions may not converge. On the other hand, our results show that we can upgrade TV convergence of policies to convergence of return distributions (see the proof of Thm 3.13, inequality (c), for example).\n\nThat said, would you please clarify what you are referring to regarding action sampling and training?\n\n> The framework relies heavily on the reference policy […] performance heavily depends on the support and smoothness of the reference policy. \n\nWhile you raise important points, the limitations you speak to are in fact fundamental limitations of every ERL method, not just our own. Moreover, we are the first to uncover this truth (see, e.g., Thm 3.2). Indeed, our results show that the best one can hope for in ERL is to approach a reference-optimal policy. (And this is true even when a uniform reference policy is used. Please see the discussion after Def 3.3.)\n\n> How sensitive is the convergence to the choice of $\\pi_{{\\rm ref}}$, especially when $\\pi_{{\\rm ref}}$ poorly overlaps with the support of the optimal policy? Can regularization adaptive to support mismatch be incorporated?\n\nIf the reference policy does not suitably overlap with some optimal policy, there is little hope for ERL to recover a classically optimal policy, even beyond our novel techniques. (Please see the discussion after Def 3.3.) Furthermore, since ERL agents only ever see actions supported by the reference policy, any scheme that might compensate or correct for a support mismatch would require external information."}, {"review_id": "mmLs2km940", "reviewer": "Reviewer_4Nx8", "summary": "In generic MDPs, there can be more than one optimal policy exist and not all of them behave as expected. One of the current methods that handle this issue is entropy-regularized RL which enforces the uniqueness of the optimal policy. This paper studies the evolution of ERL algorithms as temperature vanishes beyond tabular MDPs.", "questions": "* Why it is difficult to understand the behavior of ERL methods beyond the tabular setting? In other words, what makes continuous MDPs fundamentally different from tabular MDPs in the context of ERL? Can you give an insight to it without using any math?\n\n* Can you provide some more practical examples such as MuJoCo and Atari, other than the toy problems presented in the paper?", "limitations": "Yes, they are discussed.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* The problem studied in this paper is well-motivated from the theoretical perspective.\n\n* The paper is written in a clear and mathematically rigorous way.\n\n* Related works are adequately discussed.\n\n**Weaknesses**\n\n* As claimed in the introduction, this paper aims to study ERL beyond tabular MDPs. However, both numerical examples provided in section 3.1 and 4.3 are tabular with only a couple of states and actions.\n\n* It is less intuitive to relate the theoretical analysis presented in this draft to the practical problems one may encounter in practice. While I acknowledge the full rigor the authors made throughout the entire paper, the practical significance of this work is not very clear to me.", "was_revised": true, "final_justification": "It is solely an RL theory paper. The theoretical results look good to me, but I am not an expert in theory and may have missed something. The empirical results are rather limited, as pointed out by others, too. Overall I am neutral on this work and would leave the decision to other reviewers and the AC.", "rebuttal": "We thank the reviewer for their feedback, and for their appreciation of the mathematical rigor and the motivation of our work. \n\nBelow, we speak to the remaining points that you have raised.\n\n> It is less intuitive to relate the theoretical analysis presented in this draft to the practical problems one may encounter in practice. While I acknowledge the full rigor the authors made throughout the entire paper, the practical significance of this work is not very clear to me.\n\nWhile our paper’s primary focus is theory, our results do shed light on the behavior of existing algorithms and open the door to new algorithmic concepts that can be applied to, e.g., deep RL. \n\nMany practitioners use ERL as a means to approximate a classically optimal policy. But we show that, in reality, ERL can only recover a reference-optimal policy, which could be rather far from optimal in the classic sense.\n\nMoreover, safety is an increasingly important consideration in the design of autonomous agents. Hence, understanding the distribution over outcomes under policies is critical. Our techniques inform ways of ensuring the stability of distributional return estimates over the course of policy improvement for the first time, which can be crucial in downstream safety approaches.\n\nFinally, here are two ways in which our techniques might be applied:\n \n1. Consider Soft Q-Learning where the soft Q-function is estimated with a temperature schedule $\\sigma(\\tau\\_t)$ (say, with $\\sigma(\\tau)=\\tau^2$) using standard methods, but when interacting with the environment, the softmax policy with temperature $\\tau\\_t$ is deployed. This strategy induces more exploration. Furthermore, it's likely that this “action-policy” will converge to $\\pi^{\\mathrm{ref},\\star}$ in the vanishing temperature limit, given our theory. \n\n1. Consider, e.g., in Soft Actor-Critic, using a temperature decoupled policy as a target policy as a means to further stabilize training. \n\nActually implementing either of these ideas (or any other idea one might conceive) and the necessary work surrounding any new practical algorithm, of course, is non-trivial. Hence, while we are excited to do this work, we defer it to the future, where it can receive its due care.\n\n> […] this paper aims to study ERL beyond tabular MDPs. However, both numerical examples…are tabular with only a couple of states and actions.\n\nThe purpose of our demonstrations was to illustrate how our method differs from standard ERL. The only regime in which ERL is known to produce convergent policies in the vanishing temperature limit is the tabular setting. So the tabular setting is the best case scenario for the “baseline”. Our numerical demonstrations show that, even in this ideal case, there is a notable difference between ERL with and without temperature-decoupling—they produce different policies, and our temperature decoupling method is more robust to numerical precision errors.\n\nMoreover, beyond the tabular setting, there would be a confounding factor in our demonstrations. Any observed discrepancy between the standard ERL policies and the temperature decoupled policies could be the result of the standard ERL policies not converging! Working in the tabular case removes this confounder. So any difference between the ERL policies and the temperature-decoupled policies in our demonstrations is due only to the policy optimization scheme.\n\n> Why is it difficult to understand the behavior of ERL methods beyond the tabular setting […] what makes continuous MDPs fundamentally different from tabular MDPs in the context of ERL?\n\nThere are many factors that make ERL more complicated in continuous MDPs than in tabular ones, much like in standard RL. In ERL, we have a new stream of challenges that deal with ensuring that the regularization penalty is bounded. When the action space is continuous and bounded, for example, the uniform reference policy does not guarantee finite regularization penalty anymore. If the set of optimal actions is very sparse, for example, and the density of those actions under the reference is small, the penalty will explode.\n\nIn addition, as we show, the quality of the best policy that ERL can approximate (that is, even outside the methods we introduce) in the vanishing temperature limit depends on the “smoothness” of the optimal Q-function when the action space is continuous. For instance, if the true (unregularized) optimal Q-function has large spikes at particular actions (say, the action values are all $0$ except at a particular one), then vanishing temperature limits of ERL cannot recover optimal policies (or in the parlance of our paper, reference-optimality is not the same as optimality). This is a fundamental limit of ERL that had not been known before our work.\n\n> Can you provide some more practical examples such as MuJoCo and Atari, other than the toy problems presented in the paper?\n\nIn this submission, we establish a theoretical foundation upon which such work can be done. We view this kind of important, non-trivial empirical work as part of follow-up to our theory-oriented work."}, {"review_id": "LHuBSEjfPt", "reviewer": "Reviewer_79Aa", "summary": "This study focuses on limit theorems of the entropy-regularized RL with a vanishing temperature, including the convergence and (reference-)optimality, and then considers the return distribution convergence. They extend the existing analysis in entropy-regularized RL to the distributional context, partially connecting the theoretical results in distributional RL. They proposed a novel temperature decoupling mechanism to facilitate the proof.", "questions": "I read some parts of the appendix, which seems rigorous. Please see my concerns in the Weakness part above.", "limitations": "NAN", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n1. The proof and writing are rigorous, aligning with an RL theory requirement.\n\n2. Albeit simple, some numerical demonstrations are given in some sections, which are useful even in a theory-oriented paper.\n\n\n## Weakness\n\n1. The title is too broad. It focuses on the convergence analysis instead of the asymptotic analysis in classical statistics, such as sample efficiency, which also plays a key role in limit theorems. It is suggested to change the title to emphasize the contribution more precisely.\n\n2. The theoretical contribution should be posited better. The convergence analysis of entropy-regularized RL is relatively mature. For example, [33] provides the faster convergence rate even though it is a tabular MDP. It seems that this study extends [33] to continuous MDP qualitatively without giving a rate, but it extends to the distributional RL setting, is that correct? Compared with [34], the conclusion of this study may not be novel, but the analysis is slightly general. Is my understanding correct? Since RL theory literature is relatively packed, it is very important to highlight the contribution among the existing literature. \n\n3. In the introduction part, some descriptions are ambiguous or not clear. For example, I find it difficult to understand the ‘expected-value optimal policies.’ At least, it is not commonly used in distributional RL literature. The conclusion that entropy regularized RL ensures a unique optimal policy needs some reference. \n\n4. This paper concerns a continuous MDP beyond a tabular MDP, but it seems that we need to require a continuous action space in the analysis. Is there a fundamental gap between the paper’s result with the continuous MDP with a discrete action, or is it also able to extend to the discrete action space?\n\n5. I understand this study tries to be rigorous, but it reduces the readability among potential readers. Maybe it is more suggested to focus on the necessary notations to elaborate the conclusions instead of giving too many notations, e.g., the measure-theoretic language in Section 2.\n\n6. One point that makes me feel disappointed is the convergence for control in distributional RL, which is viewed as a very hard problem. However, I only find the convergence in an entropy-regularized distributional RL instead of distributional RL directly. I think there is still a fundamental gap. Correct me if there is a misunderstanding.\n\n7. The authors motivate the temperature decoupling mechanism given the TV distance is not vanishing in an annealing temperature. Is that consistent among different choices of distances? How do we understand this result? In addition, the authors are suggested to highlight again the motivation of this new mechanism in terms of the analysis. Also, the numerical results in Section 3.1 only suggest the learned policy under this mechanism is different. Are they also able to verify the motivation of this mechanism in some sense? I struggle to interpret Section 3.1 with the motivation of this new technique. \n\n8. A general comment is whether the established results can be useful to guide practical algorithms? If so, the contribution may be more broad.", "was_revised": true, "final_justification": "The theoretical results seem to be a potential contribution, but overall I feel this is still a borderline paper without a clear conclusion among the provided theoretical results. I keep my rating for now, but I become open-minded in the following discussion.", "rebuttal": "We thank the reviewer for their very insightful feedback, and for appreciating the mathematical rigor and the pedagogy of our numerical demonstrations. \n\nBelow, we speak to the remaining points that you have raised.\n\n> The title is too broad.\n\nThank you for pointing this out, we would appreciate any suggestions you have and we’d be happy to change the title. If you have objections to “Convergence Theorems in Entropy-Regularized and Distributional Reinforcement Learning”, please let us know.\n\n> [...] [33] provides the faster convergence rate even though it is a tabular MDP. It seems that this study extends [33] to continuous MDP qualitatively without giving a rate, but it extends to the distributional RL setting, is that correct?\n\nIn [33], the authors provide fast convergence rates of policies to an entropy-regularized optimal policy in a tabular MDP with fixed (nonzero) temperature. You correctly identify that our work extends such a result to general MDPs and additionally to convergence of return distributions, albeit with a slower rate. While our rate is slower, it is for the convergence of fundamentally different objects, so not quite comparable.\n\nBeyond this, our work asserts convergence of a scheme for approximating optimal policies (and return distributions) in the vanishing-temperature limit. [33] only has such a result for policy convergence in the bandit setting, while our results hold in much more generality.\n\n> Compared with [34], the conclusion of this study may not be novel, but the analysis is slightly general. Is my understanding correct? \n\nThis is incorrect. [34] provides policy convergence in a vanishing temperature limit in a different problem setting: they add an alternative form of (non-vanishing) regularization towards a prior distribution over neural network weights. Moreover, [34] also does not consider convergence of return distributions, in contrast to our work. Our work, on the other hand, establishes convergence to a classically optimal policy in the vanishing temperature limit, as well as the return distributions for this policy.\n\n> In the introduction part, some descriptions are ambiguous or not clear. For example, I find it difficult to understand the ‘expected-value optimal policies.’ [...]\n \nThank you for pointing this out, we agree that certain terms in the introduction were perhaps too technical before the context had been set. We will fix these. For reference, “expected-value optimal policies” refers to policies that achieve optimal expected returns (e.g., the standard risk-neutral definition of policy optimality).\n\n> The conclusion that entropy regularized RL ensures a unique optimal policy needs some reference.\n\nWe are unaware of any reference that rigorously proves the existence and uniqueness of optimal policies in ERL (for fixed nonzero temperature) outside of the discrete action space setting. As a result, we have proved the existence and uniqueness of optimal policies in ERL in general MDPs (see Theorem 2.2). If you can point us to a reference that already does this, we are happy to cite it.\n\n> [...] it seems that we need to require a continuous action space in the analysis. Is there a fundamental gap between the paper’s result with the continuous MDP with a discrete action, or is it also able to extend to the discrete action space?\n\nNone of our results require a continuous state or action space. They all hold for both discrete/continuous state spaces as well as discrete/continuous action spaces (unless specifically mentioned).\n\n> [...] Maybe it is more suggested to focus on the necessary notations to elaborate the conclusions instead of giving too many notations, e.g., the measure-theoretic language in Section 2.\n\nWe have endeavored to only put notation in the main body of the paper that is needed to understand the statements we make in the main body of the paper. If there is something that can be moved to the appendix without making it so that the reader has to go to the appendix in order to read something in the main body, we are happy to move it. Please let us know if you have found something like this. Our measure-theoretic language is essential for understanding the definitions and statements of Sec 4. We use it to recast more familiar, classical objects in RL in Sec 2 in order to give the reader some time to become acquainted with it. \n\n> [In] the convergence for control in distributional RL, which is viewed as a very hard problem[,...] I only find the convergence in an entropy-regularized distributional RL instead of distributional RL directly. [...] Correct me if there is a misunderstanding.\n\nYour understanding is correct. That said, our approach leveraging vanishing-temperature limits of distributional ERL is still a major step towards solving the long-standing open problem in distributional RL: achieving consistent and stable estimates of the performance of a policy undergoing optimization. To clarify:\n\n1. Thm 4.5 establishes (with rates) the convergence of an exact entropy-regularized distributional optimality operator at fixed nonzero temperature. This is indeed not the classic RL setting.\n\n2. Thm 4.7, leveraging Thm 4.5, considers an approximate operator to qualitatively allow one to approximate the (reference-)optimal return distribution (in the standard sense, without entropy regularization) to an arbitrary tolerance.\n\nUnder existing distributional dynamic programming methods, one cannot hope for anything more than an approximation either (since the distributional Bellman operator can be applied at most finitely-many times). Beyond that, the classic distributional optimality operator does not produce convergent iterates. So it cannot be used to approximate any particular optimal return distribution function.\n\n> The authors motivate the temperature decoupling mechanism given the TV distance is not vanishing in an annealing temperature. Is that consistent among different choices of distances? How do we understand this result?\n\nOur focus on the TV distance is tied to our proofs regarding return distributions, wherein we find that policy convergence in TV yields return distribution convergence (see the proof of Thm 3.13, inequality (c), for example). In general MDPs with continuous states and actions, the convergence of entropy-regularized optimal policies under any metric is an open problem.\n\n> the authors are suggested to highlight again the motivation of [the temperature-decoupling mechanism] in terms of the analysis.\n\nThank you, we will do this for sure. We will, for example, move Eqn (B.1) and surrounding text to the main body of the paper, in our revision.\n\n> [T]he numerical results in Section 3.1 only suggest the learned policy under this mechanism is different. Are they also able to verify the motivation of this mechanism in some sense? I struggle to interpret Section 3.1 with the motivation of this new technique.\n\nIndeed, we can explicitly provide closed forms of the limiting policies in standard ERL and the temperature-decoupling mechanism, and they line up precisely with those depicted in Figure 3.1. The derivations aren’t difficult to compute, though they are tedious and will not fit here. Please let us know if you’d like us to provide derivations in a subsequent comment.\n\nThis demonstration illustrates the diversity of the limiting temperature-decoupled policy. As predicted, the temperature-decoupled limit is uniform on the set of optimal actions. In contrast, the limiting standard ERL policy (which we can reason about since this is a tabular MDP) carries over influence from the entropy regularizer: it effectively ignores the green action from the root state, despite the fact that it can achieve optimal return for the un-regularized objective.\n\n> A general comment is whether the established results can be useful to guide practical algorithms? If so, the contribution may be more broad.\n\nPlease see our response to Reviewer 4Nx8 wherein we discuss some ways in which our results can inform practical algorithms."}]

Author rebuttal:

Meta-review: This paper studies the convergence of entropy regularized distributional RL with a diminishing temperature. The authors provide a comprehensive theoretical characterization of the problem, including the hardness. To ensure convergence, the authors provide a novel  decoupling of the temperature. Namely, in the most naive approach, the same temperature is used for both defining the optimal value function and computing the softmax. The authors instead propose to used two different temperature and one decays significantly faster than the other. This resembles the two-timescale learning rates idea and I find this novel and interesting. Sufficient theoretical analysis is provided to support the effectiveness of this approach.

The reviewers all agree that this is a technically solid paper. The main concerns lie in motivation, in presentation being too technical, and in lack of larger empirical study. I consider this paper as a theoretical paper and believe the current empirical results are sufficient for a theoretical paper. I do agree with the concerns about motivation and presentation being too technical. After calibration, I feel the novel and principled decoupling mechanism is a valuable idea to the community and outweighs the concerns. I, therefore, recommend accept.

That being said, I do have a few suggestions. (1) First, I suggest the authors to significantly tone down the claims. For example, the first few sentences in the abstract read like this paper is going to address a general RL problem. But it turns out that the paper is addressing a very specific problem in distributional RL and the way the paper addresses the problem (the decoupling mechanism) is far away from what practitioners do. It would be better to make the scope and limitation of the work absolutely clear as early as possible. (2) The paper is indeed very technical. Since the decoupling mechanism is the most important contribution, the author may consider demonstrating this contribution first in a finite state action MDP as a warm-up. This can significantly improve the readability of the work and help the reader to understand the main contribution easily.
Decision: Accept (poster)